CI/CD

Flaky tests: the poison in your pipeline

Nothing destroys trust in a pipeline faster than a flaky test: one that passes on one run and fails on the next, with nothing having changed. When the team starts clicking "re-run" by reflex, CI has become decoration.

Where flakiness comes from

  • Timing: fixed sleep instead of waiting for a condition; race conditions.
  • Shared state: tests that depend on order or on data left behind by another test.
  • Environment: network, clock, time zone, competing resources in CI.
  • Uncontrolled randomness: random data without a seed.

How to fight it

  1. Wait for a condition, not for time. Use auto-wait / web-first assertions instead of sleep(3000).
  2. Isolate state. Each test creates and cleans up its own data; no order dependencies.
  3. Make time deterministic. Freeze the clock and fix randomness seeds.
  4. Quarantine. Mark the flaky test, take it out of the gate and open a ticket — don't let it "train" the team to ignore failures.
# Detect flakiness by running the same test many times
npx playwright test suspeito.spec.ts --repeat-each=20
A flaky test is worse than no test: it costs time and, little by little, teaches the team not to trust green.

Conclusion

Flakiness is quality technical debt. Treating it with the same seriousness as a production bug is what keeps the pipeline a source of truth — and not an obstacle to be worked around.

Read also

Back to the blog