Your CI Retries Are Hiding Real Bugs
DevOps.com, Tuesday, November 3rd, 2026
An experiment shows two CI test retries let 99.2% of builds with an intermittent bug pass, and proposes a quarantine policy.
Test engineering manager Serhii Romanov planted a bug that fails one run in five in a toy Python service and ran its suite 41,200 times.
With two retries, the default in new Playwright projects, 99.2% of buggy builds stayed green, because tools flag a pass on retry as flaky but still exit with code 0.
He found a new regression stands out by its onset, when a test with a clean record starts failing, and that bisecting it needs about 14 runs per step at a 20% failure rate.
His policy: record every attempt, check onset before quarantining, give quarantined tests owners, deadlines and a cap, and keep them running.