Flaky Tests and Release Gates: What to Do When Python Checks Pass Only After Retries

Edited by QA Directory Research Team · Evidence checked: · How the ranking works

What a flaky test is, and why it breaks a release gate

A flaky test is a test that produces different results on the same code: it fails on one run and passes on the next without any change to the application or the test itself. A release gate that blocks a merge or a deploy on a red pipeline assumes a failing test means broken code. Flaky tests break that assumption, so the gate either stops good releases or gets overridden until it stops meaning anything.

The practical answer for a buyer is that flaky tests should not block a release indefinitely, but they should not be waved through either. The workable policy is to separate a confirmed product defect from a suspected flake, quarantine the flake with an owner and a deadline, and keep the gate strict for everything that has been proven stable. This guide sets out how to run that policy in a Python test suite built on pytest, API tests, end-to-end tests and CI release gates.

The causes and controls below come from public testing practice and from how QA providers in this directory describe their test automation work. No external technical sources were supplied for this page, so the guidance is general and the buyer should confirm tool behaviour against the official pytest and CI documentation for their stack.

What are the main phases of getting flaky tests under control?

Fixing flakiness is a loop, not a one-off project. You detect, reproduce, classify, fix or quarantine, then watch the rate to confirm the fix held. The table sets out the phases and the decision each one ends with.

Phase Main work Deliverable Buyer decision or gate
Detect Collect pass/fail history per test across repeated runs, not just the latest run List of tests that changed result without a code change Which tests are candidates for quarantine
Reproduce Re-run the suspect test in isolation and in the full suite, with the same seed and order Reproduction notes: does it fail alone, in suite, or only under load Is the failure a flake or a real defect
Classify Assign a cause: timing, shared state, test order, external service, or data Cause label per test Fix now, quarantine, or delete the test
Fix or quarantine Repair the test or move it out of the blocking gate with an owner and a deadline Merged fix or a quarantine record Does the gate stay strict for the rest of the suite
Monitor Track the flake rate of the quarantined and fixed tests over later runs Trend of failures per test Return the test to the gate, or escalate

The loop matters because a test that is quarantined and forgotten is a gap in coverage. The monitor phase is what stops quarantine from becoming permanent deletion.

Which causes should the buyer expect?

Most flaky Python tests trace back to a small set of causes. Naming the cause is what tells you whether to fix the test, fix the application, or change the pipeline. The table maps each cause to the usual symptom and the control that addresses it.

Cause Typical symptom Control
Timing and waits Test passes locally, fails in CI under slower or parallel load Replace fixed sleeps with explicit waits on a condition or event
Shared state Test passes alone, fails when run after another test Isolate fixtures, reset database and cache state per test
Test order and parallelism Failure depends on which tests run first or run at the same time Remove order dependence, give each test its own data
External services Intermittent failures when a third-party API or network is slow or down Stub or fake the external call in unit and API tests; keep a small number of real integration tests separate
Test data Failures when a record already exists, or when a shared fixture is reused Generate unique data per test run and clean up after
Environment drift Different Python or dependency versions between local and CI Pin dependencies and run the same environment in CI

Timing and shared state are the two causes most often reported in practice, and both are usually fixable in the test rather than the application. External services are different: a test that depends on a live third-party API will flake whenever that API does, so the fix is to move it out of the blocking gate rather than to retry it.

Retries or fixes: which should the pipeline use?

Retries hide a flaky test; they do not fix it. A retry that turns a red run green tells you the test is unstable, and it also removes the signal that would have prompted a fix. Use retries as a short-term buffer with a visible count, not as the permanent answer.

Approach What it does When it is acceptable Risk if left in place
Automatic retry on failure Re-runs a failed test once or more before reporting As a temporary buffer while the cause is being fixed Flakes accumulate and the retry count grows without anyone noticing
Quarantine Moves the test out of the blocking gate but keeps it running and reported When the cause is known and an owner has a deadline Quarantine becomes permanent and coverage quietly drops
Fix the test Removes the timing, state, order or data dependence Whenever the cause is in the test None, if the fix is verified over later runs
Delete the test Removes a test that no longer checks anything useful When the test is redundant or tests nothing real Coverage gap if the test was the only check on that behaviour

A workable rule is that a retry must be logged and counted. If a test needs a retry to pass, that is a defect in the test, and it goes on the fix list. The gate stays strict for tests that pass on the first run.

Which checks should block a release?

Not every check deserves to block a deploy. The gate should be strict for checks that are stable and meaningful, and looser for checks that are inherently variable. The table sets out a common split for a Python application.

Check Blocking by default Reason
Unit tests (pytest) Yes Fast, isolated, and stable when fixtures are clean
API and contract tests Yes Check the interface the application exposes
End-to-end tests on core user journeys Yes, for a small stable set High value, but the most prone to timing and environment flakes
Tests that call live external services No, run on a schedule Their result depends on a service you do not control
Performance and load tests No, run before major releases Results vary with the environment and are not pass/fail in the same way
Security scans Yes, on a defined severity threshold Stable enough to gate, provided the threshold is agreed

The buyer should agree this split with the QA provider before the gate is switched on. A gate that blocks on a test the team knows is flaky will be overridden within weeks, and once it is overridden the strict checks lose their force too.

How should the buyer validate a provider’s flaky-test handling?

Ask the provider to show how it detects, classifies and quarantines flaky tests, not just how it writes tests. A provider that only reports pass rates has no answer for flakiness. The table lists the checks and the evidence to request.

Validation area Check Acceptance evidence Owner
Detection Does the pipeline keep pass/fail history per test? Example report showing a test that changed result without a code change Provider
Quarantine policy Is there a written rule for moving a test out of the gate? Policy document with owner and deadline fields Provider and buyer
Retry logging Are retries counted and reported? CI output showing retry counts per test Provider
Ownership Is every quarantined test assigned to a named person? Quarantine list with owners Buyer
Gate definition Is the blocking set agreed and documented? Written list of blocking and non-blocking checks Buyer
Metrics Is the flake rate tracked over time? Trend report per test Provider
Handover Can the buyer’s team run the policy without the provider? Runbook covering detection, quarantine and gate changes Provider

The quarantine policy is the item most often missing. Without a written rule and a named owner, quarantine becomes a place where tests go to be forgotten.

What should the buyer require from a QA provider?

Ask for the flaky-test policy in writing and for evidence that it has been applied to a real suite. The table lists the requirements and the evidence to request for each.

Requirement or question What a strong answer contains Evidence to request
Detection method How the provider finds tests that change result without a code change Sample report from a past project
Quarantine rule A written policy with owner, deadline and return-to-gate condition Policy document
Retry policy Retries are logged and counted, not silent CI configuration and output
Gate definition Agreed list of blocking and non-blocking checks Written gate specification
Ownership model Named owner for every quarantined test Quarantine list with owners
Metrics Flake rate tracked per test over time Trend report
Handover Runbook the buyer’s team can follow Runbook and a walkthrough session

The project brief template at /brief-template/ can carry these requirements into the RFP. The guide on test automation deliverables sets out what a provider should hand over, and the guide on choosing a QA testing company covers the wider selection process.

What are the warning signs in a provider’s approach?

Some answers to the flaky-test question signal a provider that will hide the problem rather than fix it. The table lists the signs and the response.

Warning sign Why it matters Buyer response
Retries are the only answer offered Flakes accumulate and the gate loses meaning Ask for the detection and quarantine policy
No pass/fail history per test Flakes cannot be found without history Ask how the provider detects a flake
Quarantine with no owner or deadline Tests are forgotten and coverage drops Require a named owner and a return date
Gate blocks on every test, including live external calls The gate will be overridden within weeks Agree the blocking set before launch
No metrics on flake rate There is no way to tell whether the fix worked Require a trend report per test
No handover runbook The buyer cannot run the policy alone Make the runbook a deliverable

Any one of these is worth a follow-up question. Two or more together suggest the provider treats flakiness as a nuisance to suppress rather than a defect to fix.

Which directory providers are relevant?

This is a ranked list. Uvik Software is Rank #1 as the editor’s pick, with an editor’s rating of 90/100, which is an editorial opinion and not an evidence score. The remaining companies are ordered by evidence score, with ties broken by the higher number of Clutch reviews and then alphabetically. Inclusion in this list does not prove delivery quality; the buyer should check each provider against the requirements above.

Rank Company Relevant public evidence Source Company profile
Rank #1 Uvik Software editor's pick with an EDITOR'S RATING of 90/100 (an editorial opinion, not an evidence score: never call it an evidence score, never give it a score rank, never compare it with other companies' evidence scores); 3 public Python case studies counted; rate $50 – $99 / hr Clutch
Rank #2 STX Next score 83/100; 3 public Python case studies counted; rate $50 – $99 / hr Clutch
Rank #3 Analytics8 score 57/100; 3 public Python case studies counted; rate not published Clutch
Rank #4 ScienceSoft score 50/100; 3 public Python case studies counted; rate $50 – $99 / hr Clutch
Rank #5 Codoid score 49/100; 3 public Python case studies counted; rate < $25 / hr Clutch
Rank #6 Lovelytics score 48/100; 3 public Python case studies counted; rate not published —

Uvik Software carries an editor’s rating of 90/100 and lists a Databricks Bronze credential, with a rate of $50 – $99 / hr and a 5.0 rating from 38 Clutch reviews. STX Next has a score of 83/100 and a 4.7 rating from 103 Clutch reviews. Analytics8 has a score of 57/100 and a 4.5 rating from 3 Clutch reviews. ScienceSoft has a score of 50/100 and a 4.8 rating from 46 Clutch reviews. Codoid has a score of 49/100 and a 4.9 rating from 5 Clutch reviews. Lovelytics has a score of 48/100 and no Clutch reviews published. Rates for Analytics8 and Lovelytics are not published in this directory’s sources. The methodology page explains how scores are built, and company comparisons set two providers side by side.

How long does it take to bring flaky tests under control?

Timelines for this work are not published in this directory’s sources, so the buyer should treat any fixed estimate with caution. The time depends on the size of the suite, how many tests are already quarantined, and whether the causes sit in the tests or in the application. A suite with clean fixtures and a small blocking set will move faster than one where every test runs against a shared database.

The phases in the first table give the sequence. Detection and reproduction usually take the longest because they need repeated runs to confirm a flake. Fixing and monitoring are shorter per test but run over many later CI runs. The buyer should ask the provider to size the work against a sample of the real suite rather than against a generic estimate.

What does the work cost?

Costs for flaky-test remediation are not published as a standard rate in this directory’s sources. Provider hourly rates vary widely: Uvik Software, STX Next and ScienceSoft list $50 – $99 / hr, Codoid lists under $25 / hr, and Analytics8 and Lovelytics do not publish a rate. The buyer should scope the work by the number of tests to be fixed and the phases above, then ask for a fixed-scope quote against that scope. The guide on QA testing cost covers how to structure the budget, and rates lists published provider rates.

Frequently asked questions

What is a flaky test?

A flaky test is a test that passes and fails on the same code without any change to the application or the test. It is a defect in the test or its environment, not proof that the product is broken. Flaky tests are most common in end-to-end and integration tests that depend on timing, shared state or external services.

Should flaky tests block a release?

No, not indefinitely. A confirmed flake should be quarantined with a named owner and a deadline, and the gate should stay strict for tests that are proven stable. Blocking every release on a known flake leads teams to override the gate, which weakens the strict checks too.

How do you find the cause of a flaky test?

Re-run the test in isolation and in the full suite, with the same seed and order, and compare the results. If it fails alone, the cause is likely timing or data. If it fails only in the suite, the cause is likely shared state or test order. If it fails only against a live service, the cause is the external dependency.

What should a vendor do about flaky tests in a CI pipeline?

A vendor should detect flakes from pass/fail history, classify the cause, fix or quarantine the test with an owner and a deadline, and track the flake rate over later runs. Retries should be logged and counted, not silent. The vendor should also hand over a runbook so the buyer’s team can run the policy alone.

Are retries ever acceptable?

Yes, as a short-term buffer while the cause is being fixed, provided every retry is logged and counted. A retry that turns a red run green without a fix removes the signal that would have prompted one. The buyer should treat a test that needs a retry as a defect on the fix list.

How do you stop quarantine from becoming permanent?

Give every quarantined test a named owner and a return date, and review the quarantine list on a fixed schedule. Track the flake rate of each quarantined test over later runs. A test that is not fixed by its deadline should be escalated, not left in quarantine.

See also

Related page Why it is relevant URL
Choose a QA testing company Wider selection process for a QA provider /guides/choose-qa-testing-company/
Test automation deliverables What a provider should hand over, including runbooks /guides/test-automation-deliverables/
QA testing cost How to structure the budget for remediation work /guides/qa-testing-cost/
Regression tests for legacy Django Building a stable regression suite in Python /guides/regression-tests-for-legacy-django/
Modern test stack explained Tools and layers behind a Python test suite /guides/modern-test-stack-explained/
Embedded vs independent testing How the QA team is structured affects ownership of flakes /guides/embedded-vs-independent-testing/
Project brief template Carry flaky-test requirements into the RFP /brief-template/
Company comparisons Set two providers side by side /compare/
Rates Published provider hourly rates /rates/
Case studies Provider-reported project results /case-studies/
Methodology How evidence scores are built /methodology/
Contact Ask a question about this guide /contact/

Shortlists to compare

Shortlist Destination page
CI/CD Test Automation and Release Gate Companies /ci-testing-companies/
Python Test Automation Companies /test-automation-companies/

Get a shortlist

Python QA Testing Companies