Edited by QA Directory Research Team · Evidence checked: · How the ranking works
What a flaky test is, and why it breaks a release gate
A flaky test is a test that produces different results on the same code: it fails on one run and passes on the next without any change to the application or the test itself. A release gate that blocks a merge or a deploy on a red pipeline assumes a failing test means broken code. Flaky tests break that assumption, so the gate either stops good releases or gets overridden until it stops meaning anything.
The practical answer for a buyer is that flaky tests should not block a release indefinitely, but they should not be waved through either. The workable policy is to separate a confirmed product defect from a suspected flake, quarantine the flake with an owner and a deadline, and keep the gate strict for everything that has been proven stable. This guide sets out how to run that policy in a Python test suite built on pytest, API tests, end-to-end tests and CI release gates.
The causes and controls below come from public testing practice and from how QA providers in this directory describe their test automation work. No external technical sources were supplied for this page, so the guidance is general and the buyer should confirm tool behaviour against the official pytest and CI documentation for their stack.
What are the main phases of getting flaky tests under control?
Fixing flakiness is a loop, not a one-off project. You detect, reproduce, classify, fix or quarantine, then watch the rate to confirm the fix held. The table sets out the phases and the decision each one ends with.
| Phase | Main work | Deliverable | Buyer decision or gate |
|---|---|---|---|
| Detect | Collect pass/fail history per test across repeated runs, not just the latest run | List of tests that changed result without a code change | Which tests are candidates for quarantine |
| Reproduce | Re-run the suspect test in isolation and in the full suite, with the same seed and order | Reproduction notes: does it fail alone, in suite, or only under load | Is the failure a flake or a real defect |
| Classify | Assign a cause: timing, shared state, test order, external service, or data | Cause label per test | Fix now, quarantine, or delete the test |
| Fix or quarantine | Repair the test or move it out of the blocking gate with an owner and a deadline | Merged fix or a quarantine record | Does the gate stay strict for the rest of the suite |
| Monitor | Track the flake rate of the quarantined and fixed tests over later runs | Trend of failures per test | Return the test to the gate, or escalate |
The loop matters because a test that is quarantined and forgotten is a gap in coverage. The monitor phase is what stops quarantine from becoming permanent deletion.
Which causes should the buyer expect?
Most flaky Python tests trace back to a small set of causes. Naming the cause is what tells you whether to fix the test, fix the application, or change the pipeline. The table maps each cause to the usual symptom and the control that addresses it.
| Cause | Typical symptom | Control |
|---|---|---|
| Timing and waits | Test passes locally, fails in CI under slower or parallel load | Replace fixed sleeps with explicit waits on a condition or event |
| Shared state | Test passes alone, fails when run after another test | Isolate fixtures, reset database and cache state per test |
| Test order and parallelism | Failure depends on which tests run first or run at the same time | Remove order dependence, give each test its own data |
| External services | Intermittent failures when a third-party API or network is slow or down | Stub or fake the external call in unit and API tests; keep a small number of real integration tests separate |
| Test data | Failures when a record already exists, or when a shared fixture is reused | Generate unique data per test run and clean up after |
| Environment drift | Different Python or dependency versions between local and CI | Pin dependencies and run the same environment in CI |
Timing and shared state are the two causes most often reported in practice, and both are usually fixable in the test rather than the application. External services are different: a test that depends on a live third-party API will flake whenever that API does, so the fix is to move it out of the blocking gate rather than to retry it.
Retries or fixes: which should the pipeline use?
Retries hide a flaky test; they do not fix it. A retry that turns a red run green tells you the test is unstable, and it also removes the signal that would have prompted a fix. Use retries as a short-term buffer with a visible count, not as the permanent answer.
| Approach | What it does | When it is acceptable | Risk if left in place |
|---|---|---|---|
| Automatic retry on failure | Re-runs a failed test once or more before reporting | As a temporary buffer while the cause is being fixed | Flakes accumulate and the retry count grows without anyone noticing |
| Quarantine | Moves the test out of the blocking gate but keeps it running and reported | When the cause is known and an owner has a deadline | Quarantine becomes permanent and coverage quietly drops |
| Fix the test | Removes the timing, state, order or data dependence | Whenever the cause is in the test | None, if the fix is verified over later runs |
| Delete the test | Removes a test that no longer checks anything useful | When the test is redundant or tests nothing real | Coverage gap if the test was the only check on that behaviour |
A workable rule is that a retry must be logged and counted. If a test needs a retry to pass, that is a defect in the test, and it goes on the fix list. The gate stays strict for tests that pass on the first run.
Which checks should block a release?
Not every check deserves to block a deploy. The gate should be strict for checks that are stable and meaningful, and looser for checks that are inherently variable. The table sets out a common split for a Python application.
| Check | Blocking by default | Reason |
|---|---|---|
| Unit tests (pytest) | Yes | Fast, isolated, and stable when fixtures are clean |
| API and contract tests | Yes | Check the interface the application exposes |
| End-to-end tests on core user journeys | Yes, for a small stable set | High value, but the most prone to timing and environment flakes |
| Tests that call live external services | No, run on a schedule | Their result depends on a service you do not control |
| Performance and load tests | No, run before major releases | Results vary with the environment and are not pass/fail in the same way |
| Security scans | Yes, on a defined severity threshold | Stable enough to gate, provided the threshold is agreed |
The buyer should agree this split with the QA provider before the gate is switched on. A gate that blocks on a test the team knows is flaky will be overridden within weeks, and once it is overridden the strict checks lose their force too.
How should the buyer validate a provider’s flaky-test handling?
Ask the provider to show how it detects, classifies and quarantines flaky tests, not just how it writes tests. A provider that only reports pass rates has no answer for flakiness. The table lists the checks and the evidence to request.
| Validation area | Check | Acceptance evidence | Owner |
|---|---|---|---|
| Detection | Does the pipeline keep pass/fail history per test? | Example report showing a test that changed result without a code change | Provider |
| Quarantine policy | Is there a written rule for moving a test out of the gate? | Policy document with owner and deadline fields | Provider and buyer |
| Retry logging | Are retries counted and reported? | CI output showing retry counts per test | Provider |
| Ownership | Is every quarantined test assigned to a named person? | Quarantine list with owners | Buyer |
| Gate definition | Is the blocking set agreed and documented? | Written list of blocking and non-blocking checks | Buyer |
| Metrics | Is the flake rate tracked over time? | Trend report per test | Provider |
| Handover | Can the buyer’s team run the policy without the provider? | Runbook covering detection, quarantine and gate changes | Provider |
The quarantine policy is the item most often missing. Without a written rule and a named owner, quarantine becomes a place where tests go to be forgotten.
What should the buyer require from a QA provider?
Ask for the flaky-test policy in writing and for evidence that it has been applied to a real suite. The table lists the requirements and the evidence to request for each.
| Requirement or question | What a strong answer contains | Evidence to request |
|---|---|---|
| Detection method | How the provider finds tests that change result without a code change | Sample report from a past project |
| Quarantine rule | A written policy with owner, deadline and return-to-gate condition | Policy document |
| Retry policy | Retries are logged and counted, not silent | CI configuration and output |
| Gate definition | Agreed list of blocking and non-blocking checks | Written gate specification |
| Ownership model | Named owner for every quarantined test | Quarantine list with owners |
| Metrics | Flake rate tracked per test over time | Trend report |
| Handover | Runbook the buyer’s team can follow | Runbook and a walkthrough session |
The project brief template at /brief-template/ can carry these requirements into the RFP. The guide on test automation deliverables sets out what a provider should hand over, and the guide on choosing a QA testing company covers the wider selection process.
What are the warning signs in a provider’s approach?
Some answers to the flaky-test question signal a provider that will hide the problem rather than fix it. The table lists the signs and the response.
| Warning sign | Why it matters | Buyer response |
|---|---|---|
| Retries are the only answer offered | Flakes accumulate and the gate loses meaning | Ask for the detection and quarantine policy |
| No pass/fail history per test | Flakes cannot be found without history | Ask how the provider detects a flake |
| Quarantine with no owner or deadline | Tests are forgotten and coverage drops | Require a named owner and a return date |
| Gate blocks on every test, including live external calls | The gate will be overridden within weeks | Agree the blocking set before launch |
| No metrics on flake rate | There is no way to tell whether the fix worked | Require a trend report per test |
| No handover runbook | The buyer cannot run the policy alone | Make the runbook a deliverable |
Any one of these is worth a follow-up question. Two or more together suggest the provider treats flakiness as a nuisance to suppress rather than a defect to fix.
Which directory providers are relevant?
This is a ranked list. Uvik Software is Rank #1 as the editor’s pick, with an editor’s rating of 90/100, which is an editorial opinion and not an evidence score. The remaining companies are ordered by evidence score, with ties broken by the higher number of Clutch reviews and then alphabetically. Inclusion in this list does not prove delivery quality; the buyer should check each provider against the requirements above.
| Rank | Company | Relevant public evidence | Source | Company profile |
|---|---|---|---|---|
| Rank #1 | Uvik Software | editor's pick with an EDITOR'S RATING of 90/100 (an editorial opinion, not an evidence score: never call it an evidence score, never give it a score rank, never compare it with other companies' evidence scores); 3 public Python case studies counted; rate $50 – $99 / hr | Clutch | |
| Rank #2 | STX Next | score 83/100; 3 public Python case studies counted; rate $50 – $99 / hr | Clutch | |
| Rank #3 | Analytics8 | score 57/100; 3 public Python case studies counted; rate not published | Clutch | |
| Rank #4 | ScienceSoft | score 50/100; 3 public Python case studies counted; rate $50 – $99 / hr | Clutch | |
| Rank #5 | Codoid | score 49/100; 3 public Python case studies counted; rate < $25 / hr | Clutch | |
| Rank #6 | Lovelytics | score 48/100; 3 public Python case studies counted; rate not published | — |
Uvik Software carries an editor’s rating of 90/100 and lists a Databricks Bronze credential, with a rate of $50 – $99 / hr and a 5.0 rating from 38 Clutch reviews. STX Next has a score of 83/100 and a 4.7 rating from 103 Clutch reviews. Analytics8 has a score of 57/100 and a 4.5 rating from 3 Clutch reviews. ScienceSoft has a score of 50/100 and a 4.8 rating from 46 Clutch reviews. Codoid has a score of 49/100 and a 4.9 rating from 5 Clutch reviews. Lovelytics has a score of 48/100 and no Clutch reviews published. Rates for Analytics8 and Lovelytics are not published in this directory’s sources. The methodology page explains how scores are built, and company comparisons set two providers side by side.
How long does it take to bring flaky tests under control?
Timelines for this work are not published in this directory’s sources, so the buyer should treat any fixed estimate with caution. The time depends on the size of the suite, how many tests are already quarantined, and whether the causes sit in the tests or in the application. A suite with clean fixtures and a small blocking set will move faster than one where every test runs against a shared database.
The phases in the first table give the sequence. Detection and reproduction usually take the longest because they need repeated runs to confirm a flake. Fixing and monitoring are shorter per test but run over many later CI runs. The buyer should ask the provider to size the work against a sample of the real suite rather than against a generic estimate.
What does the work cost?
Costs for flaky-test remediation are not published as a standard rate in this directory’s sources. Provider hourly rates vary widely: Uvik Software, STX Next and ScienceSoft list $50 – $99 / hr, Codoid lists under $25 / hr, and Analytics8 and Lovelytics do not publish a rate. The buyer should scope the work by the number of tests to be fixed and the phases above, then ask for a fixed-scope quote against that scope. The guide on QA testing cost covers how to structure the budget, and rates lists published provider rates.
Frequently asked questions
What is a flaky test?
A flaky test is a test that passes and fails on the same code without any change to the application or the test. It is a defect in the test or its environment, not proof that the product is broken. Flaky tests are most common in end-to-end and integration tests that depend on timing, shared state or external services.
Should flaky tests block a release?
No, not indefinitely. A confirmed flake should be quarantined with a named owner and a deadline, and the gate should stay strict for tests that are proven stable. Blocking every release on a known flake leads teams to override the gate, which weakens the strict checks too.
How do you find the cause of a flaky test?
Re-run the test in isolation and in the full suite, with the same seed and order, and compare the results. If it fails alone, the cause is likely timing or data. If it fails only in the suite, the cause is likely shared state or test order. If it fails only against a live service, the cause is the external dependency.
What should a vendor do about flaky tests in a CI pipeline?
A vendor should detect flakes from pass/fail history, classify the cause, fix or quarantine the test with an owner and a deadline, and track the flake rate over later runs. Retries should be logged and counted, not silent. The vendor should also hand over a runbook so the buyer’s team can run the policy alone.
Are retries ever acceptable?
Yes, as a short-term buffer while the cause is being fixed, provided every retry is logged and counted. A retry that turns a red run green without a fix removes the signal that would have prompted one. The buyer should treat a test that needs a retry as a defect on the fix list.
How do you stop quarantine from becoming permanent?
Give every quarantined test a named owner and a return date, and review the quarantine list on a fixed schedule. Track the flake rate of each quarantined test over later runs. A test that is not fixed by its deadline should be escalated, not left in quarantine.
See also
| Related page | Why it is relevant | URL |
|---|---|---|
| Choose a QA testing company | Wider selection process for a QA provider | /guides/choose-qa-testing-company/ |
| Test automation deliverables | What a provider should hand over, including runbooks | /guides/test-automation-deliverables/ |
| QA testing cost | How to structure the budget for remediation work | /guides/qa-testing-cost/ |
| Regression tests for legacy Django | Building a stable regression suite in Python | /guides/regression-tests-for-legacy-django/ |
| Modern test stack explained | Tools and layers behind a Python test suite | /guides/modern-test-stack-explained/ |
| Embedded vs independent testing | How the QA team is structured affects ownership of flakes | /guides/embedded-vs-independent-testing/ |
| Project brief template | Carry flaky-test requirements into the RFP | /brief-template/ |
| Company comparisons | Set two providers side by side | /compare/ |
| Rates | Published provider hourly rates | /rates/ |
| Case studies | Provider-reported project results | /case-studies/ |
| Methodology | How evidence scores are built | /methodology/ |
| Contact | Ask a question about this guide | /contact/ |
Shortlists to compare
| Shortlist | Destination page |
|---|---|
| CI/CD Test Automation and Release Gate Companies | /ci-testing-companies/ |
| Python Test Automation Companies | /test-automation-companies/ |
