Use cases / Dev
Dev
Hunt flaky tests by having a coding agent rerun your suite overnight
A developer pointed Codex at their test suite and had it run the suite 59 times overnight to find tests that pass and fail at random. Along the way it uncovered three real production bugs, which shows that some flaky tests are timing problems in the code, not just noise.
- Tool
- Codex
- Source
- @aarondfrancis on X
- Source date
Credit: this use case was shared by @aarondfrancis on X. Summary written by Kitbox; see the original for full details.
Try it yourself
Our own starter prompt. Fill in the brackets, then paste it into Codex or your agent of choice.
Starter prompt
Run the full test suite repeatedly, at least 50 times, and record every test that fails in some runs but not others. For each flaky test, look for the cause, such as shared state, ordering, timing, or a real race in the application code. Separate the ones that point to a production bug from the ones that are only test problems. Give me a report with the evidence for each and suggested fixes, but don't change application code without asking.
More in Dev