Flaky tests, test frameworks, E2E, mocking.
Our API had 1,400 tests and still shipped a breaking field rename, a 40-second p99 under load and an object-level authorization hole. Each one belonged to a test type we were not running. This is the map we now use: which API test type catches which failure, with a small example of each.
Four engineers, one invoicing product, no QA team. This is the risk-based testing strategy we settled on: integration tests over mocks, contract tests at service edges, a handful of E2E happy paths and test data that does not rot.
A handful of tests in our Python service failed only when the whole suite ran, and only in some orders. A session-scoped fixture returned a mutable settings object that one test modified. How we proved it with random ordering and bisection, and how fixture scopes, factories and monkeypatch keep tests isolated.
A prompt tweak that looks better on five hand-picked examples can quietly break a tenth of real traffic. This guide shows how to build an eval set from production, choose graders, use LLM-as-judge without fooling yourself and gate changes in CI.
After we set "type": "module" in package.json, jest.mock stopped replacing our mailer and tests started calling the real SMTP client. Static ESM imports are linked before any mock runs. Here is the mechanism and the fix with unstable_mockModule.
Running the test suite inside our app container dropped every table in the development database. Docker Compose injected DB_DATABASE as a real environment variable, and phpunit.xml's env entries do not override existing variables. Here is the full precedence chain and three layers of defence.
A coding agent implemented a discount rule and wrote tests for it. Everything was green, the review looked fine, and production applied the discount to the wrong orders. The agent had written tests from its own implementation, so they asserted the bug. How we changed the workflow so humans own the acceptance criteria.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.
Our video-call regressions were found by customers because every test needed two humans with webcams. We automated it: fake camera and microphone feeds in Chromium and Firefox, two-party calls in one Playwright test, assertions on getStats() instead of screenshots, and tc netem for packet loss and latency.
One CI run in eight failed somewhere in our checkout E2E suite, never in the same test twice. The causes were fixed sleeps, URL checks that raced navigation and text selectors that matched two buttons. Web-first assertions, role locators and traces fixed it.
A good bug report lets someone who has never seen your screen reproduce the problem in minutes. The anatomy we use: environment, exact steps, expected versus actual, evidence, a minimal reproduction, and severity kept separate from priority.
A billing test passed for weeks and failed this morning, 31 January, for every engineer at once. JavaScript's setMonth rolled 31 January into 3 March, and the test read the real clock. We fixed the arithmetic, injected the clock and added property tests over every day.
Our shared Terraform modules broke consumers twice in one quarter. We added terraform test: fast plan-mode tests with mocked providers on every pull request, and a nightly apply-mode suite against a sandbox account. The test layout, what to assert, and what it costs.