Tests passed alone and failed in the full suite: a session-scoped pytest fixture leaked state

Our order service has about 1,800 pytest tests. Three tests in test_pricing.py started failing in CI, but only sometimes, and never when run on their own:

pytest tests/test_pricing.py          # 46 passed
pytest                                # 3 failed, 1797 passed

FAILED tests/test_pricing.py::test_vat_included_for_eu_customers
    assert quote.total == Decimal("121.00")
AssertionError: assert Decimal('100.00') == Decimal('121.00')

The VAT was missing, as if VAT calculation were disabled.

Making the order visible

We installed pytest-randomly, which shuffles test order and prints the seed, and ran the suite several times:

pytest -p randomly -q            # prints "Using --randomly-seed=1840517364"

Some seeds passed, some failed. With a failing seed, we could reproduce reliably:

pytest -p randomly --randomly-seed=1840517364

Bisecting

To find which earlier test caused the failure, we used pytest --collect-only to list the order for that seed, then ran halves of the tests that came before the failing one, plus the failing test, until one remained:

pytest tests/test_admin_settings.py::test_disable_vat_for_internal_orders \
       tests/test_pricing.py::test_vat_included_for_eu_customers
# 1 failed, 1 passed

Two tests, one failure. The admin test ran first, the pricing test failed.

The fixture and the test

# conftest.py
@pytest.fixture(scope="session")
def settings():
    return load_settings("tests/settings.toml")      # returns a mutable Settings object

# tests/test_admin_settings.py
def test_disable_vat_for_internal_orders(settings, client):
    settings.vat_enabled = False                      # mutates the shared object
    resp = client.post("/admin/orders", json={"internal": True, "lines": [...]})
    assert resp.json()["vat"] == "0.00"

The session-scoped fixture created one Settings object for the entire test run. The admin test changed it and never restored it. Every test that used settings afterwards, in the same run, saw VAT disabled.

What we ruled out

• Database state. Each test ran in a transaction rolled back at the end; the database was clean.
• Time or date dependence. The failures followed test order, not time.
• Parallel execution. The failures reproduced with a single process.

Root cause

A session-scoped fixture returned a mutable object shared by every test in the run. One test modified it in place without restoring it, so later tests depended on whether that test had run before them. The order of tests changed whenever files were added or renamed, which made the failures appear random.

Fixture scopes

Broad scopes are useful for expensive resources: a database server, a compiled application, an HTTP client pool. They are dangerous for anything a test might change, because every user of the fixture shares the same instance.

Why tests passed in isolation

Run alone, the pricing tests received a fresh Settings object with VAT enabled. Only when another test mutated the shared object earlier in the same run did they fail. Order-dependent tests are a classic source of flaky CI.

Who is affected

Test suites with module, package or session fixtures that return mutable objects: settings, clients with mutable configuration, in-memory fakes, caches, lists of fixtures data. The longer a suite grows, the more likely some test mutates a shared object. Running tests in parallel with pytest-xdist changes which tests share a worker process, so these failures often appear or disappear when CI parallelism changes.

Environment

Expensive part session-scoped, mutable part function-scoped

Loading the settings file was not expensive, but the pattern generalises: keep the costly, read-only part at a broad scope, and give each test its own copy of anything mutable:

# conftest.py
@pytest.fixture(scope="session")
def base_settings_data():
    return tomllib.loads(Path("tests/settings.toml").read_text())   # plain dict, treated as read-only

@pytest.fixture
def settings(base_settings_data):
    return Settings.model_validate(copy.deepcopy(base_settings_data))

Every test gets a fresh Settings. A test that changes it affects only itself.

Use monkeypatch for deliberate changes

When a test needs a different setting in application code that reads a global configuration, use monkeypatch, which restores the original value after the test even if it fails:

def test_disable_vat_for_internal_orders(monkeypatch, client):
    monkeypatch.setattr(app_settings, "vat_enabled", False)
    resp = client.post("/admin/orders", json={"internal": True, "lines": [...]})
    assert resp.json()["vat"] == "0.00"

Make shared fixtures harder to mutate

For fixtures that must stay session-scoped, return immutable objects where possible: frozen dataclasses, Pydantic models with frozen=True, or types.MappingProxyType around dictionaries. An accidental assignment then fails loudly in the test that does it:

@pytest.fixture(scope="session")
def feature_flags():
    return MappingProxyType({"new_checkout": True, "vat_enabled": True})

Keep random ordering on

pytest-randomly stayed in the project. Every CI run uses a new random order and prints the seed, so order dependencies show up early and can be reproduced. It also reseeds random per test, which removed another class of flaky tests.

Alternatives and trade-offs

• pytest-xdist alone changes which tests share a worker and can expose order dependencies, but it is not a reliable detector. Random ordering is.
• Restoring state in a yield fixture teardown works for some resources, but a fresh object per test is simpler and does not depend on every test cooperating.

Auditing the other broad fixtures

We listed every fixture with a non-function scope and checked what each returned:

pytest --fixtures -v | grep -E 'scope: (session|module|package|class)'

Of 31 broad-scoped fixtures, 26 returned resources that tests could not meaningfully mutate (database engines, compiled apps, read-only file paths), two returned mutable dictionaries that we converted to MappingProxyType, and three were narrowed to function scope with no measurable slowdown.

How to test the fix

for seed in $(seq 1 20); do pytest -q -p randomly --randomly-seed=$seed || echo "failed seed $seed"; done

All twenty seeds passed after the change. The previously failing pair also passed in both orders.

Confirming in production

Over the following month, CI had no failures of the pricing tests, and random ordering surfaced two other order-dependent tests in a different module, which we fixed the same way before they caused flaky builds.

What changed for the team

Developers now run pytest -p randomly locally by default through the project configuration, so they see order dependencies on their own machines before pushing. When a random-order failure appears in CI, the seed in the log reproduces it exactly, which turned "flaky test, rerun the job" into a normal bug with a clear reproduction.

Regression guard

• pytest-randomly is enabled by default in the project's configuration.
• A conftest check fails the run if a session-scoped fixture returns a list, dict or set directly (a small plugin inspects fixture results at setup).
• Code review guideline: session or module scope requires a comment explaining why the object cannot be mutated.

Security implications

None directly. Tests that pass only because of leftover state can hide real bugs, including in authorization logic, for example a test that passes because an earlier test left an admin user logged in. Isolated fixtures make such tests trustworthy.

Prevention checklist

• Keep fixtures at function scope unless they are expensive and read-only.
• Never return shared mutable objects from session or module fixtures.
• Use monkeypatch for temporary changes to global state.
• Run tests in random order and keep the seed in CI output.
• Bisect order-dependent failures with a fixed seed.

References

• pytest, Fixture scopes: https://docs.pytest.org/en/stable/how-to/fixtures.html#fixture-scopes
• pytest, monkeypatch: https://docs.pytest.org/en/stable/how-to/monkeypatch.html
• pytest-randomly: https://pypi.org/project/pytest-randomly/
• Python, types.MappingProxyType: https://docs.python.org/3/library/types.html#types.MappingProxyType