pandas cleaned values never saved: SettingWithCopyWarning and chained assignment

The analytics team flagged it in a weekly review: the customer dimension table, which our cleaning job is supposed to normalise, still had about 3,100 rows with a null country and a few hundred emails in mixed case. The job log said both steps ran. No errors, no failed tasks.

The cleaning code looked reasonable at a glance:

df[df["country"].isna()]["country"] = "unknown"

active = df[df["status"] == "active"]
active["email"] = active["email"].str.strip().str.lower()

df["signup_source"].fillna("organic", inplace=True)

Reproducing it

Running the job in a notebook with the production sample printed three warnings, which never appeared in the scheduled job's logs:

SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

The scheduled job did not show them because its entrypoint, adapted from an old notebook, started with:

import warnings
warnings.simplefilter("ignore")
pd.options.mode.chained_assignment = None

What we ruled out

• The write to the warehouse. We compared the DataFrame just before to_parquet with the output file; they matched. The nulls were already there in memory.
• Upstream data changes. The share of null countries in raw data was stable, so the cleaning step was not doing its job, rather than receiving new kinds of nulls.
• Dtype issues. country was object dtype and accepted the string "unknown" fine when assigned correctly.

Root cause

Each line wrote to a temporary object, not to df:

• df[mask] with a boolean mask always returns a new DataFrame. ["country"] = ... then sets the column on that temporary copy, which is discarded at the end of the statement.
• active = df[mask] is also a copy. Lowercasing active["email"] changes active only; the code later wrote df, not active.
• df["signup_source"].fillna(..., inplace=True) happened to work in our pandas version, but only by accident of internal caching, and pandas warns that it will stop working.

pandas told us about the first two with SettingWithCopyWarning. We had switched it off.

What is happening under the hood

In pandas 2.x without Copy-on-Write, whether an indexing result is a view of the parent's data or a copy depends on the operation and the internal block layout. Selecting a single column often returns a view; boolean masks and fancy indexing return copies. Because the rule is hard to predict, pandas raises SettingWithCopyWarning when you assign into an object that was derived from another one by indexing. The warning means "this assignment may not reach the original".

Chained assignment is the pattern obj[a][b] = value. Python evaluates it as two separate operations: tmp = obj.__getitem__(a), then tmp.__setitem__(b, value). pandas never sees the whole expression, so it cannot route the write back to obj. A single .loc[rows, cols] = value is one __setitem__ call on the original object, which is why the warning text recommends it.

Copy-on-Write (CoW) makes this predictable. With CoW enabled, any object derived from another behaves as a copy: modifying it never changes the parent, and pandas only physically copies data when one side is actually written. Under CoW, chained assignment never updates the original, and pandas raises a ChainedAssignmentError warning to say so. CoW is opt-in in pandas 2.x through pd.options.mode.copy_on_write = True, and pandas 3.0 makes it the default and only behaviour. In 2.2 there is also a "warn" mode that reports places where results would change.

Environment

We are on 2.2 because two internal libraries still pin pandas<3. The fix below works the same on both.

Minimal reproduction

import pandas as pd

df = pd.DataFrame({
    "country": ["GB", None, "SE"],
    "status": ["active", "active", "closed"],
    "email": [" A@Example.com", "b@example.com", "C@example.com"],
})

df[df["country"].isna()]["country"] = "unknown"
active = df[df["status"] == "active"]
active["email"] = active["email"].str.strip().str.lower()

print(df)

  country  status            email
0      GB  active   A@Example.com
1    None  active   b@example.com
2      SE  closed   C@example.com

Neither change reached df.

Who is affected

Anyone writing pandas code in the style df[filter][column] = value, mutating a filtered subset and expecting the parent to change, or calling inplace=True methods on a selected column. It is especially common in code migrated from notebooks, where warnings are often silenced to keep the output tidy.

The fix

1. Use one .loc call per assignment.

-df[df["country"].isna()]["country"] = "unknown"
+df.loc[df["country"].isna(), "country"] = "unknown"

2. Assign results back to the parent instead of mutating a subset.

-active = df[df["status"] == "active"]
-active["email"] = active["email"].str.strip().str.lower()
+is_active = df["status"] == "active"
+df.loc[is_active, "email"] = df.loc[is_active, "email"].str.strip().str.lower()

If you actually want an independent subset to work on, say so with .copy(); the warning disappears and the intent is clear to the next reader:

active = df[df["status"] == "active"].copy()

3. Replace inplace=True on a column with reassignment.

-df["signup_source"].fillna("organic", inplace=True)
+df["signup_source"] = df["signup_source"].fillna("organic")

4. Turn on Copy-on-Write and stop silencing warnings. At the job entrypoint:

import pandas as pd

pd.options.mode.copy_on_write = True

and we deleted the blanket warnings.simplefilter("ignore") and the chained_assignment = None line.

Alternatives and trade-offs

• pd.options.mode.chained_assignment = "raise" turns SettingWithCopyWarning into a SettingWithCopyError. Useful in tests for codebases not ready for CoW, but it has false positives and does nothing once CoW is on.
• DataFrame.assign and method chains. Writing transformations as df.assign(country=lambda d: d["country"].fillna("unknown")) avoids mutation altogether. We use this style for new code; rewriting the whole job was not worth it.
• Staying on legacy mode until pandas 3. Possible, but the bugs would continue, and the 3.0 upgrade would then change behaviour in the same lines. Enabling CoW now means the upgrade is a no-op for this job.

How to test the fix

A test that fails loudly on any chained assignment warning:

import warnings
import pandas as pd
import pytest
from cleaning.customers import clean_customers

pd.options.mode.copy_on_write = True


def test_clean_customers_fills_and_normalises():
    raw = pd.DataFrame({
        "country": ["GB", None],
        "status": ["active", "active"],
        "email": [" A@Example.com", "b@example.com"],
        "signup_source": [None, "ads"],
    })
    with warnings.catch_warnings():
        warnings.simplefilter("error")
        out = clean_customers(raw)

    assert out["country"].tolist() == ["GB", "unknown"]
    assert out["email"].tolist() == ["a@example.com", "b@example.com"]
    assert out["signup_source"].tolist() == ["organic", "ads"]

We also run the whole suite with warnings as errors for pandas categories:

pytest -W error::pandas.errors.SettingWithCopyWarning \
       -W error::pandas.errors.ChainedAssignmentError -q

Confirming in production

After the next scheduled run we checked the output table directly:

SELECT
  COUNT(*) FILTER (WHERE country IS NULL)            AS null_country,
  COUNT(*) FILTER (WHERE email <> LOWER(TRIM(email))) AS unnormalised_email
FROM analytics.dim_customer;

 null_country | unnormalised_email
--------------+--------------------
            0 |                  0

Row counts matched the previous run within normal daily growth, and the downstream country revenue report lost its "null" bucket. We also scanned the job logs for ChainedAssignmentError across all Dagster assets for a week; none appeared.

Regression guard

• copy_on_write = True is set in the shared job bootstrap module, so every pipeline gets it, and pandas 3.0 will not change their behaviour.
• The test suite runs with pandas warnings as errors.
• A Ruff configuration enables the PD (pandas-vet) rules, including PD002, which flags inplace=True.
• Code review guidelines ban global warnings.simplefilter("ignore") in jobs.

Security implications

None beyond data quality. Unnormalised emails did affect one downstream behaviour: the marketing suppression list matched on exact email, so a few customers who had unsubscribed under a different case variant could have received a newsletter. We re-ran suppression after the fix. No data left the warehouse unexpectedly.

Prevention checklist

• Write every assignment as one df.loc[rows, cols] = value.
• Call .copy() when you want an independent subset.
• Replace col.method(..., inplace=True) with reassignment.
• Enable pd.options.mode.copy_on_write = True on pandas 2.x.
• Never silence pandas warnings globally.

References

• pandas user guide, Copy-on-Write: https://pandas.pydata.org/docs/user_guide/copy_on_write.html
• pandas indexing, returning a view versus a copy: https://pandas.pydata.org/docs/user_guide/indexing.html
• pandas.errors.SettingWithCopyWarning: https://pandas.pydata.org/docs/reference/api/pandas.errors.SettingWithCopyWarning.html
• pandas.errors.ChainedAssignmentError: https://pandas.pydata.org/docs/reference/api/pandas.errors.ChainedAssignmentError.html
• pandas 2.2.0 release notes: https://pandas.pydata.org/docs/whatsnew/v2.2.0.html