A shallow copy let one request rewrite the defaults for every request after it

The ticket said a customer on the Basic plan had been shown the upload limit of an Enterprise customer: "Max file size 2 GB" where it should have said 100 MB. Support could not reproduce it. Ten minutes later a second report arrived from a different Basic customer, and this time the screenshot showed a retention period that only one specific tenant had configured.

The values were coming from our settings service, a small Node.js process that merges global defaults with per-tenant overrides and returns the effective config. Restarting it made the problem vanish, which is the first clue that something in memory was being changed when it should only have been read.

We added a log line that printed the defaults object on every request, hashed so it was easy to compare:

12:04:11 settings tenant=basic-0412 defaults_hash=9f1c2a effective.upload.maxMb=100
12:04:13 settings tenant=ent-0077   defaults_hash=9f1c2a effective.upload.maxMb=2048
12:04:15 settings tenant=basic-0413 defaults_hash=51be07 effective.upload.maxMb=2048

The defaults hash changed right after the Enterprise tenant was served. The global defaults, loaded once at boot and meant to be read only, had been rewritten by a request.

Narrowing it down

The merge function looked harmless:

// settings.js
const DEFAULTS = loadDefaults(); // { upload: { maxMb: 100 }, retention: { days: 30 }, ... }

function effectiveSettings(tenant) {
  const settings = { ...DEFAULTS };          // "copy" the defaults
  for (const [section, values] of Object.entries(tenant.overrides)) {
    Object.assign(settings[section], values); // apply the tenant's overrides
  }
  return settings;
}

The author's intent was clear: copy the defaults, then customise the copy. A one-line check in the REPL showed what really happened:

const a = { upload: { maxMb: 100 } };
const b = { ...a };
b.upload.maxMb = 2048;
console.log(a.upload.maxMb); // 2048
console.log(a.upload === b.upload); // true

Spread made a new outer object, but upload in the new object pointed at the same inner object as DEFAULTS.upload. Object.assign(settings[section], values) wrote straight into the shared defaults. Every tenant after an Enterprise tenant inherited whatever that tenant had overridden, until the next restart.

Values, references and what "copy" means

In JavaScript, Python and Go, a variable holding an object, dict, list, map or slice holds a reference to data that lives somewhere else. Copying the variable copies the reference. Most built-in copy helpers go one level deeper: they make a new container, then copy each member's reference into it. That is a shallow copy. Anything nested is still shared.

Python behaves the same way as our Node code:

DEFAULTS = {"upload": {"max_mb": 100}}

settings = DEFAULTS.copy()
settings["upload"]["max_mb"] = 2048
print(DEFAULTS["upload"]["max_mb"])  # 2048

Go is a little different because structs are values. Assigning a struct copies its fields, which feels like a deep copy until one of those fields is a map, slice or pointer:

type Settings struct {
    Limits map[string]int
    Tags   []string
}

func main() {
    defaults := Settings{Limits: map[string]int{"maxMb": 100}, Tags: []string{"basic"}}
    s := defaults          // copies the struct header only
    s.Limits["maxMb"] = 2048
    s.Tags[0] = "enterprise"
    fmt.Println(defaults.Limits["maxMb"], defaults.Tags[0]) // 2048 enterprise
}

The map header and slice header were copied, but both still point at the same backing data.

Why it only showed up sometimes

Three conditions had to line up:

1. A tenant with an override in a nested section had to be served. Most tenants only override top-level flags, which the spread did copy safely.
2. A tenant without that override had to be served afterwards by the same process.
3. No restart in between. We deploy several times a day, so the corruption usually lived for minutes to hours before a deploy cleared it.

It never appeared in tests because every test built a fresh DEFAULTS object, so there was nothing shared to corrupt.

Environment

We fixed it in three layers: stop mutating, make mutation impossible, and keep the fast path cheap.

1. Build a new object instead of writing into a shared one

The merge does not need to mutate anything. Building each section fresh removes the bug without a deep clone of the whole defaults tree:

function effectiveSettings(tenant) {
  const settings = {};
  for (const [section, defaults] of Object.entries(DEFAULTS)) {
    settings[section] = { ...defaults, ...(tenant.overrides[section] ?? {}) };
  }
  return settings;
}

Each section is now a new object made from the defaults plus the overrides. Our sections are one level deep, so this is enough. If a section ever gains nested objects, this pattern would have the same problem one level down, which is why the next two steps exist.

2. Freeze the defaults so a future mistake throws

Object.freeze is shallow too, so we wrote a small recursive version and applied it at boot:

function deepFreeze(obj) {
  for (const value of Object.values(obj)) {
    if (value && typeof value === "object" && !Object.isFrozen(value)) {
      deepFreeze(value);
    }
  }
  return Object.freeze(obj);
}

const DEFAULTS = deepFreeze(loadDefaults());

The service runs as an ES module, which is strict mode, so writing to a frozen object throws a TypeError instead of failing silently. Any future code that tries to mutate the defaults fails loudly in tests rather than quietly in production.

3. Use a real deep copy where the shape is unknown

For one admin endpoint that lets staff experiment with arbitrary nested settings, we use structuredClone, available as a global since Node.js 17:

const draft = structuredClone(DEFAULTS);
draft.upload.limits.byType.video = 4096; // safe, nothing shared

structuredClone handles nested objects, arrays, Map, Set and Date, but throws on functions and class instances lose their prototype. That is fine for plain config data and a good reason not to put behaviour in config objects.

The equivalent fixes in the other two languages:

import copy
from types import MappingProxyType

settings = copy.deepcopy(DEFAULTS)          # independent copy
READ_ONLY = MappingProxyType(DEFAULTS)      # read-only view (top level only)

func (s Settings) Clone() Settings {
    out := Settings{
        Limits: maps.Clone(s.Limits),           // Go 1.21+
        Tags:   append([]string(nil), s.Tags...),
    }
    return out
}

In Go we added an explicit Clone() method next to the type, because there is no generic deep copy in the standard library and reflection-based helpers are slow and surprising with unexported fields.

1. A test that fails on the old code. It serves an Enterprise tenant and then a Basic tenant with the same module state, which is exactly what production did:

import { test } from "node:test";
import assert from "node:assert/strict";
import { effectiveSettings, DEFAULTS } from "./settings.js";

test("tenant overrides never leak into defaults", () => {
  const before = JSON.stringify(DEFAULTS);
  effectiveSettings({ overrides: { upload: { maxMb: 2048 } } });
  const basic = effectiveSettings({ overrides: {} });

  assert.equal(basic.upload.maxMb, 100);
  assert.equal(JSON.stringify(DEFAULTS), before);
});

test("defaults are frozen", () => {
  assert.throws(() => { DEFAULTS.upload.maxMb = 1; }, TypeError);
});

On the old code the first test failed with 2048 !== 100. On the new code both pass.

2. Production check. We kept the defaults_hash log line for a week. The hash stayed identical across every request and every replica between deploys.

3. Search for siblings. We grepped the codebase for the same shape: a spread or Object.assign({}, x) followed by writes into a nested property. Two more spots turned up, a feature-flag merger and a test fixture builder. Neither had caused a visible bug yet; both now build new objects.

4. Security implications. This bug leaked configuration between tenants. Nothing secret was in those sections, but if a tenant override had held an API key or a webhook URL, one customer's secret could have been served to another. We confirmed no secrets live in the merged config, and added a review rule that per-tenant credentials never go through the shared defaults path.

Takeaways

• Spread, Object.assign, dict.copy() and Go struct assignment copy one level. Nested data is shared.
• Prefer building new objects over mutating copies. It is clearer and avoids deep clones on hot paths.
• Freeze shared defaults so mistakes throw in tests instead of corrupting production quietly.
• Use structuredClone, copy.deepcopy or an explicit Clone() method when you truly need an independent copy of unknown depth.

References: structuredClone on MDN, Python copy module, Go maps package.