Blue-green deploy broke Blue: a column rename in Green's migration and the expand/contract fix

We deploy our customer-accounts service blue-green behind an AWS Application Load Balancer: two Auto Scaling groups, two target groups, and a listener rule that forwards 100 percent of traffic to one colour. A release goes to the idle colour, gets smoke-tested through a private header route, then traffic is shifted by changing the listener weights. Rollback is supposed to be one API call.

On release 5.14, the pipeline deployed Green, ran its migrations, and began smoke tests. Blue, still serving every customer, immediately started failing on the profile endpoints:

django.db.utils.ProgrammingError: column customers_customer.phone does not exist
LINE 1: ...stomer"."email", "customers_customer"."phone", "custo...

The 5xx rate hit 31 percent. The on-call engineer did the obvious thing and pressed the rollback button, which shifted weight to... Blue, which was already receiving 100 percent. Shifting forward to Green stopped the errors after six minutes, which meant we had shipped a release nobody had finished testing, under pressure, with no way back.

Reproducing it

The release notes had one schema change: rename Customer.phone to Customer.phone_e164 to make the format explicit. Django had generated a single migration:

python manage.py sqlmigrate customers 0042

BEGIN;
--
-- Rename field phone on customer to phone_e164
--
ALTER TABLE "customers_customer" RENAME COLUMN "phone" TO "phone_e164";
COMMIT;

The instant that statement committed, every query from Blue that selected phone failed. We reproduced it in staging by running Blue's code against a database migrated to Green's schema; the failure was immediate and total for any view touching customers.

What we ruled out

A bad load balancer shift. ALB access logs showed every failing request served by Blue's target group, with no weight changes before the migration.

Connection or replica issues. All failing queries hit the primary, and the error was a schema error, not a timeout.

A failed or partial migration. django_migrations recorded 0042 as applied, and the column existed under its new name.

Root cause

Blue-green deployments duplicate the application, not the database. Both colours point at the same PostgreSQL primary, so a migration applied for Green changes the schema under Blue too. The deployment strategy assumed every migration was backward compatible with the previous release, and a rename is the textbook example of one that is not: it is atomic for the database, but no single version of the code can work with both the old and the new name. The same property also destroyed rollback, because Blue's code could not run against the migrated schema either.

What is happening under the hood

For blue-green or rolling deploys to be safe, each schema state must be readable and writable by two consecutive application versions: the one currently live and the one being introduced. That constraint is what the expand and contract pattern (also called parallel change) satisfies:

1. Expand. Add new structures alongside old ones. Nothing is removed or renamed. Old code ignores the additions.
2. Migrate. Deploy code that writes to both old and new structures, backfill existing rows, then switch reads to the new structure.
3. Contract. Once no running or rollback-eligible version reads the old structure, remove it.

Each step is its own release, and each release is compatible with the one before it, so either colour can serve traffic at every point and rollback stays a weight change.

Operations that break the previous version and need this treatment include renaming or dropping columns and tables, changing a column's type, adding NOT NULL without a default to a column old code does not write, and narrowing constraints. Adding a nullable column, adding a table or adding an index concurrently are already backward compatible.

Environment

Minimal reproduction

docker run -d --name pg -e POSTGRES_PASSWORD=devpass -p 5432:5432 postgres:16
psql "postgresql://postgres:devpass@localhost:5432/postgres" <<'SQL'
CREATE TABLE customer (id bigserial PRIMARY KEY, email text NOT NULL, phone text);
INSERT INTO customer (email, phone) VALUES ('a@example.com', '+15550100');
-- Blue's query works
SELECT id, email, phone FROM customer;
-- Green's migration
ALTER TABLE customer RENAME COLUMN phone TO phone_e164;
-- Blue's query now fails
SELECT id, email, phone FROM customer;
SQL

The last statement fails with ERROR: column "phone" does not exist.