FATAL: sorry, too many clients already: autoscaled pods exhausted Postgres connections

On a Monday morning a marketing email went out, traffic tripled, and our orders API started returning 500s within four minutes. The Kubernetes HPA had done its job and scaled the API from 8 to 30 pods. The new pods crashed on startup and the old ones began failing health checks. The application logs were full of:

error: sorry, too many clients already
    at Parser.parseErrorMessage (node_modules/pg-protocol/dist/parser.js:283:98)

The Postgres log said the same from the server side, along with a second message we had not seen before:

FATAL:  sorry, too many clients already
FATAL:  remaining connection slots are reserved for roles with the SUPERUSER attribute

Our on-call engineer could not even open psql as the application role to look around.

Reproducing it

Simple arithmetic reproduced it. Each pod ran a pg.Pool with max: 20. Postgres had max_connections = 300 and superuser_reserved_connections = 3.

 8 pods x 20 = 160 connections   fine
15 pods x 20 = 300 connections   at the limit
30 pods x 20 = 600 connections   refused

Plus a migration job, a cron worker and two monitoring agents. On staging we started 20 replicas of the API with a load test and hit the error in under a minute.

What we ruled out

• Connection leak. pg_stat_activity showed connections evenly spread across pods, about 20 each, mostly idle. Nothing was holding connections forever.
• Slow queries. p95 query time was 4 ms before the spike. The pool was not exhausted because queries were slow; it was simply sized per pod with no global view.
• Database CPU. The primary was at 35 percent CPU when it started refusing connections. It had capacity; it did not have slots.

Root cause

The total number of connections is pool size times pod count, and only one of those numbers was under our control at deploy time. Autoscaling multiplied the second one. Most of those 600 connections would have been idle at any moment, but Postgres reserves a backend process per connection whether it is busy or not, and refuses new ones past max_connections.

What is happening under the hood

PostgreSQL uses a process per connection. Every client connection gets its own backend process with its own memory, catalog caches and snapshot bookkeeping. That is why max_connections is a hard limit, and why setting it to several thousand is a bad idea: each backend uses several MB even when idle, and with many active backends, snapshot computation and lock contention grow with the connection count.

When the limit is reached, Postgres keeps superuser_reserved_connections slots for superusers, and since PostgreSQL 16 also reserved_connections for roles granted pg_use_reserved_connections. Everyone else gets the too many clients error, or the remaining connection slots are reserved error when only reserved slots are left.

The fix is to decouple client connections from server connections. A pooler such as PgBouncer accepts thousands of cheap client connections and multiplexes them onto a small number of real Postgres backends.

PgBouncer has three pool modes:

For a stateless API, transaction mode is the sweet spot. A pod can keep 20 client connections open, and they only occupy a server connection while a transaction is running, typically a few milliseconds.

Environment

Minimal reproduction

A quick way to see the limit locally:

docker run -d --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 postgres:17 \
  -c max_connections=20

# open 25 idle connections
for i in $(seq 1 25); do
  psql "postgresql://postgres:secret@localhost:5432/postgres" -c "SELECT pg_sleep(30)" &
done
wait

The last few psql calls print FATAL: sorry, too many clients already.

Who is affected

Any process-per-connection database with per-instance pools and horizontal autoscaling: Kubernetes deployments, serverless functions (each instance holds its own connections), and job runners that fan out workers. MySQL hits the same wall with Too many connections, although its thread-per-connection model makes each idle connection cheaper.

The fix

1. Put PgBouncer between the API and Postgres in transaction mode. The core of our pgbouncer.ini:

[databases]
orders = host=10.20.0.15 port=5432 dbname=orders

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
max_client_conn = 5000
default_pool_size = 40
reserve_pool_size = 10
reserve_pool_timeout = 3
max_db_connections = 100
server_idle_timeout = 300
max_prepared_statements = 200

With two PgBouncer replicas, the database sees at most 200 server connections from the API, however many pods exist. max_client_conn is the cap on the client side and is cheap to raise, since PgBouncer is a single-threaded event loop that handles thousands of idle sockets.

max_prepared_statements matters for transaction mode. Since PgBouncer 1.21, it tracks protocol-level named prepared statements and re-prepares them on whichever server connection a transaction lands on. Without it, drivers that use named prepared statements fail with prepared statement "..." does not exist.

2. Point the app at the pooler and keep app pools modest.

- DATABASE_URL=postgresql://orders_api@10.20.0.15:5432/orders
+ DATABASE_URL=postgresql://orders_api@pgbouncer.example.com:6432/orders

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

connectionTimeoutMillis turns an overloaded pool into a fast error instead of a hung request.

3. Remove session state from the request path. Transaction pooling means two consecutive transactions from one client may run on different server connections. We audited the code for features that rely on a session:

• SET search_path and SET statement_timeout per request: moved to SET LOCAL inside the transaction, or to role-level defaults with ALTER ROLE ... SET.
• Session advisory locks (pg_advisory_lock): replaced by transaction-scoped pg_advisory_xact_lock.
• LISTEN: moved to a separate direct connection that bypasses PgBouncer.

4. Keep an escape hatch. We created a dedicated admin role and set reserved_connections = 5 with GRANT pg_use_reserved_connections TO ops_admin, so on-call can always connect directly, even if something saturates the server again.

ALTER SYSTEM SET reserved_connections = 5;
GRANT pg_use_reserved_connections TO ops_admin;

reserved_connections needs a restart, which we did during the PgBouncer rollout window.

Alternatives and trade-offs

• Raise max_connections. It buys a little headroom and costs memory and throughput, and the next scale-out hits the new limit anyway. We kept it at 300.
• Cap the HPA. Limiting pods to 15 avoids the error but caps capacity. It is a reasonable stopgap while a pooler is being deployed.
• Managed proxies. RDS Proxy or the pooler built into some managed Postgres offerings provide the same multiplexing with less to operate, with their own pinning rules for session state.
• Session mode. Safest compatibility, but each client still pins a server connection, so it barely helps with many idle clients.

How to test the fix

On staging we scaled the API to 40 pods and ran the same load test:

kubectl -n orders scale deploy/orders-api --replicas=40
k6 run --vus 800 --duration 10m load/orders.js

During the run we watched PgBouncer's admin console:

psql -h pgbouncer.db.svc -p 6432 -U pgb_admin pgbouncer -c "SHOW POOLS;"

cl_active reached about 400 client connections (40 pods with 10 each) while sv_active plus sv_idle stayed under 80 server connections, and cl_waiting stayed at zero except for brief bursts under 50 ms.

Confirming in production

After rollout, the connection count on Postgres became flat regardless of pod count:

SELECT usename, state, count(*)
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY usename, state
ORDER BY count(*) DESC;

Three weeks later another campaign scaled the API to 38 pods. Server connections peaked at 92, no connection errors were logged, and API p95 stayed at 41 ms. We also checked SHOW STATS; in PgBouncer to confirm average transaction time (avg_xact_time) was in the low milliseconds, which is what makes transaction pooling effective.

Regression guard

• Alert when pg_stat_activity client backends exceed 80 percent of max_connections.
• Alert on PgBouncer cl_waiting above zero for more than 30 seconds, which means pools are too small or transactions too long.
• A CI check rejects any service manifest that connects to the database host directly on port 5432 instead of through the pooler.
• New services must document max pool size x max replicas in their deployment README.

Security implications

Connection exhaustion is a denial-of-service vector: any component that can open connections, including a buggy job, can lock everyone out, including operators. The reserved-connections role restores operator access. PgBouncer adds a new network hop that holds credentials, so we use scram-sha-256, keep userlist.txt in a Kubernetes secret, restrict the admin console to admin_users = pgb_admin, and keep TLS on the server side with server_tls_sslmode = verify-full. Moving to transaction pooling also forced us to remove per-session SET calls, which reduced the risk of a request inheriting another request's settings.

Prevention checklist

• Budget connections globally: pool size times maximum replicas, plus jobs and tooling, must stay below max_connections.
• Put a pooler in front of Postgres before enabling horizontal autoscaling.
• Prefer transaction pooling for stateless services, and remove session state from the request path.
• Reserve connection slots for operators.
• Alert on connection usage, not just CPU.

References

• PostgreSQL 17 docs, Connections and Authentication: https://www.postgresql.org/docs/17/runtime-config-connection.html
• PostgreSQL 17 docs, Advisory Locks: https://www.postgresql.org/docs/17/explicit-locking.html
• PgBouncer configuration: https://www.pgbouncer.org/config.html
• PgBouncer usage and admin console: https://www.pgbouncer.org/usage.html
• node-postgres Pool: https://node-postgres.com/apis/pool