nginx "worker_connections are not enough" at peak: proxied requests use two connections each

We launched a new feature with a marketing push, and at around 9,000 concurrent visitors our edge nginx, which proxies to the application servers, started returning errors. Some users got immediate connection resets; others saw 502 and 504 responses. The error log filled with:

2025/01/07 18:02:14 [alert] 1290#1290: 1024 worker_connections are not enough
2025/01/07 18:02:14 [alert] 1291#1291: 1024 worker_connections are not enough
2025/01/07 18:02:15 [crit] 1290#1290: *88213 accept4() failed (24: Too many open files)

Two messages: one about nginx's own connection limit, and one about the operating system's file descriptor limit. CPU and memory on the server were fine.

The configuration

user nginx;
worker_processes auto;            # 4 on this server

events {
    worker_connections 1024;
}

http {
    keepalive_timeout 65s;

    upstream app {
        server 10.20.1.11:8080;
        server 10.20.1.12:8080;
        server 10.20.1.13:8080;
    }

    server {
        listen 443 ssl;
        http2 on;
        location / {
            proxy_pass http://app;
        }
    }
}

worker_connections 1024 with four workers suggests 4,096 connections. The team had read that as "4,096 clients", plenty for the expected peak, since many browsers multiplex over HTTP/2.

Counting connections

At peak, on the server:

ss -s

TCP:   15201 (estab 8230, closed 6214, orphaned 12, timewait 6105)

ss -tn state established '( sport = :443 )' | wc -l     # client side
ss -tn state established '( dport = :8080 )' | wc -l    # upstream side

4071
4102

About 4,000 client connections and about 4,000 upstream connections. Each worker was holding around 2,000 connections in total, twice its limit of 1,024. And the process limit:

cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'
# Max open files            1024                 524288               files

Each worker could open only 1,024 file descriptors, and each connection is a file descriptor.

What we ruled out

• Application server overload. The app servers had headroom; their latency rose only because nginx was dropping and retrying connections.
• A DDoS. Traffic matched the marketing campaign; request patterns were normal.
• TLS overhead. CPU on nginx stayed below 40%.

Root cause

worker_connections limits all connections a worker holds, not just client connections. A request proxied to an upstream uses one connection to the client and one to the upstream, so a worker with 1,024 connections can serve at most about 512 concurrent proxied requests. On top of that, the workers' open file limit was 1,024, so even raising worker_connections alone would have hit "Too many open files".

What a worker connection is

Every socket nginx opens or accepts in a worker counts: client connections (including idle keep-alive connections waiting for the next request), connections to upstream servers, connections to resolvers, and so on. For a reverse proxy, a busy request typically holds two. Idle client keep-alive connections hold one each for up to keepalive_timeout.

So the realistic capacity is roughly:

max concurrent proxied requests ~= worker_processes x worker_connections / 2

minus whatever idle keep-alive connections are holding.

File descriptors

The kernel limits file descriptors per process (RLIMIT_NOFILE). Every connection uses one, plus open log files and cached files. nginx's worker_rlimit_nofile raises the limit for worker processes, up to the hard limit allowed by the service manager. If it is lower than worker_connections, the descriptor limit is the real cap.

Who is affected

Any nginx instance used as a reverse proxy or load balancer with default or copied events settings, especially when HTTP/1.1 clients hold idle keep-alive connections and upstream keep-alive is not enabled. Static file servers are less affected, because they need one connection per client, but they still hit the file descriptor limit if worker_rlimit_nofile is never raised.

Environment

Raise both limits, together

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    multi_accept on;
}

And make sure the service manager allows it; with systemd:

# systemctl edit nginx
[Service]
LimitNOFILE=65535

worker_rlimit_nofile should be comfortably above worker_connections, because each connection can need more than one descriptor briefly (for example when buffering responses to temporary files).

Reuse upstream connections

Without upstream keep-alive, nginx opens a new TCP connection to the app server for every request, which also fills the system with TIME_WAIT sockets (6,105 at peak). With keep-alive, a pool of connections is reused:

upstream app {
    server 10.20.1.11:8080;
    server 10.20.1.12:8080;
    server 10.20.1.13:8080;
    keepalive 64;              # idle connections kept per worker
    keepalive_timeout 60s;
}

location / {
    proxy_pass http://app;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
}

proxy_http_version 1.1 and clearing the Connection header are required, otherwise nginx sends Connection: close and the upstream closes the connection after each response. The app servers' keep-alive timeout must be longer than nginx's upstream keepalive_timeout, or the app may close a connection nginx is about to reuse.

Shorter client keep-alive

keepalive_timeout 65s kept idle client connections open for over a minute. Under load, thousands of idle browsers hold worker connections. We reduced it to 15 seconds, which covers the next request in a normal page load.

Alternatives and trade-offs

• More workers do not help if each is limited by the same settings; scaling connections per worker is cheap in nginx because it is event-driven.
• More nginx instances behind a load balancer add capacity and redundancy; we added a second one, but the configuration fix was what mattered.

How to test the fix

A load test with 12,000 concurrent connections against staging, holding connections open:

k6 run -e TARGET=https://staging.example.com --vus 12000 --duration 5m soak.js
grep -c 'worker_connections are not enough' /var/log/nginx/error.log   # 0
ss -tn state established '( dport = :8080 )' | wc -l                 # stable, a few hundred

Confirming in production

During the next campaign, peak client connections reached 11,000 with no errors in the nginx log. Upstream connections stayed around 250, thanks to keep-alive reuse, and TIME_WAIT sockets dropped from thousands to under a hundred.

Capacity after the change

With 4 workers and 16,384 connections each, the proxy can hold about 32,000 concurrent proxied requests before the connection limit matters, far above what the app servers can serve. The limit is now the application, which is where it should be.

Regression guard

• The nginx stub_status metrics are scraped; an alert fires when active connections per worker exceed 70% of worker_connections.
• A configuration test asserts that worker_rlimit_nofile is at least four times worker_connections and that every upstream block has keepalive.

Security implications

Connection limits are also a denial of service surface: slow clients holding connections (slowloris-style) can exhaust them. Higher limits raise the bar, and we added limit_conn per client IP, sensible client_header_timeout and client_body_timeout values, and a CDN in front for the public site.

Prevention checklist

• Remember that a proxied request uses two worker connections.
• Raise worker_rlimit_nofile and the service's LimitNOFILE along with worker_connections.
• Enable upstream keep-alive with HTTP/1.1 and an empty Connection header.
• Keep client keep-alive timeouts short on busy proxies.
• Monitor active connections against limits.

References

• nginx, worker_connections: https://nginx.org/en/docs/ngx_core_module.html#worker_connections
• nginx, worker_rlimit_nofile: https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile
• nginx, upstream keepalive: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
• nginx, stub_status module: https://nginx.org/en/docs/http/ngx_http_stub_status_module.html