Go service OOM-killed under a traffic burst: one goroutine per request to a slow dependency

Our event ingestion service receives order events over HTTP from several internal systems and forwards them to a partner's API. Normal traffic was a few hundred events per second. During a batch import, a source system sent about 8,000 events per second for ten minutes. The service's pods were OOM-killed, restarted, and killed again:

$ kubectl get pods -n ingest
NAME                      READY   STATUS      RESTARTS   AGE
ingest-api-7b9d6c-4xk2p   0/1     OOMKilled   6          40m

Memory limit was 1 GiB; normal usage was about 120 MiB.

What the handler did

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "bad request", http.StatusBadRequest)
		return
	}
	go h.forward(body)            // fire and forget
	w.WriteHeader(http.StatusAccepted)
}

func (h *Handler) forward(body []byte) {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	req, _ := http.NewRequestWithContext(ctx, http.MethodPost, h.partnerURL, bytes.NewReader(body))
	resp, err := h.client.Do(req)
	if err != nil {
		h.log.Error("forward failed", "err", err)
		return
	}
	resp.Body.Close()
}

The handler returned 202 Accepted immediately and forwarded in a new goroutine.

Profiling

We reproduced the burst against a staging partner mock that added two seconds of latency, and captured profiles with net/http/pprof:

curl -s localhost:6060/debug/pprof/goroutine?debug=1 | head -5

goroutine profile: total 61893
61811 @ 0x43d7ae 0x44f2c5 ... net/http.(*persistConn).roundTrip ...
#	0x7a1b2c	main.(*Handler).forward+0x1ac	/src/handler.go:31

Sixty thousand goroutines, nearly all waiting on the partner's HTTP response. The heap profile showed where the memory went:

go tool pprof -top http://localhost:6060/debug/pprof/heap

      flat  flat%   sum%        cum   cum%
  410.22MB 48.31% 48.31%   410.22MB 48.31%  io.ReadAll
  171.03MB 20.14% 68.45%   171.03MB 20.14%  bufio.NewWriterSize (net/http transport buffers)
  ...

Each goroutine kept its request body (around 6 KB) plus HTTP transport buffers and its own stack. Multiply by 60,000 and growing.

What we ruled out

• A memory leak in a library. Memory dropped back to normal after the burst when the process survived; it was accumulation, not a leak.
• Large payloads. Event bodies were small; it was the count.
• Go's GC settings. GOGC tuning would not help memory that is genuinely live.

Root cause

The handler started an unbounded number of goroutines, one per event, each waiting on a dependency that could process far fewer requests per second than arrived. Accepting work faster than it could be completed meant in-flight work, and its memory, grew without bound until the container was killed. The 202 response also told callers their events were safe, which was false when the process died.

Goroutines are cheap, not free

A goroutine starts with a small stack (a few kilobytes) that grows as needed, so creating thousands is fine. But whatever a goroutine holds while it waits, request bodies, buffers, connections, stays in memory. When arrival rate exceeds completion rate, Little's Law tells you the in-flight count grows over time:

in-flight = arrival rate x time in system
8,000 events/s x 2 s partner latency = 16,000 in flight, and rising as latency grows under load

As the partner slowed under our load, latency rose, and in-flight work grew faster.

Backpressure

The fix for any producer faster than its consumer is backpressure: when capacity is full, the system must slow down the producer, queue durably, or reject work. Accepting everything into memory is none of those.

Environment

Step 1: bound concurrency with a worker pool

We replaced fire-and-forget goroutines with a fixed number of workers reading from a bounded channel:

type Forwarder struct {
	jobs   chan []byte
	client *http.Client
	url    string
}

func NewForwarder(workers, queue int, client *http.Client, url string) *Forwarder {
	f := &Forwarder{jobs: make(chan []byte, queue), client: client, url: url}
	for i := 0; i < workers; i++ {
		go f.run()
	}
	return f
}

func (f *Forwarder) run() {
	for body := range f.jobs {
		f.send(body)
	}
}

// TryEnqueue returns false when the queue is full instead of blocking or growing.
func (f *Forwarder) TryEnqueue(body []byte) bool {
	select {
	case f.jobs <- body:
		return true
	default:
		return false
	}
}

With 64 workers per pod and a queue of 2,000, memory is bounded at roughly the size of 2,064 events plus fixed overhead, regardless of the arrival rate.

Step 2: push back on the caller

When the queue is full, the handler responds 503 with Retry-After, so callers retry later instead of assuming success:

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	body, err := io.ReadAll(io.LimitReader(r.Body, 64<<10))
	if err != nil {
		http.Error(w, "bad request", http.StatusBadRequest)
		return
	}
	if !h.fwd.TryEnqueue(body) {
		w.Header().Set("Retry-After", "2")
		http.Error(w, "busy", http.StatusServiceUnavailable)
		return
	}
	w.WriteHeader(http.StatusAccepted)
}

io.LimitReader also caps the size of a single body, which the old code did not.

Step 3: durable queue for real guarantees

A 202 should mean the event will be delivered. An in-memory queue loses events on restart. For this service, we moved to writing events to a durable queue (SQS in our case) in the handler, and a separate consumer forwards to the partner at a controlled rate. The worker pool pattern remains in the consumer.

Alternatives and trade-offs

• golang.org/x/sync/semaphore or errgroup with SetLimit bounds concurrency without a queue: callers block until a slot is free. Good for bounded fan-out inside a request; less suitable for shedding load at the edge.
• Rate limiting the partner calls (golang.org/x/time/rate) controls throughput but not memory unless combined with a bounded queue.

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(16)
for _, item := range items {
	g.Go(func() error { return process(ctx, item) })
}
err := g.Wait()

How to test the fix

A load test at 10,000 events per second against the staging partner mock with two seconds latency:

k6 run --rps 10000 --duration 5m ingest.js
kubectl top pods -n ingest
curl -s localhost:6060/debug/pprof/goroutine?debug=1 | head -1   # total stays near workers + baseline

Expected: memory stable, some 503 responses with Retry-After while the partner is slow, no restarts.

Confirming in production

The next batch import pushed 9,000 events per second. Pods stayed at 180 MiB, goroutines around 300 per pod, and the durable queue absorbed the burst, draining at the partner's rate over 25 minutes. No events were lost and no pods restarted.

Regression guard

• A metric for goroutine count (go_goroutines) per pod with an alert above 5,000.
• A lint rule flags go statements inside HTTP handlers unless the code uses the shared bounded pool.
• Load tests with a slow dependency mock run before releases.

Security implications

Unbounded goroutines per request are a denial of service vulnerability: any client able to send requests quickly could exhaust memory. Bounded queues, body size limits and 503 responses with retry hints make the service degrade predictably instead of crashing.

Prevention checklist

• Never start a goroutine per request for work that waits on a slower system.
• Bound concurrency with a worker pool, semaphore or errgroup.SetLimit.
• Reject or queue durably when capacity is full; respond 503 with Retry-After.
• Limit request body sizes.
• Watch goroutine counts and test with slow dependencies.

References

• Go blog, Pipelines and cancellation: https://go.dev/blog/pipelines
• golang.org/x/sync/errgroup: https://pkg.go.dev/golang.org/x/sync/errgroup
• golang.org/x/sync/semaphore: https://pkg.go.dev/golang.org/x/sync/semaphore
• net/http/pprof: https://pkg.go.dev/net/http/pprof