Goroutine count climbed forever: senders blocked on an unbuffered channel after the caller timed out

Our pricing service is a Go HTTP API that returns prices for product lists. Its memory graph had a slow, steady slope: from 90 MiB after a deploy to over 600 MiB after three days, at which point we usually deployed again and nobody looked closer. Then a holiday freeze stopped deploys for ten days, and pods started hitting their memory limit.

The go_goroutines metric told a clearer story than memory:

after deploy     ~210 goroutines
after 24 hours   ~48,000
after 72 hours   ~140,000

Goroutines were being created and never finishing.

Goroutine profile

curl -s 'localhost:6060/debug/pprof/goroutine?debug=1' | head -12

goroutine profile: total 139802
139588 @ 0x43d7ae 0x40a2f5 0x40a0b8 0x7c1e55 0x472a61
#	0x7c1e54	pricing/internal/discounts.(*Client).WithTimeout.func1+0x94	/src/internal/discounts/client.go:58

12 @ 0x43d7ae ... net/http.(*conn).serve ...

Almost all goroutines were parked at the same line, in a helper called WithTimeout. With debug=2, the state was visible:

goroutine 88123 [chan send, 1412 minutes]:
pricing/internal/discounts.(*Client).WithTimeout.func1()
	/src/internal/discounts/client.go:58 +0x94

chan send, for 1,412 minutes. They were trying to send on a channel.

The helper

// WithTimeout runs a discount lookup but returns early if it takes too long.
func (c *Client) WithTimeout(sku string, d time.Duration) (Discount, error) {
	result := make(chan Discount)       // unbuffered
	errCh := make(chan error)           // unbuffered

	go func() {
		disc, err := c.lookup(sku)      // slow call to the discount service
		if err != nil {
			errCh <- err
			return
		}
		result <- disc                   // line 58
	}()

	select {
	case disc := <-result:
		return disc, nil
	case err := <-errCh:
		return Discount{}, err
	case <-time.After(d):
		return Discount{}, ErrTimeout
	}
}

When the lookup finished before the timeout, the select received the result and everything ended. When the timeout fired first, WithTimeout returned, and nobody would ever receive from result or errCh. The goroutine finished its lookup, tried to send, and blocked forever on the unbuffered channel.

What we ruled out

• HTTP server goroutines. Connection-handling goroutines were stable at a few dozen.
• The time.After timer. Timers created by time.After are released when they fire; on older Go versions unfired timers could linger until expiry, but these all fired.
• The discount service hanging. Lookups completed within seconds; they were just slower than the 100 ms timeout during busy periods.

Root cause

The helper started a goroutine that sent its result on an unbuffered channel. A send on an unbuffered channel blocks until a receiver is ready. After the caller timed out and returned, no receiver would ever come, so each timed-out lookup left one goroutine blocked forever, along with everything it referenced. The discount service exceeded 100 ms on a few percent of requests at peak, which was enough to leak tens of thousands of goroutines a day.

Channel send semantics

Goroutines blocked forever are never collected: the garbage collector cannot free a goroutine that might, in principle, be resumed. Everything it references, including the lookup result and closures, stays in memory.

The deeper problem: work continued after nobody wanted it

Even with a buffer, the old helper kept the lookup running after the caller gave up, wasting a connection to the discount service and adding load exactly when it was slow. Go's answer is context.Context: the caller's cancellation should propagate into the work itself.

Who is affected

Any Go code that starts a goroutine to do work and then selects on its result channel with a timeout or a cancellation case: "run with timeout" helpers, first-response-wins fan-out across replicas, and wrappers around libraries that do not accept a context. The leak only appears when the timeout branch wins, so it grows with dependency latency and is invisible in tests that use fast mocks.

Environment

Minimal fix: a buffered channel

A buffer of one lets the goroutine send its single result and exit, whether or not anyone receives:

result := make(chan Discount, 1)
errCh := make(chan error, 1)

That stops the leak. The goroutine finishes, the channel becomes unreachable and both are collected.

Better fix: cancel the work with context

We replaced the helper with context-aware code, so the lookup is cancelled when the caller stops waiting:

func (c *Client) Discount(ctx context.Context, sku string) (Discount, error) {
	ctx, cancel := context.WithTimeout(ctx, 100*time.Millisecond)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/discounts/"+url.PathEscape(sku), nil)
	if err != nil {
		return Discount{}, err
	}
	resp, err := c.http.Do(req)
	if err != nil {
		if errors.Is(err, context.DeadlineExceeded) {
			return Discount{}, ErrTimeout
		}
		return Discount{}, err
	}
	defer resp.Body.Close()

	var d Discount
	if err := json.NewDecoder(resp.Body).Decode(&d); err != nil {
		return Discount{}, err
	}
	return d, nil
}

No extra goroutine at all: the HTTP client honours the context deadline and aborts the request. The handler passes r.Context(), so a client disconnecting also cancels the lookup.

Find similar patterns

We searched the codebase for goroutines that send on locally created unbuffered channels inside functions that can return early:

grep -rn 'make(chan [A-Za-z.]*)' --include=*.go internal/ | grep -v '_test.go'

Three more helpers had the same shape. All were converted to context-based code.

Alternatives and trade-offs

• golang.org/x/sync/singleflight if many callers request the same SKU concurrently, combined with context timeouts.
• Fallback values on timeout (for example "no discount") are a product decision; we return the base price and log the timeout, which matches previous behaviour.

How to test the fix

go.uber.org/goleak fails a test if goroutines are still running at the end:

func TestDiscountTimeoutDoesNotLeak(t *testing.T) {
	defer goleak.VerifyNone(t)

	slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		select {
		case <-time.After(2 * time.Second):
		case <-r.Context().Done():
		}
	}))
	defer slow.Close()

	c := NewClient(slow.URL, slow.Client())
	_, err := c.Discount(context.Background(), "sku-1")
	if !errors.Is(err, ErrTimeout) {
		t.Fatalf("want timeout, got %v", err)
	}
}

The same test against the old helper fails with a report naming the leaked goroutine and its stack.

Confirming in production

After deployment, go_goroutines stayed flat between 180 and 260 per pod for two weeks, including peak periods with many discount timeouts. Memory stayed under 110 MiB. Connections to the discount service dropped by about 4% at peak, because cancelled lookups no longer continued in the background.

Regression guard

• goleak.VerifyTestMain runs in every package's tests.
• An alert on go_goroutines increasing steadily over six hours.
• A code review guideline: functions that start goroutines must document how they end.

Security implications

A goroutine leak triggered by slow dependencies is an availability risk an attacker can amplify, for example by sending requests for SKUs that are slow to price. Context cancellation bounds the work per request and releases resources as soon as the client disconnects.

Prevention checklist

• Every goroutine you start needs a guaranteed way to finish.
• Use buffered channels of size one for single results that may not be received.
• Prefer context.Context cancellation over helper goroutines with timeouts.
• Pass request contexts into downstream calls.
• Run goleak in tests and alert on rising goroutine counts.

References

• Go blog, Go Concurrency Patterns: Context: https://go.dev/blog/context
• Go specification, Send statements: https://go.dev/ref/spec#Send_statements
• go.uber.org/goleak: https://pkg.go.dev/go.uber.org/goleak
• net/http/pprof: https://pkg.go.dev/net/http/pprof