OpenTelemetry Collector dropping spans under load: memory_limiter, batch and queue tuning

The first sign was in a postmortem. We pulled traces for the slowest checkout requests during a traffic spike and found that many of them were broken: a root span from the API gateway, then nothing from the payment service, then an orphaned database span with no parent. Outside peak hours, traces were complete. The tracing backend showed no ingestion errors. The data was being lost before it got there.

Our Collector gateway pods were logging two kinds of messages during peaks:

2024-12-12T18:04:11.532Z  warn  memorylimiter/memorylimiter.go:225  Memory usage is above soft limit. Refusing data.  {"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "cur_mem_mib": 372}
2024-12-12T18:04:12.907Z  error exporterhelper/queue_sender.go:92   Dropping data because sending_queue is full. Try increasing queue_size.  {"kind": "exporter", "data_type": "traces", "name": "otlp", "dropped_items": 1024}

And the services' SDKs were logging export failures back from the Collector:

rpc error: code = Unavailable desc = data refused due to high memory usage

Reproducing it

We reproduced it in staging with telemetrygen, the load generator that ships with the Collector contrib repository, against a copy of the production Collector configuration and a deliberately slowed backend:

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@v0.115.0
telemetrygen traces --otlp-endpoint otelcol-gateway.observability:4317 --otlp-insecure \
  --workers 8 --rate 1500 --duration 10m --child-spans 9

At 8 workers times 1,500 traces per second times 10 spans, the gateway started refusing data within two minutes, and dropped spans followed shortly after.

What we ruled out

• SDK-side drops. The SDK's batch span processor also drops when its own queue (default 2,048 spans) is full. We enabled the SDK's self-diagnostics on one service and saw no local drops; its export calls were failing on the Collector side.
• Sampling. We use parent-based sampling at a fixed ratio upstream. Broken traces with a sampled root and missing children cannot come from that, since the decision propagates.
• Backend rejection. The backend's ingestion metrics showed no refused or rate-limited writes. It was slow during compaction, not refusing.

Root cause

The Collector's internal metrics made it concrete. During the spike, receiver refusals and exporter enqueue failures rose together:

otelcol_receiver_refused_spans{receiver="otlp",transport="grpc"}       climbing
otelcol_exporter_enqueue_failed_spans{exporter="otlp"}                  climbing
otelcol_exporter_queue_size{exporter="otlp"} == otelcol_exporter_queue_capacity{exporter="otlp"}

Four configuration problems combined:

1. The backend slowed down for a few minutes during compaction, so export requests took longer.
2. The exporter's sending queue was at its default size with default consumers, and filled quickly. Once full, new batches were dropped.
3. The memory_limiter was set with a fixed limit_mib that did not match the pod's memory limit, and it was placed after the batch processor, so batches had already been built in memory before the limiter could refuse anything.
4. All three gateway replicas sat behind a Kubernetes ClusterIP Service. OTLP over gRPC uses long-lived HTTP/2 connections, so each SDK picked one pod at startup and stayed there. One pod held more than half of all connections and was the one hitting its limits.

What is happening under the hood

A traces pipeline in the Collector is a chain: receivers accept data, processors transform it in order, and exporters send it out. Back-pressure travels backwards through that chain:

• The exporter puts each batch into its sending queue. Consumer goroutines take batches off the queue and send them, retrying on failure according to retry_on_failure. If the queue is full, the batch is dropped and counted in otelcol_exporter_enqueue_failed_spans. queue_size counts batches (requests), not spans.
• The batch processor accumulates spans until send_batch_size or timeout is reached, so it holds data in memory.
• The memory_limiter checks heap usage every check_interval. Above the soft limit (limit minus spike_limit) it refuses incoming data, which the receiver turns into a retryable error to the client. Above the hard limit it also forces garbage collection.

Refusal is the healthy failure mode: the client SDK gets an error and can retry. Dropping from a full queue is the silent one. Our configuration produced both, and the ordering made the limiter less effective, because it is meant to sit first so it can refuse data before anything else allocates memory for it.

Environment

Minimal reproduction

The relevant part of the configuration we started with:

processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_mib: 400

exporters:
  otlp:
    endpoint: traces-backend.example.com:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [otlp]

Run it with a 512 MiB limit, point telemetrygen at it, and add latency to the backend (we used a toxiproxy in front of a test backend). Spans start dropping as soon as export latency rises.

Who is affected

Any Collector deployment that relies on default queue settings at high span rates, places memory_limiter anywhere but first, or sets memory limits that do not match the container. The gRPC connection pinning issue affects any gateway behind a plain Kubernetes Service with long-lived clients.