Building a slim OpenTelemetry Collector with ocb: why we stopped running the contrib image

For two years we ran the OpenTelemetry Collector contrib image everywhere: as an agent DaemonSet on every node and as a gateway deployment in each cluster. It was convenient, because every receiver, processor and exporter we might need was already inside. It was also a 250 MB-plus binary with hundreds of components, most of which we never configured, each with its own dependencies to patch and its own vulnerabilities to triage.

The OpenTelemetry Collector Builder, ocb, builds a Collector binary containing only the components you list. We switched to our own distribution. This post shows the workflow end to end and what changed for us.

Why a custom distribution

The Collector project publishes several distributions. The core distribution is small but lacks components many teams need, such as the Kubernetes attributes processor or vendor exporters. The contrib distribution includes nearly everything in the contrib repository. Building your own gives:

• A smaller binary and image: our build was about a quarter of the contrib image's size.
• A smaller attack surface: code that is not compiled in cannot be exploited, and vulnerability scans list only dependencies you actually ship.
• Explicit versions: every component is pinned in a manifest reviewed like code.
• Your own components: in-house processors or exporters can be compiled in.

The cost is a build pipeline you own and the work of upgrading versions yourself.

The manifest

ocb reads a YAML manifest describing the distribution and its components:

# builder-config.yaml
dist:
  name: otelcol-acme
  description: Acme OpenTelemetry Collector distribution
  output_path: ./build
  version: 0.135.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver v0.135.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.135.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.135.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.135.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.135.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.135.0
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.135.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.135.0
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.135.0

providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.41.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.41.0

Keep the versions aligned: components from core and contrib are released together, and mixing versions from different releases is the most common source of build errors. Core modules that have reached 1.0, such as the confmap providers, use a separate v1.x version line released alongside; take both from the same release notes. The version numbers here are illustrative of one release; use the release you are targeting.

Building it

ocb is itself a small binary, published with each Collector release. It generates Go source for the distribution and compiles it:

curl -fsSLo ocb \
  https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.135.0/ocb_0.135.0_linux_amd64
chmod +x ocb
./ocb --config builder-config.yaml
./build/otelcol-acme --version

The output directory contains the generated main.go, components.go, a go.mod and the binary. Commit the manifest; generated files can be regenerated in CI.

A reproducible container image

# syntax=docker/dockerfile:1
FROM golang:1.24 AS build
WORKDIR /src
ARG OCB_VERSION=0.135.0
RUN curl -fsSLo /usr/local/bin/ocb \
      "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv${OCB_VERSION}/ocb_${OCB_VERSION}_linux_amd64" \
 && chmod +x /usr/local/bin/ocb
COPY builder-config.yaml .
RUN CGO_ENABLED=0 ocb --config builder-config.yaml

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /src/build/otelcol-acme /otelcol-acme
ENTRYPOINT ["/otelcol-acme"]
CMD ["--config", "/etc/otelcol/config.yaml"]

CGO_ENABLED=0 produces a static binary that runs on a distroless static base image with no shell and no libc, which keeps the image small and the vulnerability list short. Pin the Go version to what the Collector release requires.

In CI, the pipeline builds the image, runs otelcol-acme validate --config against every configuration we deploy, scans the image and signs it.

Configuration does not change

Configuration files for a custom distribution are the same as for any Collector, as long as every component they reference is compiled in. A trimmed gateway configuration:

extensions:
  health_check: {}

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes: {}
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://otlp.observability.example.com

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]

If a configuration references a component that is not in the build, the Collector fails at startup with an "unknown type" error. Running validate in CI catches that before deployment.

Operating your own distribution

Upgrades

We upgrade monthly. A script bumps every version in the manifest to the new release, CI builds and validates, and the new image rolls to a staging cluster for a few days before production. Read the release notes for breaking changes in components you use; with fewer components, there are fewer notes to read.

Adding a component

Adding a component is a pull request to the manifest, with a short justification. That small friction is useful: it is the moment someone asks whether the component is maintained, what it depends on and who will own its configuration.

Debugging

The debug exporter and zpages extension are kept in the build for troubleshooting in staging; the production configuration does not enable them by default.

Results

The startup and size gains matter most for the DaemonSet on every node, where the image is pulled on every new node.

Common mistakes

Key takeaways

• ocb builds a Collector with only the components you list in a manifest.
• A custom distribution is smaller, has fewer vulnerabilities and pins every component version.
• Keep all component versions from the same release, and validate configurations in CI.
• Build static binaries for distroless images, and sign and scan the result.
• Treat the manifest as code: monthly upgrades and reviewed additions.

References

• OpenTelemetry, Building a custom collector: https://opentelemetry.io/docs/collector/custom-collector/
• OpenTelemetry Collector distributions: https://opentelemetry.io/docs/collector/distributions/
• OpenTelemetry Collector releases (ocb binaries): https://github.com/open-telemetry/opentelemetry-collector-releases/releases
• Collector configuration: https://opentelemetry.io/docs/collector/configuration/