Prometheus ServiceMonitor not discovered: target missing because of a label selector mismatch

Our team added a new orders-api service with a /metrics endpoint, a ServiceMonitor, and a Grafana dashboard. The dashboard stayed empty. In Prometheus, the query up{job="orders-api"} returned no series at all, not even 0, which means Prometheus was not trying to scrape it.

Empty query result

The ServiceMonitor existed and looked correct:

kubectl -n orders get servicemonitor orders-api

NAME         AGE
orders-api   2d

There was no event, no error status on the object, and nothing obvious in Prometheus logs. Every other service in the cluster was being scraped.

Reproducing it

Opening the Prometheus UI at /service-discovery confirmed it: there was no serviceMonitor/orders/orders-api/0 entry at all. Other ServiceMonitors appeared with names in that format. So the problem was earlier than scraping: the operator never turned our ServiceMonitor into scrape configuration.

What we ruled out

The application endpoint. A port-forward to a Pod and curl localhost:8080/metrics returned the usual Prometheus text format.

RBAC. If Prometheus cannot list Endpoints or EndpointSlices in a namespace, it logs errors like failed to list *v1.EndpointSlice ... is forbidden. The chart's ClusterRole already covered all namespaces, and the logs were clean.

NetworkPolicy. A blocked scrape would still create a target, shown as down with a timeout. We had no target, so this was not the network.

Root cause

The operator selects ServiceMonitors using fields on the Prometheus custom resource. We looked at ours:

kubectl -n monitoring get prometheus -o yaml | yq '.items[0].spec | {"serviceMonitorSelector": .serviceMonitorSelector, "serviceMonitorNamespaceSelector": .serviceMonitorNamespaceSelector}'

serviceMonitorSelector:
  matchLabels:
    release: monitoring
serviceMonitorNamespaceSelector: {}

The namespace selector was open, but the object selector required the label release: monitoring, the Helm release name of our kube-prometheus-stack installation. Other teams' ServiceMonitors had it because they had copied an older example. Ours had only app.kubernetes.io/name: orders-api. The operator silently ignored it.

After adding the label, the scrape job appeared on the service discovery page, but with zero targets, because of a second mismatch: our ServiceMonitor referenced port: metrics, while the Service named its port http-metrics.

What is happening under the hood

Prometheus Operator watches Prometheus, ServiceMonitor, PodMonitor and related resources. For each Prometheus instance it builds the list of ServiceMonitors that match both:

• serviceMonitorNamespaceSelector, applied to namespace labels. Empty {} means all namespaces; a missing (null) field means only the Prometheus object's own namespace.
• serviceMonitorSelector, applied to the ServiceMonitor's own labels. Empty {} means all; null means none.

Matching ServiceMonitors are rendered into kubernetes_sd_configs scrape jobs and written to a Secret that the config-reloader sidecar loads into Prometheus. Non-matching ones are not an error. The operator has no reason to complain about a resource that is simply not addressed to it, which is why there is no event.

kube-prometheus-stack sets the selector from the chart value prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues. With the default true, an unset selector becomes matchLabels: {release: <release name>}, so only ServiceMonitors carrying that label are picked up.

Inside a matched ServiceMonitor there are two more joins:

1. spec.selector must match labels on the Service, not on the Pods.
2. spec.endpoints[].port must equal the name of a port on that Service. A number does not work there; targetPort is the field for numbers or container port names.

If the second join fails, the target job exists but has no targets, which looks almost as empty as the first failure.

Environment

Minimal reproduction

apiVersion: v1
kind: Service
metadata:
  name: orders-api
  namespace: orders
  labels:
    app.kubernetes.io/name: orders-api
spec:
  selector:
    app.kubernetes.io/name: orders-api
  ports:
    - name: http-metrics
      port: 8080
      targetPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orders-api
  namespace: orders
  labels:
    app.kubernetes.io/name: orders-api
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: orders-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Applied to a cluster running kube-prometheus-stack with default values, this produces no target at all.

Who is affected

Anyone using kube-prometheus-stack with default values and ServiceMonitors created outside the chart, anyone running several Prometheus instances that partition ServiceMonitors by label, and teams that restrict serviceMonitorNamespaceSelector to labelled namespaces and create a new namespace without the label.

The fix

We fixed both joins in the ServiceMonitor:

 metadata:
   name: orders-api
   namespace: orders
   labels:
     app.kubernetes.io/name: orders-api
+    release: monitoring
 spec:
   selector:
     matchLabels:
       app.kubernetes.io/name: orders-api
   endpoints:
-    - port: metrics
+    - port: http-metrics
       path: /metrics
       interval: 30s

Longer term we did not want every team to know our Helm release name. We changed the Prometheus selector to a label that expresses intent, and gave the chart that value:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelector:
      matchLabels:
        monitoring.example.com/scrape: "true"
    serviceMonitorNamespaceSelector: {}

Existing ServiceMonitors got the new label in the same change, so nothing dropped out during the switch.

Alternatives and trade-offs

• Select everything. Setting serviceMonitorSelectorNilUsesHelmValues: false with no selector makes the operator pick up every ServiceMonitor in the cluster. Simple, but third-party charts often ship ServiceMonitors you may not want, and with multiple Prometheus instances you lose the ability to partition.
• Keep the release label. Works, but ties every team's manifests to an installation detail. Renaming the release later breaks all of them at once.
• PodMonitor. Skips the Service entirely and selects Pods directly. Useful for workloads without a Service, but the same selector rules apply on the Prometheus side (podMonitorSelector).

How to test the fix

First check that the operator rendered our job into the generated configuration:

kubectl -n monitoring get secret prometheus-monitoring-kube-prometheus-prometheus \
  -o jsonpath='{.data.prometheus\.yaml\.gz}' | base64 -d | gunzip | grep -n "serviceMonitor/orders/orders-api"

Then check the target through the Prometheus API:

kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090 &
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq -r '.data.activeTargets[] | select(.labels.job=="orders-api") | "\(.scrapeUrl) \(.health)"'

http://10.42.2.31:8080/metrics up
http://10.42.4.19:8080/metrics up

Confirming in production

up{job="orders-api"} now returned two series with value 1, and the Grafana dashboard filled in within one scrape interval. We also confirmed the number of targets matched the number of ready Pods, so the Service selector was not missing any replicas:

count(up{job="orders-api"}) == on() kube_deployment_status_replicas_ready{namespace="orders", deployment="orders-api"}

Regression guard

The real problem was silence, so we made silence loud. Every service that should be scraped gets an absent() rule in its own PrometheusRule:

- alert: OrdersApiMetricsAbsent
  expr: absent(up{job="orders-api"})
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "No scrape target for orders-api; check ServiceMonitor selector and port name"

We also added a CI check with kubeconform and a small yq script that fails a pull request if a ServiceMonitor lacks the monitoring.example.com/scrape label or references a port name not defined in the Service from the same chart.

Security implications

Metrics endpoints often expose internal details: route names, dependency hosts, queue names, sometimes build versions useful for targeting known vulnerabilities. Our fix does not change exposure, but while debugging we confirmed /metrics is only on the cluster-internal port and not routed through the Ingress. If ServiceMonitors from any namespace are accepted, a tenant can also point Prometheus at arbitrary endpoints and add load or misleading series; restricting serviceMonitorNamespaceSelector or using a scrape label owned by the platform team limits that.

Prevention checklist

• Know your Prometheus resource's serviceMonitorSelector and namespace selector, and document the required label.
• Reference Service port names, not numbers, in ServiceMonitor endpoints.
• Use /service-discovery and /api/v1/targets before touching the application.
• Add an absent(up{job=...}) alert for every service you expect to scrape.
• Validate ServiceMonitor labels and port names in CI.

References

• Prometheus Operator documentation, ServiceMonitor and Prometheus API: https://prometheus-operator.dev/docs/api-reference/api/
• Prometheus Operator documentation, Troubleshooting ServiceMonitor changes: https://prometheus-operator.dev/docs/platform/troubleshooting/
• kube-prometheus-stack chart values: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
• Prometheus documentation, HTTP API targets: https://prometheus.io/docs/prometheus/latest/querying/api/