Kubernetes Service returns connection refused: empty endpoints from a selector and port mismatch

We moved our invoice-rendering service from a hand-written set of manifests to the shared Helm chart our platform team maintains. The deploy went green: Pods were Running, 1/1 Ready, and their logs showed the HTTP server listening. Two minutes later the checkout service started failing every call to it.

From a debug Pod in the same namespace:

$ curl -sS http://invoice-render.billing.svc.cluster.local/healthz
curl: (7) Failed to connect to invoice-render.billing.svc.cluster.local port 80 after 2 ms: Couldn't connect to server

And the ingress controller logged this for the public route:

W0723 10:41:07.112903       7 controller.go:1214] Service "billing/invoice-render" does not have any active Endpoint.

DNS resolved fine, the Service existed, the Pods were healthy. The connection was refused in 2 milliseconds, which meant something on the path actively rejected it rather than timing out.

Reproducing it

Every request, from any Pod, to the Service name or ClusterIP failed. Hitting a Pod IP directly on port 3000 worked:

kubectl -n billing get pod -l app.kubernetes.io/name=invoice-render -o wide
kubectl -n billing exec deploy/checkout -- curl -sS http://10.42.3.17:3000/healthz

{"status":"ok"}

So the application was fine and the problem was between the Service and the Pods.

What we ruled out

DNS. nslookup invoice-render.billing.svc.cluster.local returned the ClusterIP 10.96.144.21, matching kubectl get svc. Not DNS.

NetworkPolicy. A policy drop would look like a timeout, not an instant refusal. We also confirmed no policy in the namespace had changed.

kube-proxy. Other Services on the same nodes worked, and kube-proxy logs had no sync errors.

Root cause

kubectl describe svc gave it away:

Name:              invoice-render
Namespace:         billing
Selector:          app=invoice-render
Type:              ClusterIP
IP:                10.96.144.21
Port:              http  80/TCP
TargetPort:        http/TCP
Endpoints:         <none>

The Service was still the old hand-written one, kept because Helm adopted it under the same name. It selected app=invoice-render. The new chart labelled Pods with the recommended labels, app.kubernetes.io/name=invoice-render and app.kubernetes.io/instance=invoice-render, and no longer set app. The selector matched zero Pods, the Service had no endpoints, and kube-proxy installed a rule that rejects traffic to a Service with no endpoints, which is the instant refusal we saw.

There was a second, hidden problem we only found after fixing the first: the Service used targetPort: http, a named port, and the new chart named the container port web. After we fixed the selector, the EndpointSlice was still empty, because the named port could not be resolved on any Pod.

What is happening under the hood

A Service with a selector is not a proxy process. It is a record that the EndpointSlice controller uses: the controller lists Pods matching the selector in the same namespace, keeps those that are Ready, resolves targetPort against each Pod's container ports, and writes EndpointSlices with Pod IP and port. kube-proxy on every node watches EndpointSlices and programs iptables, IPVS or nftables rules for the ClusterIP.

Two details matter here:

• When a Service has zero ready endpoints, kube-proxy installs a reject rule for its ClusterIP and port, so clients get Connection refused immediately instead of hanging.
• A named targetPort is resolved per Pod by matching ports[].name. If no container port has that name, the Pod is left out of the EndpointSlice. A numeric targetPort is not validated at all: endpoints are created even if nothing listens there, and the kernel in the Pod refuses the connection.

The Endpoints field printed by kubectl describe svc is built from the legacy Endpoints API. From Kubernetes 1.33 that API is deprecated in favour of EndpointSlice, so it is better to inspect EndpointSlices directly.

Environment

Minimal reproduction

apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  selector:
    app: demo
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: demo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: demo
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - name: web
              containerPort: 80

Apply it and run kubectl get endpointslices -l kubernetes.io/service-name=demo. The slice exists with no endpoints, and curling the Service returns connection refused.

Who is affected

Anyone changing label conventions (common when adopting a Helm chart or Kustomize base), renaming container ports, or creating a Service separately from its Deployment. Rollouts that change labels are particularly sneaky because the Deployment's own selector is immutable, so teams often create a new Deployment and forget the Service.

The fix

We changed the Service in the chart values so its selector matches the labels the chart puts on Pods, and pointed targetPort at the real port name:

 spec:
   selector:
-    app: invoice-render
+    app.kubernetes.io/name: invoice-render
+    app.kubernetes.io/instance: invoice-render
   ports:
     - name: http
       port: 80
-      targetPort: http
+      targetPort: web

Then we let Helm own the Service fully, instead of adopting the old object, so the selector and Pod labels come from the same template helper and cannot drift again:

selector:
  {{- include "app.selectorLabels" . | nindent 4 }}

Alternatives and trade-offs

• Add the old app label to the Pods. Fastest during the incident, and it is what we did first to stop the errors. Fine as a bridge, but it leaves two label schemes alive indefinitely.
• Use numeric targetPort. Avoids name lookups, but a wrong number fails silently because endpoints are still published. Named ports fail loudly (empty endpoints), which we prefer.
• Stop the deploy with a pre-check. Covered below; it complements, not replaces, the template fix.

How to test the fix

Endpoints must contain the Pod IPs and the right port before any traffic test:

kubectl -n billing get endpointslices -l kubernetes.io/service-name=invoice-render \
  -o custom-columns=NAME:.metadata.name,ADDR:.endpoints[*].addresses[0],PORT:.ports[*].port,READY:.endpoints[*].conditions.ready

NAME                   ADDR                    PORT   READY
invoice-render-8kq2v   10.42.3.17,10.42.1.22   3000   true,true

Then confirm that the selector really matches the Pods it should, using the selector string from the Service itself:

SEL=$(kubectl -n billing get svc invoice-render -o go-template='{{range $k,$v := .spec.selector}}{{$k}}={{$v}},{{end}}' | sed 's/,$//')
kubectl -n billing get pods -l "$SEL"

Finally, test through the Service from inside the cluster:

kubectl -n billing run curl --rm -it --restart=Never --image=curlimages/curl:8.14.1 -- \
  curl -sS -o /dev/null -w "%{http_code}\n" http://invoice-render/healthz

Expected output: 200.

Confirming in production

After the fix, ingress-nginx stopped logging the missing endpoint warning and the checkout error rate returned to baseline within a minute. We also checked that every Service in the namespace had at least one ready endpoint:

for s in $(kubectl -n billing get svc -o name | cut -d/ -f2); do
  n=$(kubectl -n billing get endpointslices -l kubernetes.io/service-name=$s \
      -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}' | grep -c .)
  echo "$s $n"
done

Any line ending in 0 is a Service pointing at nothing.

Regression guard

• A Prometheus alert on kube-state-metrics: Services whose EndpointSlices have no ready addresses for five minutes, scoped to namespaces that should always be serving.
• A post-deploy smoke test in the pipeline that curls each Service by name from a throwaway Pod, so a passing Deployment rollout cannot hide a dead Service.
• A chart unit test (helm-unittest) asserting that the Service selector equals the Pod template labels.

Security implications

Selectors deserve a security note. A Service whose selector is too broad (or one that a team copies with a generic label like app: api) can pull Pods from another workload in the same namespace into its endpoints, sending traffic and credentials to the wrong backend. Keep selectors specific, use the instance label, and avoid sharing namespaces between unrelated services. The fix itself does not change the attack surface.

Prevention checklist

• Generate Service selectors and Pod labels from the same template helper.
• Look at EndpointSlices, not just Pod status, when a Service misbehaves.
• Prefer named targetPort values so mismatches surface as empty endpoints.
• Add a smoke test that exercises the Service name after every deploy.
• Alert on Services with zero ready endpoints.

References

• Kubernetes documentation, Service: https://kubernetes.io/docs/concepts/services-networking/service/
• Kubernetes documentation, EndpointSlices: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
• Kubernetes documentation, Debug Services: https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/
• Kubernetes documentation, Recommended Labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/