When a new engineer on our platform team asked why a Deployment "took 70 seconds to come up" while kubectl apply had returned instantly, we realised most of us could name the components but not the order in which they touch a Pod. That gap matters during incidents: if you do not know which stage is slow, you guess, and guessing on a production cluster is expensive.

This post walks a single Deployment from YAML file to running process on Kubernetes 1.31, the current release as we write. For every hop we name the component responsible, what it writes back to the API, and the command that shows you that stage. By the end you should be able to look at a stuck or slow Pod and say which part of the pipeline owns the delay.

The manifest we will follow

We use a deliberately ordinary workload so nothing exotic gets in the way.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
  namespace: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: catalog
  template:
    metadata:
      labels:
        app: catalog
    spec:
      initContainers:
        - name: wait-config
          image: busybox:1.36
          command: ["sh", "-c", "test -f /config/app.yaml"]
          volumeMounts:
            - name: config
              mountPath: /config
      containers:
        - name: app
          image: registry.example.com/shop/catalog:1.8.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
          volumeMounts:
            - name: config
              mountPath: /config
      volumes:
        - name: config
          configMap:
            name: catalog-config

Step 1: kubectl talks to the API server, and only the API server

kubectl apply reads the file, finds your cluster in kubeconfig, and sends HTTPS requests to kube-apiserver. With client-side apply it first GETs the current object, computes a three-way merge against the last-applied-configuration annotation, and sends a PATCH. With --server-side the merge happens on the server and field ownership is tracked in managedFields.

Nothing else in the cluster is contacted. kubectl never talks to nodes, the scheduler or etcd. Add -v=6 to see the exact calls:

kubectl apply -f catalog.yaml -v=6 2>&1 | grep -E "GET|PATCH|POST"

Step 2: authentication, authorization, admission

Inside the API server the request passes three gates in order.

  1. Authentication turns your client certificate, bearer token or OIDC token into a user and groups.
  2. Authorization, usually RBAC, decides whether that identity may patch deployments in namespace shop. A failure here is a 403 and your YAML never gets further.
  3. Admission runs in two passes. Mutating admission plugins and webhooks may change the object (for example inject a sidecar or default a field). Then schema validation runs, then validating admission plugins, webhooks and ValidatingAdmissionPolicy (GA in 1.30) may reject it.

A slow or unreachable admission webhook is a classic source of "apply hangs and then times out". The API server logs the webhook name, and you can list them with:

kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations

Keep in mind that the Deployment passes admission here, but the Pods created later pass admission again on their own. A webhook that only matches Pods will not complain until step 4.

Step 3: etcd stores the desired state

Once admitted, the API server writes the object to etcd and returns. This is the moment kubectl apply prints deployment.apps/catalog created. The object now has a resourceVersion and a metadata.generation, but no Pod exists yet. Everything from here on is asynchronous and driven by controllers that watch the API server.

Step 4: controllers turn a Deployment into Pods

kube-controller-manager runs dozens of control loops. Two of them matter now.

  • The Deployment controller sees a Deployment whose template has no matching ReplicaSet. It creates one named catalog-<pod-template-hash>.
  • The ReplicaSet controller sees a ReplicaSet wanting 2 replicas and owning 0 Pods. It creates 2 Pod objects through the API server.

Those Pod creations go through admission themselves, which is where quota (ResourceQuota), LimitRange defaults and Pod Security Admission are enforced. If a Pod is rejected here, the Pod object never exists, so kubectl get pods shows nothing. The evidence lives on the ReplicaSet:

kubectl -n shop describe rs -l app=catalog | sed -n '/Events/,$p'

An event such as FailedCreate ... exceeded quota is the tell. Many people look only at Pods and miss it.

Step 5: the scheduler picks a node

New Pods have an empty spec.nodeName. kube-scheduler watches for exactly that. For each Pod it runs a scheduling cycle.

  • Filtering removes nodes that cannot run the Pod: not enough allocatable CPU or memory for the requests, taints the Pod does not tolerate, node selectors or affinity that do not match, volume topology conflicts.
  • Scoring ranks the remaining nodes, for example spreading replicas and favouring nodes that already have the image.
  • Binding writes a Binding to the API server, which sets spec.nodeName.

If no node passes filtering, the Pod stays Pending with a FailedScheduling event that lists how many nodes failed each filter. Scheduling is usually milliseconds; when it is not, the event explains why.

kubectl -n shop get events --field-selector involvedObject.kind=Pod --sort-by=.lastTimestamp

Step 6: the kubelet takes over

Each node's kubelet watches for Pods bound to its own node name. When one appears it runs a sync loop that, roughly in order:

  1. Checks admission on the node (enough resources from the kubelet's view, runtime class available).
  2. Prepares volumes. For our ConfigMap this means fetching it and writing files into a directory under /var/lib/kubelet/pods/<uid>/volumes. For a CSI volume the attach and mount operations happen here and can take tens of seconds on cloud disks.
  3. Asks the container runtime to create the Pod sandbox.

A missing ConfigMap or Secret stops the Pod at this point with FailedMount or CreateContainerConfigError events, while the Pod sits in ContainerCreating.

Step 7: CRI, the sandbox and the network

The kubelet does not run containers itself. It calls the Container Runtime Interface, a gRPC API implemented by containerd or CRI-O. RunPodSandbox creates the sandbox: a small pause container that holds the Pod's network namespace and, optionally, IPC namespace.

While creating the sandbox, the runtime invokes the CNI plugin configured on the node (Calico, Cilium, the AWS VPC CNI and so on). The plugin creates the interface inside the namespace, assigns the Pod IP and programs routes. If CNI fails or the node has run out of IPs, you see FailedCreatePodSandBox events, and the Pod never gets an IP.

You can watch this layer directly on the node:

sudo crictl pods --name catalog
sudo crictl inspectp <pod-id> | grep -i '"ip"'

Step 8: images are pulled, init containers run

For each container the kubelet asks the runtime to pull the image, honouring imagePullPolicy and any imagePullSecrets. Pulls are serialised per node by default (serializeImagePulls: true in the kubelet config), so one huge image can delay unrelated Pods on the same node. Pull time appears in the Pulled event:

Successfully pulled image "registry.example.com/shop/catalog:1.8.2" in 41.2s (41.2s including waiting). Image size: 412337715 bytes.

Init containers then run one at a time, each to completion. Our wait-config container must exit 0 before anything else starts. If it fails, the kubelet restarts it according to the Pod's restart policy and the status shows Init:CrashLoopBackOff or Init:Error.

Step 9: app containers start

The kubelet calls CreateContainer and StartContainer for each regular container. The runtime hands the OCI spec to runc (or crun), which sets up namespaces, cgroups with the CPU and memory settings derived from requests and limits, mounts, and finally execs the entrypoint.

At this point the container state is Running, but that only means the process exists. It says nothing about whether the application is ready.

Step 10: probes, readiness and EndpointSlices

The kubelet now runs probes. A startupProbe, if defined, gates the others. Our readinessProbe hits /healthz; only after it succeeds does the kubelet set the Pod condition Ready=True.

The EndpointSlice controller watches that condition. Ready Pods matching a Service selector get added to the Service's EndpointSlices, and kube-proxy (or the CNI's eBPF replacement) on every node programs rules so traffic to the Service IP reaches the new Pod. This is the last hop: only now does the Pod receive real traffic.

kubectl -n shop get pod -l app=catalog -o wide
kubectl -n shop get endpointslices -l kubernetes.io/service-name=catalog

Reading the timeline of a real Pod

All of the stages above leave timestamps. The Pod conditions record when each milestone happened:

kubectl -n shop get pod catalog-7d9c6b5f4-x2klp \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.lastTransitionTime}{"\n"}{end}'

PodReadyToStartContainers   2024-11-04T09:12:07Z
Initialized                 2024-11-04T09:12:51Z
Ready                       2024-11-04T09:13:16Z
ContainersReady             2024-11-04T09:13:16Z
PodScheduled                2024-11-04T09:12:03Z

In this example scheduling took effectively no time, the sandbox was ready 4 seconds later, init finished 44 seconds after that because both images had to be pulled cold, and readiness took another 25 seconds. That breakdown told us to look at image size and probe timing, not at the scheduler.

Where the time usually goes

Stage Typical symptom First command to run
Admission webhook apply hangs or times out kubectl get validatingwebhookconfigurations
Quota or LimitRange no Pods at all kubectl describe rs
Scheduling Pending, FailedScheduling kubectl describe pod
Volumes ContainerCreating, FailedMount kubectl get events
CNI FailedCreatePodSandBox crictl pods on the node
Image pull long gap before Started Pulled event duration
Readiness Running but 0/1 ready probe config and app logs

Key takeaways

  • kubectl apply only writes desired state to the API server; every other step is a controller reacting to a watch.
  • Pods pass admission separately from their Deployment, so quota and policy rejections show up on the ReplicaSet, not on a Pod.
  • The scheduler only sets spec.nodeName; volumes, sandbox, CNI, image pulls and probes are all the kubelet's job on the node.
  • Running is not Ready: traffic arrives only after readiness passes and EndpointSlices are updated.
  • Pod condition timestamps plus events give you a per-stage timeline, which beats guessing during an incident.

References