For most of Kubernetes' history, a pod's resource requests and limits were fixed for its lifetime. To give a container more memory you changed the Deployment, and Kubernetes replaced the pod. For stateless web servers that is fine. For a JVM that takes three minutes to warm its caches, a stateful service with long-lived client connections, or a job halfway through a two-hour computation, a restart is expensive.

In-place pod resize removes that restriction: you can change CPU and memory on a running pod, and the kubelet adjusts the container's cgroup limits, often without restarting anything. The feature graduated to beta, enabled by default, in Kubernetes 1.33, and has been maturing since. This post explains how it works and how we use it.

The basic operation

Resizing uses a dedicated resize subresource on the pod. Changing spec.containers[].resources through a normal update is rejected; going through the subresource is what marks the change as an intentional resize.

kubectl patch pod report-worker-0 --subresource resize --patch '
{"spec":{"containers":[{"name":"worker","resources":{
  "requests":{"cpu":"2","memory":"6Gi"},
  "limits":{"cpu":"4","memory":"6Gi"}}}]}}'

kubectl needs to be version 1.32 or newer for --subresource resize. The pod spec now shows the desired resources, and the pod status shows what the kubelet has actually applied:

kubectl get pod report-worker-0 -o jsonpath='{.status.containerStatuses[0].resources}'

{"limits":{"cpu":"4","memory":"6Gi"},"requests":{"cpu":"2","memory":"6Gi"}}

When the two match, the resize is complete.

Resize status: pending and in progress

Not every resize can happen immediately. The pod reports progress through conditions:

Condition Reason Meaning
PodResizePending Deferred Not possible right now (node lacks free capacity), kubelet will retry
PodResizePending Infeasible Cannot ever happen on this node (more than node capacity)
PodResizeInProgress Kubelet accepted the resize and is applying it 
kubectl get pod report-worker-0 -o jsonpath='{.status.conditions[?(@.type=="PodResizePending")]}'

A Deferred resize is a useful signal: the node is too full. The pod keeps running with its old resources, and the resize happens if capacity frees up. Kubernetes does not move the pod to another node for you; that is a job for an autoscaler or for you.

resizePolicy: restart or not

Each container can declare, per resource, whether a resize requires a container restart:

spec:
  containers:
    - name: worker
      image: registry.example.com/report-worker:4.2.0
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired
        - resourceName: memory
          restartPolicy: RestartContainer
      resources:
        requests: { cpu: "1", memory: "4Gi" }
        limits:   { cpu: "2", memory: "4Gi" }
  • NotRequired (the default) applies the new value to the running container. CPU changes are almost always safe this way: the cgroup quota changes and the process simply gets more or less CPU time.
  • RestartContainer restarts the container to apply the change. This is right for applications that size themselves at startup, such as a JVM with a heap based on container memory, or a database with a fixed buffer pool size. The restart is a container restart in place, not a new pod: same node, same IP, same volumes.

Memory decreases are special

Increasing memory limits is easy. Decreasing them below what the process currently uses would trigger an out-of-memory kill. The kubelet checks current usage and only applies a memory limit decrease when it is safe, otherwise it keeps the resize in progress. Applications rarely give memory back to the operating system on their own, so in practice memory decreases work best with RestartContainer.

QoS class does not change

A pod's quality of service class (Guaranteed, Burstable, BestEffort) is fixed at creation. A resize that would change it is rejected. For example, a Guaranteed pod (requests equal limits for every resource) must stay Guaranteed: you can raise both request and limit together, but not just one. A Burstable pod cannot become Guaranteed through resize. Plan the QoS class you want before creating the workload.

What can and cannot be resized

Can resize Cannot resize (at the time of writing)
CPU and memory requests and limits of regular containers Other resources such as GPUs, ephemeral storage
Sidecar containers (restartable init containers) in recent versions Non-restartable init containers
Pods on Linux nodes with cgroup v2 Windows pods

Check the documentation for your exact cluster version; the list has been growing with each release.

Deployments and controllers

Resizing a pod directly does not change its controller's template. If the Deployment's pod template still says 1 CPU, any new pod, after a rollout, a node drain or a crash, gets 1 CPU again. That is the intended design: in-place resize is a runtime adjustment, the template remains the declared baseline.

In practice there are two patterns:

  1. Permanent change: update the Deployment template as usual. New pods get the new values through a normal rollout. In-place resize is not involved.
  2. Temporary or automatic change: something resizes running pods on demand, such as an operator, a script responding to a workload pattern, or the Vertical Pod Autoscaler.

Vertical Pod Autoscaler

VPA has always been limited by restarts: to apply a recommendation it had to evict pods. VPA has added an update mode that applies recommendations in place where possible, falling back to eviction when an in-place resize is not possible. Check the VPA release notes for the mode name and status in your version. This combination is the most useful application of the feature: pods converge to right-sized resources without the churn of evictions.

Where we use it

  • Report workers. A StatefulSet of workers processes large monthly reports. On the first days of the month we resize CPU up in place for the duration of the run, then back down. No restarts, no lost progress.
  • JVM services during incidents. When a service is CPU-starved during a traffic spike, raising CPU in place takes seconds, much faster than a rollout of new pods that each need to warm up. Memory changes for these services use RestartContainer, because the heap size is computed at startup.
  • Right-sizing experiments. Before changing a template permanently, we resize one pod and watch its latency and throttling metrics for a day.

Things to watch

  • Node capacity. Resize does not reschedule. On full nodes, resizes stay Deferred. Autoscalers that react to pending pods do not see deferred resizes as demand, so keep headroom or handle the condition yourself.
  • CPU throttling metrics. After a CPU limit increase, container_cpu_cfs_throttled_periods_total should fall. If it does not, the application may be limited by something else.
  • Admission policies. Policy engines that validate resources on pod create may not see resizes; make sure your policies also cover the pods/resize subresource if you enforce limits.
  • RBAC. Resizing needs patch on pods/resize, which can be granted separately from general pod update rights.

Key takeaways

  • In-place resize changes running pods' CPU and memory through the resize subresource, beta and on by default since Kubernetes 1.33.
  • Pod status and the PodResizePending and PodResizeInProgress conditions show what has actually been applied.
  • resizePolicy decides per resource whether the container restarts; use RestartContainer for applications that size themselves at startup.
  • QoS class cannot change, and controllers' templates are not updated by a resize.
  • Combined with VPA, it lets pods converge to the right size without eviction churn.

References