Autoscaling

This guide is the basic deployment plus a HorizontalPodAutoscaler. The HPA scales the number of replica groups through the LWS scale subresource (it monitors leader pods only), targeting 50% CPU utilization between the minReplicas and maxReplicas each manifest sets. It needs metrics-server.

Deploy

• vLLM
• SGLang
• nginx (no GPU)

export HF_TOKEN=<your-hf-token>
curl https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/leaderworkerset/autoscaling/vllm.yaml -s | envsubst | kubectl apply -f -

export HF_TOKEN=<your-hf-token>
curl https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/leaderworkerset/autoscaling/sglang.yaml -s | envsubst | kubectl apply -f -

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/leaderworkerset/autoscaling/nginx.yaml

The vLLM and SGLang examples need GPUs and a Hugging Face token. The nginx one runs on any cluster with metrics-server, including kind, so it is the quickest way to see the behavior described here.

Watch the HPA react to load:

kubectl get hpa -w

See basic for how to reach the service once pods are running.

Resource requests are required

The HPA computes utilization as a percentage of a pod’s resource requests. If a container has no request for the metric being targeted, the HPA reports <unknown> for it and will not scale. That is why every container in the manifests above requests the metric its HPA targets: CPU in all three, plus memory in the nginx one, which is the deployment the variants below use.

Scaling on other metrics

The examples above scale on CPU. Two variants of the same HPA are included for the nginx deployment, one on memory and one on CPU and memory together.

An HPA takes ownership of its target’s replica count, so pointing a second one at the same LeaderWorkerSet makes the two fight. Delete the CPU HPA before applying a variant:

kubectl delete hpa lws-hpa

• Memory
• CPU + memory

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/leaderworkerset/autoscaling/nginx-memory-hpa.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/leaderworkerset/autoscaling/nginx-multi-metric-hpa.yaml

With several metrics the HPA computes a desired replica count for each and takes the largest, so either one alone can scale the deployment up.

The CPU + memory variant also sets behavior, which is worth tuning for LWS in particular: each replica is a whole group, so scaling down aggressively tears down a leader and its workers together. It scales up immediately but waits five minutes below target before scaling down.

Generating load

The HPA reads leader pods only, so load has to land on a leader to trigger scaling. With the nginx deployment running:

kubectl exec leaderworkerset-sample-0 -- /bin/sh -c "for i in \$(seq 1 4); do yes > /dev/null & done"

CPU utilization should cross the 50% target within a couple of minutes and the replica count should climb. To clear the load, restart the pod, since the nginx image has no pkill:

kubectl delete pod leaderworkerset-sample-0

Scale-down waits for the HPA stabilization window, five minutes by default, before it starts.

Troubleshooting

kubectl describe hpa <name> reports why a decision was or was not made.

• Targets show <unknown>: metrics-server is not running, or a container is missing the resource request for the metric being targeted.
• Load does not trigger scaling: confirm it is on a leader pod. Worker pod usage is not read by the HPA.
• Replica count oscillates: raise behavior.scaleDown.stabilizationWindowSeconds, as the CPU + memory variant does.

Cleanup

Delete whichever HPA you ended up with, then the LeaderWorkerSet:

kubectl delete hpa lws-hpa lws-memory-hpa lws-multi-metric-hpa --ignore-not-found
kubectl delete leaderworkerset leaderworkerset-sample