Rollout Strategy
Rolling update is vital to online services requiring high availability and zero downtime. For LLM inference services, this is particularly important to mitigate stockout and maintain serving capacity during updates.
LeaderWorkerSet supports three parameters within .spec.rolloutStrategy.rollingUpdateConfiguration:
maxUnavailable: Maximum number (or percentage) of replicas (groups of pods) allowed to be unavailable during the update, relative tospec.replicas(percentages are rounded down). Defaults to1.maxSurge: Maximum number (or percentage) of extra replicas that can be created abovespec.replicasduring the update (percentages are rounded up; inOrdinalmode, surge is capped atspec.replicas). Defaults to0.partition: Lowest ordinal updated to the new template during a rolling update. Replicas withordinal >= partitionreceive the new template, while replicas withordinal < partitionstay on the previous revision. Defaults to0. Only supported when.spec.groupIdentityisOrdinal.
maxSurge and maxUnavailable cannot both be zero at the same time.
Example Configuration
Here is a LeaderWorkerSet configured with a rolling update strategy (see a full runtime example here):
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
name: leaderworkerset-sample
spec:
rolloutStrategy:
type: RollingUpdate
rollingUpdateConfiguration:
maxUnavailable: 2
maxSurge: 2
replicas: 4
leaderWorkerTemplate:
size: 4
workerTemplate:
spec:
containers:
- name: nginx
image: nginxinc/nginx-unprivileged:1.27
Rolling Update Process
In both Group Identity modes (Ordinal and Hash), maxUnavailable and maxSurge operate on complete leader-worker groups: a group counts as ready only when its leader pod and all of its worker pods are Ready.
Ordinal Mode (Default)
In Ordinal mode, leader pods are managed by a StatefulSet. The controller drives the rollout by coordinating the leader StatefulSet’s replicas and partition from highest ordinal to lowest:
- Surge creation: If
maxSurge > 0, the controller temporarily scales the leader StatefulSet up tospec.replicas + maxSurgeand setspartitiontospec.replicas. - Batch progression: The controller lowers
partitiontoward0(or.spec.rolloutStrategy.rollingUpdateConfiguration.partition) in steps bounded bymaxUnavailableand active surge groups, waiting for higher ordinals (leader and workers) to be updated andReadybefore advancing. - Surge reclamation: Once the remaining unready target groups (
0throughspec.replicas - 1) fit withinmaxUnavailable, the controller scales the leader StatefulSet back down tospec.replicas.
Below is a step-by-step trace for 4 replicas with maxUnavailable=2 and maxSurge=2 (step size = maxUnavailable + maxSurge = 4).
Status indicators:
- ✅ Replica has been updated to the new revision
- ❎ Replica has not yet been updated (running old revision)
- ⏳ Replica is currently undergoing rolling update (not yet ready)
| Stage | Partition | Replicas | R-0 | R-1 | R-2 | R-3 | R-4 (Surge) | R-5 (Surge) | Description |
|---|---|---|---|---|---|---|---|---|---|
| Stage 1 | 0 | 4 | ✅ | ✅ | ✅ | ✅ | Before the rolling update | ||
| Stage 2 | 4 | 6 | ❎ | ❎ | ❎ | ❎ | ⏳ | ⏳ | Rolling update starts; 2 surge replicas created |
| Stage 3 | 2 | 6 | ❎ | ❎ | ⏳ | ⏳ | ⏳ | ⏳ | Partition decreases to 2; R-2 & R-3 begin update |
| Stage 4 | 2 | 6 | ❎ | ❎ | ⏳ | ⏳ | ✅ | ⏳ | R-4 becomes ready; partition waits for R-5 |
| Stage 5 | 0 | 6 | ⏳ | ⏳ | ⏳ | ⏳ | ✅ | ✅ | R-5 becomes ready; partition drops to 0; R-0 & R-1 begin update |
| Stage 6 | 0 | 6 | ⏳ | ⏳ | ✅ | ✅ | ✅ | ✅ | R-2 and R-3 become ready |
| Stage 7 | 0 | 4 | ⏳ | ⏳ | ✅ | ✅ | Scaled down to 4 replicas; surge replicas reclaimed | ||
| Stage 8 | 0 | 4 | ⏳ | ✅ | ✅ | ✅ | R-1 becomes ready | ||
| Stage 9 | 0 | 4 | ✅ | ✅ | ✅ | ✅ | R-0 becomes ready; rolling update complete |
Hash Mode
When .spec.groupIdentity is Hash, leader pods are managed by a Deployment instead of a StatefulSet, and each new group gets a fresh random identity rather than updating an ordinal in place (see Group Identity for details):
- Deployment-driven rollout:
maxUnavailableandmaxSurgeare passed directly to the leader Deployment’sRollingUpdatestrategy, which scales up the newReplicaSetand scales down the oldReplicaSet. (partitionis not supported inHashmode.) - Whole-group readiness: Multi-pod groups use the
leaderworkerset.sigs.k8s.io/group-readyreadiness gate on the leader pod, which becomesTrueonly after all workers in the group areReady. The Deployment therefore counts a leader as available—and frees budget to terminate another old group—only when its entire group is ready. - Replacement pacing: New leader pods start with the
leaderworkerset.sigs.k8s.io/group-replacementscheduling gate:- With
groupReplacementPolicy: PostTermination(default), surge groups are ungated immediately, while replacement groups staySchedulingGated(without creating workers) until the terminating old group’s leader and worker pods are fully removed. - With
groupReplacementPolicy: Immediate, replacement groups are ungated immediately without waiting for old groups to finish terminating. - At most
spec.replicasgroups on a given revision are ungated at the same time; any extra pods created bymaxSurgebeyondspec.replicasstay gated and are removed first when scaling down.
- With
Below is a trace in Hash mode (groupReplacementPolicy: PostTermination) for 2 replicas with maxUnavailable=1 and maxSurge=1:
| Stage | Old Groups (rev1) | New Groups (rev2) | Description |
|---|---|---|---|
| Stage 1 | G-aaa ✅, G-bbb ✅ | none | Steady state before rolling update (2 available groups). |
| Stage 2 | G-aaa ✅, G-bbb 🗑️ (tearing down) | G-ccc ⏳ (surge, ungated)G-ddd 🔒 (SchedulingGated) | Deployment creates G-ccc (surge) and G-ddd (replacement) on rev2 and terminates G-bbb. G-ccc is ungated immediately; G-ddd waits for G-bbb (leader + workers) to finish terminating. |
| Stage 3 | G-aaa ✅ | G-ccc ⏳, G-ddd ⏳ (ungated) | G-bbb finishes terminating. G-ddd is ungated and creates its workers. |
| Stage 4 | G-aaa 🗑️ (tearing down) | G-ccc ✅, G-ddd ⏳ | G-ccc becomes ready, restoring 2 available groups (G-aaa + G-ccc). The Deployment terminates G-aaa. |
| Stage 5 | none | G-ccc ✅, G-ddd ✅ | G-aaa finishes terminating and G-ddd becomes ready. Rollout complete on rev2. |
Combined Template Update and Scale-Up
When one update changes .spec.leaderWorkerTemplate and increases .spec.replicas from baseline, or increases spec.replicas during an active rollout, both modes follow the same rules:
- Added capacity starts on the new revision:
- In
Ordinalmode, the controller setspartition = baselineand scales tospec.replicasplus anymaxSurge. New ordinals (baseline .. spec.replicas - 1) start directly with the new template. - In
Hashmode,group-replacementscheduling gates keep outdated leader pods unscheduled without workers while the newReplicaSetadds pods.
- In
- Availability-floor pacing:
- The controller paces disruption against the non-surge floor
max(0, min(baseline, spec.replicas) - maxUnavailable). For this floor, a percentagemaxUnavailableresolves againstmin(baseline, spec.replicas), so a large scale-up does not widen disruption on existing groups. Elsewhere, percentagemaxUnavailableandmaxSurgeresolve against desiredspec.replicas:- Capacity-constrained reconfiguration (
maxUnavailable >= 1):Pendingadditions do not consume themaxUnavailablebudget of existing groups. For example, an update from1 × 8 GPUsto2 × 4 GPUswithmaxUnavailable: 1, maxSurge: 0replaces the old group immediately. The two4-GPU groups then schedule into the freed capacity. - Zero-downtime growth (
maxUnavailable: 0, maxSurge > 0): WithmaxUnavailable: 0, the floor equalsbaseline, so the controller disrupts no existing group while new groups start. Each added or surge group that reaches whole-groupReady(leader and all workers) grants credit to replace one old group. At leastbaselineReadygroups stay available throughout the rollout. - Superseding updates and scale-down: A superseding template update raises
baselineto the current count of whole-groupReadygroups, up tospec.replicas. This update protectsReadyadditions and ignoresPendingadditions. The rollout stays active until every group runs the target revision (and, with a userpartition, groups at or above it areReady). A scale-down clamps the floor tomax(0, min(baseline, spec.replicas) - maxUnavailable). InOrdinalmode, a scale-down during an active rollout can keepReadyextra ordinals and raisepartitionto hold the floor. A scale-down with no active rollout removes excess groups immediately, even when some remaining groups are notReady.
- Capacity-constrained reconfiguration (
- The controller paces disruption against the non-surge floor
MaxUnavailable Feature Gate
In Ordinal mode, the controller sets maxUnavailable on the leader StatefulSet. The StatefulSet controller applies that field only when the Kubernetes MaxUnavailableStatefulSet feature gate is enabled. The default of the gate depends on the Kubernetes version:
| Kubernetes version | MaxUnavailableStatefulSet default |
|---|---|
| 1.34 | Disabled (Alpha) |
| 1.35.0 through 1.35.3 | Enabled (Beta) |
| 1.35.4 and later 1.35 patches, 1.36 | Disabled (Beta, see kubernetes#137904) |
| 1.37 and later | Enabled (Beta) |
LeaderWorkerSet does not need the feature gate. A rolling update replaces up to maxUnavailable groups at a time with the gate enabled or disabled:
- Gate enabled: the StatefulSet controller replaces the outdated leaders.
- Gate disabled: the StatefulSet controller replaces one leader at a time and waits at a leader that is not
Ready. The LeaderWorkerSet controller then deletes the next outdated leader pods itself, as far asmaxUnavailableand the availability floor allow. It records aGroupsUpdatingevent on the LeaderWorkerSet for each leader pod that it deletes.
The controller does not set maxUnavailable on worker StatefulSets. In Hash mode, the leader Deployment manages rollout availability and the feature gate has no effect.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.