Exclusive Topology Placement
Large distributed AI/ML workloads require high-bandwidth, low-latency inter-node communication (e.g., NVLink, RoCE, or InfiniBand) between pods participating in tensor-parallel or pipeline-parallel model execution.
LeaderWorkerSet provides topology-aware placement features to ensure that all pods within an LWS replica land in the same physical topology domain.
Placement Policy
spec.placementPolicy defines a 1:1 mapping between an LWS replica and a topology domain:
spec:
placementPolicy:
type: ExclusiveTopology
topology: cloud.google.com/gke-nodepool
spec.placementPolicy.subGroup does the same for each subgroup, and needs leaderWorkerTemplate.subGroupPolicy. The two levels must use different topology keys:
spec:
placementPolicy:
type: ExclusiveTopology
topology: topology.kubernetes.io/zone
subGroup:
type: ExclusiveTopology
topology: cloud.google.com/gke-nodepool
The field replaces the annotations described below, which are deprecated and still work. Notes on using it:
- Setting a level in the field and in the matching annotation is accepted only when both name the same topology key. A level whose
typeisNoneuses the annotation. - The effective placement, from the field or the annotations, is part of the LeaderWorkerSet revision. Moving a value from an annotation to the field keeps the revision and replaces no group. Changing the placement, through either one, rolls out through a new revision.
- Group level exclusive placement cannot be combined with gang scheduling or workload topology constraints in
spec.scheduling. Subgroup level placement can. - DisaggregatedSet roles cannot combine exclusive placement, from the field or the annotations, with a non-
NoneDisaggregatedSetspec.placementPolicy, on any topology key. The slice and group affinity terms cannot both hold, so kube-scheduler would split a slice across domains, and admission rejects the combination. - Kueue topology-aware scheduling does not account for the affinity that exclusive placement injects. Admission returns a warning when a pod template also carries Kueue topology annotations.
Exclusive Topology Annotation
The annotation leaderworkerset.sigs.k8s.io/exclusive-topology defines a 1:1 mapping between an LWS replica and a topology domain.
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
name: leaderworkerset-sample
annotations:
leaderworkerset.sigs.k8s.io/exclusive-topology: topology.kubernetes.io/zone
spec:
replicas: 3
leaderWorkerTemplate:
size: 4
workerTemplate:
spec:
containers:
- name: worker
image: worker-image:latest
Common topology keys include:
topology.kubernetes.io/zonekubernetes.io/hostname- Custom node labels representing racks, blocks, or network switches (e.g.,
cloud.google.com/gke-placement-group,topology.kubernetes.io/rack, ortopology.kubernetes.io/block).
How It Works
Topology Constraint Enforcement: When an LWS replica is created, the LWS controller configures pod affinity/anti-affinity rules so that all pods belonging to a single replica (the leader and its workers) must be scheduled within the same topology domain instance (e.g., the same rack).
Mutual Exclusivity: Pods from different replicas will not share the same topology domain instance if exclusive placement is requested, preventing noisy-neighbor interference and optimizing cross-node bandwidth.
Subgroup Placements: If you need exclusive placement for subsets of pods within a replica (e.g., placing workers on GPU racks while the leader runs on CPU nodes), use Subgroups in combination with
leaderworkerset.sigs.k8s.io/subgroup-exclusive-topology.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.