Installation
Before you begin
Make sure the following conditions are met:
- A Kubernetes cluster with version >= 1.34 is required (officially supported and tested on Kubernetes 1.34 through 1.37), or it may behave unexpectedly. Learn how to install the Kubernetes tools.
- LeaderWorkerSet does not require the MaxUnavailableStatefulSet feature gate. Rolling updates honor
maxUnavailablewith the gate enabled or disabled. The default of the gate differs between Kubernetes versions. See MaxUnavailable Feature Gate and the discussion here.
- LeaderWorkerSet does not require the MaxUnavailableStatefulSet feature gate. Rolling updates honor
- Your cluster has at least 1 node with 1+ CPUs and 1G of memory available for the LeaderWorkerSet controller manager Deployment to run on. NOTE: On some cloud providers, the default node machine type will not have sufficient resources to run the LeaderWorkerSet controller manager and all the required kube-system pods, so you’ll need to use a larger machine type for your nodes.
- The kubectl command-line tool has communication with your cluster.
Install the development version (main branch)
These commands install unreleased builds from the main branch only.
For a very recent commit on main, the documentation may update before its Helm
chart is published. If the chart is not yet available, retry the same command later.
For released installations, use the latest stable documentation.
Install by kubectl
kubectl apply --server-side -k 'github.com/kubernetes-sigs/lws/config/default?ref=main'
To wait for LeaderWorkerSet to be fully available, run:
kubectl wait deploy/lws-controller-manager -n lws-system --for=condition=available --timeout=5m
Install by Helm
To install lws from the main branch in your cluster by Helm, run the following command:
CHART_VERSION=0.0.0-main-b869a69
CHART_REFERENCE=oci://us-central1-docker.pkg.dev/k8s-staging-images/lws/charts/lws
helm install lws "$CHART_REFERENCE" \
--version="$CHART_VERSION" \
--namespace lws-system \
--create-namespace \
--wait --timeout 300s
Upgrade by Helm
Helm only installs the chart’s CRDs during the initial helm install. It does
not update or delete CRDs on helm upgrade (see the
Helm documentation),
so CRD schema changes and newly added CRDs do not reach the cluster through
helm upgrade alone.
Apply the CRDs explicitly before upgrading, then upgrade the release in place:
CHART_VERSION=0.0.0-main-b869a69
CHART_REFERENCE=oci://us-central1-docker.pkg.dev/k8s-staging-images/lws/charts/lws
helm pull "$CHART_REFERENCE" --version="$CHART_VERSION" --untar &&
kubectl apply --server-side --force-conflicts -f lws/crds &&
helm upgrade lws ./lws \
--namespace lws-system \
--wait --timeout 300s
helm upgrade does not update CRD schemas — Helm never modifies CRDs placed
in the crds/ directory after the initial helm install, and does not delete
them on helm uninstall either. Always reconcile CRD schemas explicitly with
the kubectl apply step above before upgrading the chart.
Upgrading from v0.7.0 or earlier
Chart versions up to v0.7.0 rendered the LeaderWorkerSet CRD from
templates/crds/, so the CRD is part of the Helm release manifest. Starting
with v0.8.0 the CRD ships from the special crds/ directory and is no longer
part of the release. Without preparation, the first helm upgrade across that
boundary treats the CRD as removed from the release and deletes it — cascading
to the deletion of every LeaderWorkerSet in the cluster (see
#880).
Before the first upgrade from v0.7.0 or earlier, run this one-time step so Helm keeps the CRD when it leaves the release:
kubectl annotate crd leaderworkersets.leaderworkerset.x-k8s.io \
helm.sh/resource-policy=keep --overwrite
Then follow the regular upgrade flow above (apply the CRDs, then
helm upgrade). Subsequent upgrades no longer need the annotation step.
Uninstall
kubectl delete -k 'github.com/kubernetes-sigs/lws/config/default?ref=main'
helm uninstall lws --namespace lws-system
Enable native gang scheduling
Native gang scheduling is available in LWS v0.11.0 and later. It is supported
on Kubernetes 1.37, where LWS creates the
scheduling.k8s.io/v1beta1 Workload and PodGroup resources introduced in
that Kubernetes release. Earlier supported Kubernetes minors do not provide
these API versions.
The LWS integration is alpha, and its required switches are disabled by
default in both Kubernetes and LWS. Configure the cluster and the LWS
controller before creating a LeaderWorkerSet with spec.scheduling.
Configure Kubernetes
Enable the GenericWorkload feature gate on all three control-plane
components:
kube-apiserverkube-controller-managerkube-scheduler
Also enable the scheduling API on kube-apiserver with
--runtime-config=scheduling.k8s.io/v1beta1=true. How these settings are
configured depends on your Kubernetes provider. For example, a Kind cluster
configuration contains:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
GenericWorkload: true
runtimeConfig:
scheduling.k8s.io/v1beta1: true
Configure LeaderWorkerSet
Enable the LWS WorkloadAwareScheduling feature gate and select the native
Kubernetes scheduler provider. With Helm, add these values to the install or
upgrade command shown above:
--set featureGates.WorkloadAwareScheduling=true \
--set gangSchedulingManagement.schedulerProvider=kubernetes
For a configuration-file installation, edit the lws-manager-config
ConfigMap in the controller namespace and add the equivalent settings to the
controller manager configuration:
featureGates:
WorkloadAwareScheduling: true
gangSchedulingManagement:
schedulerProvider: kubernetes
Restart the lws-controller-manager Deployment after changing an existing
configuration ConfigMap:
kubectl rollout restart deployment/lws-controller-manager -n lws-system
kubectl rollout status deployment/lws-controller-manager -n lws-system --timeout=5m
Verify the prerequisites
Confirm that Kubernetes serves both required APIs:
kubectl api-resources --api-group=scheduling.k8s.io | grep -E 'workloads|podgroups'
The output must include workloads and podgroups at version v1beta1.
Then follow the native gang scheduling quickstart.
Operational notes:
- When
gangSchedulingManagement.schedulerProvideris set tokubernetes,lws-controller-managerindexes and watchesscheduling.k8s.io/v1beta1WorkloadandPodGroupat startup and exits if those APIs are not served (for example, ifGenericWorkloador thescheduling.k8s.io/v1beta1runtime config is disabled), even forLeaderWorkerSetobjects that do not setspec.scheduling. Remove thegangSchedulingManagementblock before disabling the upstream Kubernetes APIs.- Disabling
featureGates.WorkloadAwareSchedulingor removing thegangSchedulingManagementblock rejects newLeaderWorkerSetobjects that requestspec.scheduling, while keeping existing opted-in objects updateable so operators can scale them down (spec.replicas: 0) and delete them cleanly.
Build and install from source
To build LeaderWorkerSet from source and install LeaderWorkerSet in your cluster, run the following commands:
git clone https://github.com/kubernetes-sigs/lws.git
cd lws
IMAGE_REGISTRY=<registry>/<project> make image-push deploy
Uninstall
To uninstall LeaderWorkerSet, run the following command:
make undeploy
Install in a different namespace
To install the leaderWorkerSet controller in a different namespace rather than lws-system, you should first:
git clone https://github.com/kubernetes-sigs/lws.git
cd lws
Then change the kustomization.yaml namespace field as:
namespace: <your-namespace>
Optional: Use cert manager instead of internal cert
The webhooks use an internal certificate by default. However, if you wish to use cert-manager (which supports cert rotation), instead of internal cert, follow the cert manage guide.
Install with Helm chart
See Install by Helm above.
DisaggregatedSet
Starting from v0.9.0, DisaggregatedSet is bundled with the LWS controller manager.
For kubectl and Kustomize installs, the standard v0.9.0+ manifests include the DisaggregatedSet CRD, controller permissions, and validating webhook. No separate DisaggregatedSet installation step is required.
Helm charts v1.0.0 and prior require --set enableDisaggregatedSet=true for the
validating webhook and editor/viewer/admin ClusterRoles. Despite its name, this
option never controlled the DisaggregatedSet controller or CRDs, which were
already installed. The Helm chart now always installs these resources, matching
the standard manifests. Follow the Helm installation steps
without the extra flag.
Old enableDisaggregatedSet values are now ignored, even with
--reuse-values. If you previously set it to false, upgrading enables webhook
validation for new or updated DisaggregatedSets. You can remove the obsolete
option from your values files.
Verify Installation
- Wait for the controller manager to become available:
kubectl wait deploy/lws-controller-manager -n lws-system \
--for=condition=available --timeout=5m
- Confirm the DisaggregatedSet CRD is registered:
kubectl get crd disaggregatedsets.disaggregatedset.x-k8s.io
- Confirm the validating webhook configuration:
kubectl get validatingwebhookconfiguration lws-validating-webhook-configuration \
-o yaml | grep disaggregatedsets
Upgrade from an older version
Migrate pre-slices DisaggregatedSets before v1.0.0
DisaggregatedSet first shipped in v0.9.0, before the slices feature. If a
DisaggregatedSet was originally created by v0.9.x and has not completed a rollout while
running v0.10.x or v0.11.x, do not upgrade directly to v1.0.0. First install either
v0.10.x or v0.11.x, then trigger and complete one template rollout for every affected
DisaggregatedSet. For example, changing a container image in a role’s pod template
triggers a rollout.
This replaces the generated LeaderWorkerSets and pods with objects that carry
slice-aware names and labels. If the DisaggregatedSet has already completed such a
rollout, no additional migration is required.
Before upgrading to v1.0.0, verify that no pre-slices LeaderWorkerSets remain. The following command must produce no output:
kubectl get leaderworkersets -A \
-l 'disaggregatedset.x-k8s.io/name,!disaggregatedset.x-k8s.io/slice'
helm upgrade does not install newly added CRDs. DisaggregatedSet ships two:
disaggregatedsets since v0.9.0 and disaggregatedsetrolescalers since v0.10.0. Apply only
disaggregatedsets and the controller cannot create the DisaggregatedSetRoleScaler a role with
scaling.mode: External depends on, so External scaling never takes effect.
Follow the Upgrade by Helm steps to apply all CRDs before upgrading the chart. The chart version you are upgrading to determines the flag:
- The chart documented here: omit
--set enableDisaggregatedSet=true; the webhook and user-facing ClusterRoles are always installed, ignoring any saved flag value. - v1.0.0 and prior: add
--set enableDisaggregatedSet=trueto install those resources.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.