LeaderWorkerSet
Multi-node inference with LeaderWorkerSet, including autoscaling and topology-aware placement.
Examples
Deployment examples for LWS, organized by API and inference runtime.
Examples focus on two inference runtimes, vLLM and SGLang, across two APIs.
LeaderWorkerSet covers multi-node deployment patterns with autoscaling and topology-aware placement.
DisaggregatedSet covers disaggregated (prefill/decode) deployments with disagg-aware rollouts, autoscaling, and topology placement.
DisaggregatedSet
Disaggregated (prefill/decode) inference with DisaggregatedSet, including disagg-aware rollouts, autoscaling, and topology placement.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.