High-Performance Framework for Language Models

SGLang is a high-performance serving framework designed for large language models and multimodal models. It focuses on delivering low-latency and high-throughput inference across various setups, from single GPUs to large distributed clusters.

Key features include:

• Fast Runtime - Utilizes RadixAttention for efficient serving, alongside a zero-overhead CPU scheduler and various parallelism techniques.
• Broad Model Support - Compatible with numerous models including Llama, Qwen, and DeepSeek, with easy extensibility for new models.
• Extensive Hardware Support - Runs on NVIDIA, AMD, Intel, and Google TPU hardware.
• Active Community - Open-source with widespread industry adoption, powering over 400,000 GPUs globally.

SGLang is recognized as the industry standard for LLM inference engines, trusted by leading enterprises and institutions.