Top/inference

github.com

whichllm helps you find the best local LLM for your hardware, optimizing AI inference with real-time benchmarks.

github.com

SGLang is an open-source framework for efficient serving of large language and multimodal models, ensuring low-latency and high-throughput performance.

github.com

vLLM is an efficient engine for LLM inference and serving, designed for high throughput and memory management.

github.com

Oumi is an open-source platform for training and deploying LLMs and VLMs, providing tools for evaluation and data synthesis.

github.com

xLLM is an efficient inference engine for large language models, optimized for AI accelerators, enabling cost-effective enterprise deployment.