Top/transformer

github.com

A framework for evaluating language models with a focus on few-shot tasks, supporting various model backends and benchmarks.

github.com

SGLang is an open-source framework for efficient serving of large language and multimodal models, ensuring low-latency and high-throughput performance.

github.com

vLLM is an efficient engine for LLM inference and serving, designed for high throughput and memory management.

github.com

MNN is a lightweight deep learning framework optimized for on-device inference and training, supporting various AI models and platforms.