Promptfoo: CLI for LLM Evaluation and Security Testing

Promptfoo is a command line interface (CLI) and library designed for evaluating and red-teaming large language model (LLM) applications. It aims to streamline the process of ensuring the security and reliability of AI applications, moving away from trial-and-error methods.

Key features include:

• Automated evaluations - Test your prompts and models systematically.
• Red teaming and vulnerability scanning - Secure your LLM applications against potential threats.
• Model comparison - Compare various models side-by-side, including OpenAI, Anthropic, and others.
• CI/CD automation - Integrate checks into your continuous integration and deployment workflows.
• Local execution - All evaluations run locally, ensuring privacy and security.

Use cases for Promptfoo include enhancing the security of AI applications, automating testing processes, and facilitating collaboration among development teams through shared evaluation results.