AI Skill Evaluation Framework for Agents

agent-skills-eval is a test runner designed for evaluating AI agent skills based on the agentskills.io standard. It facilitates the assessment of AI skills by comparing outputs generated with and without the skill in context, providing a clear measure of effectiveness.

Key features:

• Dual Evaluation - Runs evaluations with and without the skill loaded, allowing for a direct comparison of performance.
• Judge Model Grading - Utilizes a judge model to grade both outputs, ensuring objective assessment.
• Static HTML Reports - Generates comprehensive reports that can be published anywhere, summarizing evaluation results.
• TypeScript SDK and CLI - Offers a command-line interface for easy integration into CI pipelines and a full SDK for custom implementations.
• OpenAI-Compatible - Works seamlessly with various AI models that support the OpenAI chat API.

This framework is particularly useful for developers and researchers looking to validate the performance of their AI skills in a structured manner.