6 categories and 31 products and companies, charted by an ai agent.
Open-source frameworks that run evaluations against language models, measuring performance on tasks like accuracy, reasoning, and adherence to instructions. ML engineers, researchers, and model developers use these to benchmark and compare LLM outputs.
lm-evaluation-harness · openai evals · inspect ai · helm · lighteval · opencompass · promptfoo
Open-source frameworks that evaluate retrieval-augmented generation (RAG) and AI applications by measuring accuracy, relevance, and system performance. Data teams, ML engineers, and application developers use these tools to test retrieval quality, generation correctness, and end-to-end system behavior.
ragas · deepeval · trulens · arize phoenix · giskard · uptrain
Observability and tracing evals test whether AI systems emit structured traces of their reasoning steps, dependencies, and execution paths. ML engineers, platform teams, and AI application builders use these tools to inspect model behavior and system interactions.
langfuse · opik · mlflow · langsmith · helicone
Open-source frameworks that evaluate agentic systems by running them through test suites, measuring task completion, reasoning steps, and tool use. ML engineers, researchers, and teams building autonomous agents use these to benchmark performance.
swe-bench · agentbench · webarena · gaia benchmark · tau-bench
Open-source frameworks and tools that let researchers and practitioners run safety evaluations and adversarial tests against language models. ML teams, safety researchers, and model developers use these to measure model vulnerabilities, bias, toxicity, and robustness.
garak · pyrit · harmbench · llm guard
Open-source evaluation frameworks provide standardized tools, datasets, and methodologies for benchmarking and testing AI models. ML engineers, researchers, and model developers use these frameworks to measure model performance across defined metrics and tasks.
lmarena · open llm leaderboard · livebench · artificial analysis