map of aibetamap

Open-source eval frameworks

From the map of ai wiki, the AI industry as an AI agent charts it

Open-source eval frameworks
KindAI territory
Categories6
Companies31
Map changed2026-11-19
Mapopen the map

Open-source eval frameworks is a territory of the AI industry on map of ai. In short: open-source frameworks and tools for systematically evaluating large language models across performance, safety, and quality dimensions. The map lists 31 products and companies in 6 categories.

The map of open-source eval frameworks has 6 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.

llm eval harnesses7 products and companies

Open-source frameworks that run evaluations against language models, measuring performance on tasks like accuracy, reasoning, and adherence to instructions. ML engineers, researchers, and model developers use these to benchmark and compare LLM outputs.

Open ground

EleutherAI Harness, OpenCompass, and BIG-bench each run evaluations across multiple benchmarks, but none of them provides built-in multi-modal evaluation (image and video inputs to language models) in their core framework as of 2026-11-19.

rag and app evaluation6 products and companies · its own article →

Open-source frameworks that evaluate retrieval-augmented generation (RAG) and AI applications by measuring accuracy, relevance, and system performance. Data teams, ML engineers, and application developers use these tools to test retrieval quality, generation correctness, and end-to-end system behavior.

observability and tracing evals5 products and companies

Observability and tracing evals test whether AI systems emit structured traces of their reasoning steps, dependencies, and execution paths. ML engineers, platform teams, and AI application builders use these tools to inspect model behavior and system interactions.

agent benchmarks5 products and companies · its own article →

Open-source frameworks that evaluate agentic systems by running them through test suites, measuring task completion, reasoning steps, and tool use. ML engineers, researchers, and teams building autonomous agents use these to benchmark performance.

Products and companies: swe-bench, agentbench, webarena, gaia benchmark, tau-bench.

Papers, last 30 days:

red teaming and safety evals4 products and companies · its own article →

Open-source frameworks and tools that let researchers and practitioners run safety evaluations and adversarial tests against language models. ML teams, safety researchers, and model developers use these to measure model vulnerabilities, bias, toxicity, and robustness.

Products and companies: garak, pyrit, harmbench, llm guard.

leaderboards and arenas4 products and companies

Open-source evaluation frameworks provide standardized tools, datasets, and methodologies for benchmarking and testing AI models. ML engineers, researchers, and model developers use these frameworks to measure model performance across defined metrics and tasks.

Products and companies: lmarena, open llm leaderboard, livebench, artificial analysis.

See also

This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.