Benchmark leaderboard infra
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 14 |
| Companies | 66 |
| Map changed | 2026-08-29 |
| Map | open the map |
Benchmark leaderboard infra is a territory of the AI industry on map of ai. In short: benchmark leaderboard infrastructure runs comparative evaluation tables that rank llm systems across standardized test sets; ml teams, researchers, and model vendors use these to track performance deltas over time. The map lists 66 products and companies in 14 categories.
The map of benchmark leaderboard infra has 14 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
crowd preference arenas4 products and companies
Crowd preference arenas are platforms where users vote, rank, and compare options to create benchmark leaderboards and community-driven rankings. Used by product teams, researchers, and communities to validate preferences and discover winners.
Products and companies
Open ground
Most platforms lack real-time cross-domain preference aggregation with persistent identity and reputation scoring for voting participants.
composite intelligence indexes5 products and companies
Composite intelligence indexes aggregate and rank AI model performance across multiple benchmarks and dimensions, providing standardized comparison frameworks. Used by researchers, enterprises, and AI practitioners to evaluate and select models for deployment.
Products and companies
agentic & long-horizon evals7 products and companies
Products and companies
coding & software evals5 products and companies
Products and companies
math & reasoning benchmarks5 products and companies
Products and companies
Papers, last 30 days
computer use & ui evals4 products and companies
Products and companies
safety & hallucination evals4 products and companies
Products and companies
embedding & retrieval evals3 products and companies
Products and companies: mteb, beir, bright.
multimodal benchmark suites4 products and companies
Multimodal benchmark suites are standardized evaluation frameworks that test AI systems across multiple modalities (vision, language, audio) to measure their capabilities. Researchers, AI labs, and companies use them to compare model performance and track progress.
Products and companies: mmmu, mathvista, videomme, charxiv.
llm leaderboards5 products and companies · its own article →
LLM benchmark leaderboards rank language models on standardized tasks to measure capabilities across reasoning, coding, knowledge, and instruction-following. Researchers, practitioners, and model developers use these leaderboards to track progress and compare systems.
Products and companies: hugging face open llm leaderboard, livebench, scale ai seal leaderboards, epoch ai, vellum.
Papers, last 30 days:- SOTA or Luck? The Winner's Curse in LLM Leaderboards 2026-11-17
- AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models 2026-11-10
- Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks 2026-11-02
- Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models 2026-11-02
evaluation harnesses6 products and companies
Evaluation harnesses provide infrastructure and frameworks for running benchmarks, tests, and leaderboard competitions against AI models. ML researchers, model developers, and benchmark operators use them to execute standardized evaluations and track performance metrics.
Products and companies: eleutherai lm evaluation harness, inspect ai, openai evals, lighteval, deepeval, promptfoo.
Papers, last 30 days:llm observability and eval platforms6 products and companies · its own article →
Platforms that run standardized tests against large language models and publish comparative rankings of model performance. ML researchers, model vendors, and procurement teams use these to track model capabilities over time.
Products and companies: langsmith, braintrust, arize phoenix, weights & biases weave, galileo, langfuse.
vertical domain leaderboards4 products and companies · its own article →
Vertical domain leaderboards rank and compare AI models on benchmarks specific to narrow domains like medicine, law, or finance. Researchers, practitioners, and vendors in those domains use them to track model performance on task-specific datasets.
Products and companies: vals ai, berkeley function calling leaderboard, copilot arena, aider leaderboards.
consumer model comparison tools4 products and companies
Tools that display ranked comparisons of consumer AI models across benchmarks and performance metrics. Researchers, developers, and enterprises use these to evaluate which models fit their use cases.
Products and companies: openrouter rankings, yupp, nano gpt, t3 chat.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.