Llm-as-a-judge tooling
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 11 |
| Companies | 57 |
| Map changed | 2026-08-29 |
| Map | open the map |
Llm-as-a-judge tooling is a territory of the AI industry on map of ai. In short: tooling that uses llms to evaluate outputs from agentic pipelines by scoring, ranking, or classifying agent actions and results. The map lists 57 products and companies in 11 categories.
The map of llm-as-a-judge tooling has 11 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
llm eval frameworks10 products and companies · its own article →
LLM-as-a-judge frameworks are platforms that use language models to automatically evaluate and score outputs from other LLMs, replacing human raters at scale. Used by AI teams, researchers, and companies building LLM applications for rapid iteration and quality assurance.
Open ground
Most frameworks still struggle with calibrating judge agreement across domains and lack transparent reasoning explanations that match human evaluator logic.
judge model providers7 products and companies
LLM-as-a-judge tooling enables AI models to evaluate, score, and compare outputs from other AI systems using natural language reasoning. Used by AI teams, researchers, and enterprises to automate quality assessment and benchmarking of language model responses.
Products and companies
observability & tracing platforms10 products and companies
rag & retrieval eval3 products and companies · its own article →
Products and companies
continuous llm testing5 products and companies
LLM-as-a-judge tooling enables automated evaluation and testing of large language models using other LLMs to assess outputs against criteria. Used by ML teams, enterprises, and LLM providers to benchmark and validate model quality at scale.
Products and companies
human-in-the-loop annotation4 products and companies · its own article →
Human-in-the-loop annotation involves AI systems evaluating content with human reviewers in the loop to improve quality and reduce hallucinations. Used by enterprises building safer AI systems and content platforms needing reliable judgment at scale.
red teaming & adversarial eval4 products and companies
Products and companies: giskard, rebuff, lakera guard, adversarial robustness (pyrit).
benchmark leaderboard hosting3 products and companies
Platforms that host and rank LLM model performance across standardized benchmarks, using AI judges to evaluate outputs. Used by researchers, ML engineers, and organizations comparing language model capabilities.
Products and companies: lmsys chatbot arena, hugging face open llm leaderboard, alpacaeval.
judge models3 products and companies
LLM-as-a-judge tooling provides frameworks and APIs for using language models to evaluate outputs from other LLMs, typically by scoring or classifying generated text against criteria. Product teams, researchers, and ML practitioners use these tools to automate evaluation workflows.
Products and companies: atla, nvidia, prometheus eval.
Papers, last 30 days:- Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents 2026-11-12
- AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators 2026-11-19
- RoboJev: An Open Vision-Language Judge for Robot Success Detection 2026-10-24
- Rubric Semantics or Numerical Priors? Auditing Scale-Direction Bias in LLM Judges 2026-11-19
- Same Judgement, Different Scores: Numerical Scoring Topology in LLM-as-a-Judge Evaluation 2026-11-19
cloud provider eval services4 products and companies
Services that use LLMs as evaluators to judge the quality, correctness, or safety of other LLM outputs. Used by teams building LLM applications who need automated evaluation at scale.
Products and companies: amazon bedrock evaluations, azure ai foundry evaluation, vertex ai gen ai evaluation, databricks mosaic ai agent evaluation.
agent and rag testing4 products and companies
LLM-as-a-judge tooling automates evaluation of language model outputs by using other LLMs to score, rank, or judge responses against criteria. AI teams, researchers, and product builders use these tools to measure model quality at scale without manual annotation.
Products and companies: arize ax, maxim ai, langwatch, coval.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.