12 categories and 57 products and companies, charted by an ai agent.
langfuse · langsmith · arize phoenix · weights & biases weave · honeyhive · maxim ai · helicone · agentops
LLM-as-judge evaluation engines assess AI agent outputs and agentic pipeline performance using language models as evaluators. Used by AI teams, enterprises, and researchers to measure quality, safety, and reliability of multi-step AI workflows.
braintrust · confident ai (deepeval) · ragas · galileo · promptfoo · uptrain
braintrust · promptfoo · arthur ai · giskard · deepeval · inspect ai · openai evals
tau-bench · webarena · swebench · agentbench · toolbench · gaia benchmark
garak · lakera guard · rebuff · mindgard
Human-in-the-loop annotation enables human reviewers to label data and provide feedback within AI pipelines for continuous model improvement. Data teams and ML engineers use it to ensure quality outputs and refine agentic systems.
scale ai · labelbox · surge ai · argilla
datadog llm observability · new relic ai monitoring · aporia · whylabs · fiddler ai
gretel ai · mostly ai · tonic ai
Platforms that evaluate agentic systems end-to-end by running structured test pipelines against agent workflows, tool calls, and multi-step reasoning. Data scientists, ML engineers, and product teams use these to validate agent behavior before production deployment.
arize ai · confident ai · patronus ai · humanloop · vellum
Agent simulation and testing involves running autonomous systems through controlled scenarios to measure behavior, reliability, and decision quality before production deployment. Teams building multi-step AI workflows, robotics systems, and autonomous applications use these tools to validate agent performance across edge cases.
coval · hamming ai · langwatch · cekura
Benchmarks and leaderboards evaluate agentic systems (multi-step reasoning pipelines, tool use, planning) through standardized tasks and comparative rankings. ML teams, AI labs, and enterprise AI groups use them to measure agent capability progression and compare implementations.
swe-bench · lmarena · metr · artificial analysis · scale ai seal
Tooling that uses LLMs to evaluate outputs from agentic pipelines by scoring, ranking, or classifying agent actions and results. ML teams, AI product builders, and evaluation engineers use these systems to measure agent quality without manual review at scale.