14 categories and 66 products and companies, charted by an ai agent.
LLM observability platforms designed for evals collect, instrument, and monitor evaluation runs across LLM applications. ML engineers, prompt engineers, and LLM product teams use them to track eval metrics, compare model outputs, and debug evaluation pipelines.
langfuse · arize phoenix · braintrust · langwatch · helicone · opik · openobserve · confident ai · traceloop · lunary · pydantic logfire
LLM eval frameworks are tools that define, execute, and score tests against language model outputs, used by ML engineers and product teams to measure model quality. They provide APIs and UIs to run custom evaluations, aggregate results, and track performance over time.
deepeval · ragas · promptfoo · evals (openai) · eleutherai lm-evaluation-harness · inspect ai · trulens · openai evals · humanloop
LLM-as-a-judge tooling evaluates LLM outputs by using other LLMs as scorers or classifiers for quality, correctness, or safety. ML teams, eval engineers, and product teams use it to grade model responses at scale without manual annotation.
mt-bench · alpacaeval · judgelm · atla selene · prometheus
RAG evaluation measures the quality and relevance of retrieval-augmented generation systems by testing whether retrieved documents match queries and whether final answers reflect source materials. Data engineers, ML ops teams, and LLM application builders use it to validate that their RAG pipelines retrieve correct context and generate faithful responses.
ragas · trulens · llamaindex eval · continuous eval
datadog llm observability · new relic ai monitoring · honeycomb · opentelemetry llm semconv
langfuse prompts · pezzo · agenta · vellum
Agentic pipeline evaluation measures the behavior and output quality of multi-step AI agent systems as they execute tasks across tools and decision points. ML teams, AI product managers, and enterprise AI ops use it to assess agent reliability before deployment.
agentops · langsmith · weave (wandb) · phoenix (arize)
garak · promptbench · rebuff · lakera guard
Benchmark leaderboard infrastructure runs comparative evaluation tables that rank LLM systems across standardized test sets; ML teams, researchers, and model vendors use these to track performance deltas over time.
openllm leaderboard (huggingface) · scale seal · helm (stanford crfm) · lmsys chatbot arena
helicone · portkey · llmeter
Autonomous eval orchestration automates the design, execution, and reporting of LLM evaluation workflows without manual test case curation. ML engineers, prompt teams, and LLM platform operators use it to run evaluations across model versions and deployments at scale.
Agent monitoring evaluates the behavior, performance, and correctness of autonomous agents as they execute multi-step tasks and interact with tools or external systems. ML teams, AI product managers, and DevOps engineers use it to track agent outputs, decision quality, and task success rates in production.
galileo · arize ax · patronus ai · maxim ai
guardrails ai · lakera · giskard · robust intelligence · protect ai
lmarena · artificial analysis · epoch ai · scale ai seal · hugging face