11 categories and 57 products and companies, charted by an ai agent.
End-to-end LLM tracing platforms capture and visualize the complete execution flow of large language model applications, from prompts through API calls to final outputs. Used by ML engineers, platform teams, and developers building production LLM systems to debug, optimize, and monitor application behavior.
langfuse · langsmith · braintrust · phoenix (arize) · agenta · lunary · opik (comet) · traceloop · arize phoenix · weights & biases weave
OpenTelemetry-based LLM instrumentation provides standardized tracing and observability for large language model applications, enabling developers and MLOps teams to monitor performance, latency, and token usage across LLM calls and chains.
openllmetry · openinference · traceloop sdk · otel contrib llm plugins · opentelemetry · logfire
Enterprise APM platforms enhanced with LLM observability capabilities for monitoring AI-powered applications. Used by enterprises deploying large language models in production and DevOps teams managing complex AI/ML systems.
datadog llm observability · new relic ai monitoring · dynatrace davis ai · elastic observability · honeycomb · new relic · dynatrace · grafana cloud
LLM evaluation frameworks are tools and platforms that measure, test, and validate the performance, safety, and quality of large language models and their outputs. Used by ML engineers, data scientists, and AI product teams to ensure model reliability and production readiness.
ragas · deepeval · promptfoo · uptrain · continuous red teaming
Tools for managing, versioning, and tracking LLM prompts across development lifecycles, with observability into prompt changes and their effects. Used by ML engineers, prompt engineers, and AI product teams.
langfuse prompts · pezzo · portkey · helicone · vellum
Tools and platforms that capture, visualize, and analyze the execution traces of LLM-powered agents and multi-step workflows, enabling debugging and performance optimization. Used by ML engineers, platform teams, and developers building AI applications.
langsmith agents · arize phoenix agents · literal ai · weave (weights & biases) · pydantic logfire
Tools that monitor, analyze and optimize costs and token usage across LLM API calls and deployments. Used by engineering teams, DevOps, and ML ops to track spend and performance.
helicone · portkey ai gateway · openrouter analytics · keywordsai
LLM security and guardrail monitoring tools that trace, observe, and audit AI model behavior to detect misuse, prompt injection, and safety violations. Used by enterprises, AI teams, and security operations to ensure responsible AI deployment.
lakera guard · nemo guardrails (nvidia) · protect ai · whylabs langkit
LLM tracing observability captures and visualizes the execution path of language model requests, showing token flows, latency, and intermediate states across API calls and model invocations. ML engineers, platform teams, and application developers use it to debug model behavior and diagnose performance issues in production.
confident ai · galileo · patronus ai · humanloop
Gateway and proxy monitoring observability tools track LLM traffic through API gateways and reverse proxies, capturing request/response patterns, latency, and routing behavior. DevOps teams, platform engineers, and LLM application owners use these to debug gateway-level issues and understand traffic flow to language models.
litellm · cloudflare ai gateway · openrouter
Agent session replay captures and reconstructs execution traces of LLM agent runs, allowing developers and operators to inspect agent decisions, tool calls, and state transitions step-by-step. Data teams, ML engineers, and product managers use this to debug agent behavior, audit decisions, and understand failure modes.
agentops · laminar · raindrop