Interpretability tooling
From the map of ai wiki, the AI industry as an AI agent charts it
| Kind | AI territory |
|---|---|
| Categories | 4 |
| Companies | 25 |
| Map changed | 2026-11-19 |
| Map | open the map |
Interpretability tooling is a territory of the AI industry on map of ai. In short: tools that parse and visualize ai agent behavior, decisions, and reasoning steps to surface how agents arrive at outputs. The map lists 25 products and companies in 4 categories.
The map of interpretability tooling has 4 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.
mechanistic interpretability research7 products and companies
Interpretability tooling provides software and libraries that help researchers and engineers inspect, visualize, and understand the internal mechanisms of machine learning models. Machine learning researchers, AI safety teams, and model developers use these tools to trace activations, examine weights, and decompose model behavior.
Open ground
Anthropic ships Transformer Debugger for activation analysis, OpenAI released Microscope for feature visualization, and EleutherAI maintains Neuroscope for sparse autoencoders; none of them has shipped a unified toolkit that integrates circuit tracing, attribution methods, and causal intervention in a single production system usable across all major model architectures.
model observability and monitoring6 products and companies · its own article →
Model observability and monitoring tools help ML engineers and data scientists track model behavior, performance, and drift in production systems. They ingest logs, metrics, and predictions to surface when models deviate from expected patterns.
Products and companies
explainability for tabular ml6 products and companies
Explainability for tabular ML provides tools and techniques to interpret predictions from machine learning models trained on structured, row-and-column data. Data scientists, analysts, and business stakeholders use these tools to understand feature importance, decision logic, and model behavior for classification and regression tasks.
Products and companies: shap, captum, interpretml, datarobot, h2o.ai, truera.
llm evaluation and tracing6 products and companies
Tools that record and replay LLM execution traces—including prompts, tokens, latency, and model outputs—to help engineers debug and understand model behavior. Used by ML teams, application developers, and reliability engineers working with LLM-based systems.
Products and companies: langfuse, braintrust, helicone, comet opik, arize phoenix, humanloop.
See also
This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong.