Interpretability
Why the model did that — citations, confidence, a trail you can show.
Interpretability is what lets you defend an AI decision to a customer, an auditor or your own team. For an operator it means three things visible in the product: citations to the exact source, a confidence signal you can act on, and an exportable decision trail. Almost no vendor ships all three — the best score across 62 audited tools is a 6.
You need this when
- You make decisions people can dispute: hiring, credit, pricing, claims, account suspension.
- One wrong output costs more than a human check — a few hundred euros or a regulator asking why.
- Volume outruns review: thousands of AI-touched decisions a week and nobody reads them all.
You can skip it when
- Drafting and summarising where a human reads 100% before it ships — instead keep an edit log and track how often the human corrects the machine.
- Pilots under ~50 decisions a week — instead export 20 outputs to a spreadsheet each Friday and grade them yourself; that beats any vendor dashboard at this size.
- A vendor upselling interpretability as 'AI Act compliance' — instead check whether your use case is actually Annex III high-risk before paying for the tier.
The long version
What it is — in one coffee-break
Interpretability answers one question after the fact: why did the system do that? In a lab that means tracing internal circuits. In your business it means three surfaces you can show a customer or an auditor: a citation pointing at the exact source, a confidence signal you can act on, and a decision trail you can export.
The trap is the "reasoning" panel: a model that prints its chain of thought looks transparent and is not. Anthropic's own researchers found that when a hint changed the answer, Claude 3.7 Sonnet mentioned that hint only 25% of the time and DeepSeek R1 39%; for a hint framed as unauthorised access, 41% and 19% (Reasoning models don't always say what they think, April 2025). The text is a story about the answer, not a record of how it was produced. Colour, never evidence.
When you actually need it (and when you don't)
You need it when a decision can be disputed — hiring shortlists, credit and pricing calls, claims handling, account suspensions. When volume outruns review: thousands of AI-touched decisions a week nobody reads. And when one wrong output costs more than the check would, call it a few hundred euros per incident.
You do not need it for drafting or summarising where a human reads every output before it leaves the building; keep an edit log and watch how often the human corrects the machine. Nor for a pilot under fifty decisions a week — pull twenty outputs into a spreadsheet each Friday and grade them yourself. And be sceptical when interpretability is sold as "AI Act compliance": most sales, marketing and support tools are not Annex III high-risk, and you are the deployer, not the provider.
How to recognize good vs bad implementations
Good looks like citations that resolve to a sentence, not a 40-page document; confidence with a stated methodology, calibrated against outcomes; a per-decision record of inputs, model version, retrieved documents, overrides and timestamps; a diff view when a score moves. Bad looks like a chat-style thinking pane, a "trust score" with no methodology, explanations written after the fact by a second model, and a transparency page linking to a paper instead of a feature.
Three questions expose the difference in a demo. Show me the exact source sentence behind this claim, inside the app, without a support ticket. Pull up a decision from six weeks ago, show the inputs and model version that produced it, then export it as a file I could send to a customer. This score moved from 62 to 81 — name the fields that changed and how much each contributed. Vendors with real surfaces answer all three in five minutes. Most answer none.
What this costs
From commercial vendors it is rarely a line item; it is bundled into the top tier, so the price is the tier upgrade — typically €20,000 to €60,000 a year for a 50-seat go-to-market stack, plus an audit-log export add-on. Ask for the surface during the trial, not after signature.
The free path is real. Langfuse's self-hosted core is open source and free; its hosted Hobby tier includes 50k units a month at no cost, Core $29/month, Pro $199/month (Langfuse pricing). Self-hosting on a 4 vCPU / 8 GB VPS runs €25–40 a month plus about two days of setup — €1,200 to €2,000 if you buy that time. Open-weight models (DeepSeek, Llama, Qwen) expose token logprobs, a real confidence signal most closed APIs hide. For attribution on your own scoring models, SHAP, LIME and Captum are free. A decision trail you own for under €1,000 in year one.
Where to see it scored
Only 9 of 62 audited tools scored here at all, and the ceiling is low. Gong (6) is the best score on the entire table for Interpretability — nothing reaches 7. Behind it, DeepSeek (4.5), Cognism (4.5) and Claude / Anthropic (4.5) tie. At the bottom, Apollo.io (3) and Decagon (2).
Anthropic's 4.5 is the instructive score. Its lab publishes world-leading interpretability work and almost none reaches you in the console: you cannot attribute an output to features, and there is no per-decision trail to export. The rule is what a vendor ships to customers, not what it publishes — which is why this column is nearly empty. Legal pressure is coming, slowly: Article 13 of the EU AI Act obliges providers of high-risk systems to give deployers what they need to interpret outputs, and after the digital omnibus those duties apply from 2 December 2027 for standalone Annex III systems and 2 August 2028 for embedded ones (enforcement timeline). Until then it is a procurement question, not a compliance one — yours to ask for.
Flashcards
Check yourself
1. A vendor demos a 'reasoning' panel showing the model's step-by-step thinking. What is it worth as an explanation?
2. Anthropic leads the interpretability research field but scores only 4.5. Why?
3. Which is the strongest practical interpretability surface for an ops manager?
4. After the digital omnibus, when do obligations for standalone high-risk (Annex III) systems apply?
5. Cheapest credible way to build a decision trail you own?
Cheat sheet
- Interpretability = citations + stated confidence + exportable decision trail.
- Chain-of-thought is a story, not an explanation. Never accept it as evidence.
- Score what the vendor shows customers, not what its research lab publishes.
- Anthropic scores 4.5: world-class research, thin customer-facing surfaces.
- Demo test: reproduce a 6-week-old decision and export it as a file.
- DIY trail: free Langfuse core + ~EUR 30/mo VPS + two days of setup.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Gong | 6 | ISO 42001 certification, a dedicated AI governance team, a published AI whitepaper in the Trust Center, stated human-validation and model-oversight practice, and per-agent audit tr |
| Attio | 4.5 | Research Agent answers come back with citations and the MCP audit log shows exactly what an agent read or wrote, but there is no explanation of how AI Attributes derive a classific |
| Claude / Anthropic | 4.5 | The research leader in interpretability and it open-sourced circuit-tracing tooling, but none of that is exposed as a production surface on Claude models themselves. |
| Cognism | 4.5 | Diamond Data confidence scores expose per-record data-quality transparency, but there is no explanation of how AI-surfaced insights, ICP fit or generated personas are derived. |
| DeepSeek | 4.5 | Raw chain-of-thought exposure — which R1 pioneered and rivals later copied — is a real transparency surface for users, though there is nothing deeper: no attribution or model-inter |
| Sierra | 4 | Decision traces and audit logs (see Tracing) let you see what an agent did, but there's no feature-attribution or model-explainability surface beyond that operational logging. |
| Apollo.io | 3 | A published AI Policy and cost-confirmation prompts are the extent of it; AI lead scores and AI research fields arrive without an explanation of how they were derived. |
| Manus | 3 | Transparent reasoning shows plan and steps in real time, and users can intervene — but no feature attribution, transparency reports or model-level interpretability surfaces exposed |
Scored on 9 of 62 audited tools. Every score links to the full audit and its reasoning.