Interpretability
Why the model did that — citations, confidence, a trail you can show.
Interpretability is what lets you defend an AI decision to a customer, an auditor or your own team. For an operator it means three things visible in the product: citations to the exact source, a confidence signal you can act on, and an exportable decision trail. Almost no vendor ships all three — the best score across 62 audited tools is a 6.
You need this when
- You make decisions people can dispute: hiring, credit, pricing, claims, account suspension.
- One wrong output costs more than a human check — a few hundred euros or a regulator asking why.
- Volume outruns review: thousands of AI-touched decisions a week and nobody reads them all.
You can skip it when
- Drafting and summarising where a human reads 100% before it ships — instead keep an edit log and track how often the human corrects the machine.
- Pilots under ~50 decisions a week — instead export 20 outputs to a spreadsheet each Friday and grade them yourself; that beats any vendor dashboard at this size.
- A vendor upselling interpretability as 'AI Act compliance' — instead check whether your use case is actually Annex III high-risk before paying for the tier.
The long version — open when you want the depth
What it is — in one coffee-break
Interpretability answers one question after the fact: why did the system do that? In a lab that means tracing internal circuits. In your business it means three surfaces you can show a customer or an auditor: a citation pointing at the exact source, a confidence signal you can act on, and a decision trail you can export.
The trap is the "reasoning" panel: a model that prints its chain of thought looks transparent and is not. Anthropic's own researchers found that when a hint changed the answer, Claude 3.7 Sonnet mentioned that hint only 25% of the time and DeepSeek R1 39%; for a hint framed as unauthorised access, 41% and 19% (Reasoning models don't always say what they think, April 2025). The text is a story about the answer, not a record of how it was produced. Colour, never evidence.
When you actually need it (and when you don't)
You need it when a decision can be disputed — hiring shortlists, credit and pricing calls, claims handling, account suspensions. When volume outruns review: thousands of AI-touched decisions a week nobody reads. And when one wrong output costs more than the check would, call it a few hundred euros per incident.
You do not need it for drafting or summarising where a human reads every output before it leaves the building; keep an edit log and watch how often the human corrects the machine. Nor for a pilot under fifty decisions a week — pull twenty outputs into a spreadsheet each Friday and grade them yourself. And be sceptical when interpretability is sold as "AI Act compliance": most sales, marketing and support tools are not Annex III high-risk, and you are the deployer, not the provider.
How to recognize good vs bad implementations
Good looks like citations that resolve to a sentence, not a 40-page document; confidence with a stated methodology, calibrated against outcomes; a per-decision record of inputs, model version, retrieved documents, overrides and timestamps; a diff view when a score moves. Bad looks like a chat-style thinking pane, a "trust score" with no methodology, explanations written after the fact by a second model, and a transparency page linking to a paper instead of a feature.
Three questions expose the difference in a demo. Show me the exact source sentence behind this claim, inside the app, without a support ticket. Pull up a decision from six weeks ago, show the inputs and model version that produced it, then export it as a file I could send to a customer. This score moved from 62 to 81 — name the fields that changed and how much each contributed. Vendors with real surfaces answer all three in five minutes. Most answer none.
What this costs
From commercial vendors it is rarely a line item; it is bundled into the top tier, so the price is the tier upgrade — typically €20,000 to €60,000 a year for a 50-seat go-to-market stack, plus an audit-log export add-on. Ask for the surface during the trial, not after signature.
The free path is real. Langfuse's self-hosted core is open source and free; its hosted Hobby tier includes 50k units a month at no cost, Core $29/month, Pro $199/month (Langfuse pricing). Self-hosting on a 4 vCPU / 8 GB VPS runs €25–40 a month plus about two days of setup — €1,200 to €2,000 if you buy that time. Open-weight models (DeepSeek, Llama, Qwen) expose token logprobs, a real confidence signal most closed APIs hide. For attribution on your own scoring models, SHAP, LIME and Captum are free. A decision trail you own for under €1,000 in year one.
Where to see it scored
Only 9 of 62 audited tools scored here at all, and the ceiling is low. Gong (6) is the best score on the entire table for Interpretability — nothing reaches 7. Behind it, DeepSeek (4.5), Cognism (4.5) and Claude / Anthropic (4.5) tie. At the bottom, Apollo.io (3) and Decagon (2).
Anthropic's 4.5 is the instructive score. Its lab publishes world-leading interpretability work and almost none reaches you in the console: you cannot attribute an output to features, and there is no per-decision trail to export. The rule is what a vendor ships to customers, not what it publishes — which is why this column is nearly empty. Legal pressure is coming, slowly: Article 13 of the EU AI Act obliges providers of high-risk systems to give deployers what they need to interpret outputs, and after the digital omnibus those duties apply from 2 December 2027 for standalone Annex III systems and 2 August 2028 for embedded ones (enforcement timeline). Until then it is a procurement question, not a compliance one — yours to ask for.
Flashcards
Check yourself
1. A vendor demos a 'reasoning' panel showing the model's step-by-step thinking. What is it worth as an explanation?
2. Anthropic leads the interpretability research field but scores only 4.5. Why?
3. Which is the strongest practical interpretability surface for an ops manager?
4. After the digital omnibus, when do obligations for standalone high-risk (Annex III) systems apply?
5. Cheapest credible way to build a decision trail you own?
Cheat sheet
- Interpretability = citations + stated confidence + exportable decision trail.
- Chain-of-thought is a story, not an explanation. Never accept it as evidence.
- Score what the vendor shows customers, not what its research lab publishes.
- Anthropic scores 4.5: world-class research, thin customer-facing surfaces.
- Demo test: reproduce a 6-week-old decision and export it as a file.
- DIY trail: free Langfuse core + ~EUR 30/mo VPS + two days of setup.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Commure | 7.5 | Better than the category deserves. A toggleable Citation feature shows where each line of the note came from in the transcript, the transcript tab is speaker-separated and time-ali |
| OpenEvidence | 7.5 | Two real customer-facing transparency surfaces: 'Why Was This Source Cited' labels, and EvidenceGrade, whose A–D methodology and its two-phase GRADE-derived scoring are published o |
| Abridge | 7 | Linked Evidence is a real customer-facing attribution surface, and building on Nemotron gives visibility into model weights and training data. It answers 'where did this line come |
| Roark | 7 | Every flag drills down to the turn-by-turn transcript, issues cluster recurring failures across deploys, and human review measures how often each metric agrees with your own labels |
| Bluejay | 6.5 | Every metric returns a reason, red-team runs produce OWASP and content-safety scorecards plus a risk radar, and Issues groups recurring failures with frequency. What is missing is |
| Cekura | 6.5 | Insights auto-clusters failing evals into a ranked list of root causes instead of a pile of logs, the Metric Optimizer surfaces every unresolved case rather than hiding disagreemen |
| Cyara | 6.5 | AI Trust exposes the exact response, the reason it was flagged and the turn-by-turn transcript behind every violation, with violation trends by attack mode, and the Bias module ana |
| Nabla | 6.5 | Nabla publishes a CHAI Applied Model Card v2 in the public CHAI registry, holds a stated EU AI Act transparency position with a dedicated AI governance team, and traces note lines |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| 1mind | 2.5 | No feature attribution, no confidence surfacing, no source citation on answers, no transparency report. The Ethical Stance promises third-party audits and published incident report |
| Suki AI | 2.5 | The transcript is available beside the note and that is the whole attribution story. No feature attribution, no confidence signal, no line-level evidence linking. Publishing evalua |
| Decagon | 2 | Decagon's own glossary correctly defines real interpretability (LIME, SHAP, attention maps) then redirects to its observability tooling, which is tracing, not model-level interpret |
| Freed | 2 | You can read the transcript next to the note, which is the floor for this category. No attribution, no confidence signal, no evidence links, no way to ask why a line was written. |
Scored on 66 of 163 audited tools. Every score links to the full audit and its reasoning.