Tracing
Flight recorder for your AI: what it did, which tools, how long, what it cost.
AI systems fail politely: a confident, well-formed, wrong answer, and nothing crashes. Tracing is the only record that shows the wrong document was retrieved, the same tool was called five times, or a cheaper model quietly took over. Without it every incident is reconstructed from memory; with it, it is a timeline you read in minutes.
You need this when
- The system runs unattended — a nightly pipeline, a customer-facing assistant, an agent with access to your systems.
- Someone will ask you why it did that: a customer, a manager, an auditor. You need the answer in minutes, not guesses.
- You pay per token and cannot see which workflow, customer or prompt is burning the budget.
You can skip it when
- You are experimenting in a chat window — the conversation is the trace; export the thread if a result matters.
- One-off generation where a human reads every output before it ships — spot-check the outputs instead of instrumenting.
- Your tool already replays every run step by step (n8n scores 8 on exactly this) — use the built-in execution log, don't bolt on a second stack.
The long version — open when you want the depth
What it is — in one coffee-break
Tracing means recording what your AI system actually did: which model was called, with what input, what it answered, which tools it used, how long it took and what it cost. A trace is the flight recorder for an AI workflow. Without it, an AI failure is a mystery you reconstruct from memory; with it, it's a timeline you read in minutes.
AI systems need this more than normal software because they fail politely: an agent produces a well-formed, confident, wrong answer, and nothing crashes. The 2026 observability guides put it plainly — agents fail in ways that look like success. Only the trace shows that the model retrieved the wrong document, or called the same tool five times, or silently fell back to a cheaper model.
When you actually need it (and when you don't)
Experimenting in a chat window? You don't. The moment an AI system runs unattended — a nightly pipeline, a customer-facing assistant, an agent with access to your systems — tracing stops being optional. It's how you answer the three questions every operator eventually asks: why did it do that, what did this month cost, and is quality drifting?
The good news: this has standardized. OpenTelemetry — the same open standard normal software uses for monitoring — now has GenAI conventions covering LLM calls, agent steps and tool executions, with MCP tool-call tracing added in 2026. Open-source tools like Langfuse (28,000+ GitHub stars, self-hostable, acquired by ClickHouse in January 2026) and Phoenix give small teams the whole stack for free. The bad news, per a Gartner figure in the same research: only about 15% of AI deployments instrument observability at all. The other 85% are flying blind.
How to recognize good vs bad implementations
In the audits, Tracing (Tr) is where platform maturity shows. Gemini scores 8.5 and Microsoft 365 Copilot 8 — enterprise stacks with real audit trails. n8n scores 8: every workflow execution is replayable step by step, which is exactly what a small business needs when an automation misbehaves. The other end is instructive: DeepSeek scores 2 (an engine with almost nothing around it) and consumer creative tools like Suno (3.5) offer effectively no visibility. The buyer's question: when this tool does something wrong, what exactly will you be able to show me?
What this costs
Self-hosted Langfuse or Phoenix: €0 in licenses, a small server, and an afternoon of setup — the same shape as the automations in our Blueprints. Managed options run roughly €0–50/month at small-business volume. Compare that with one undiagnosed bad week of a customer-facing assistant, and tracing is the cheapest insurance in the stack.
Where to see it scored
Tr scores with reasons: n8n (8), Gemini (8.5), Claude (8), DeepSeek (2). Tracing is also what makes the next discipline possible: evaluating whether outputs are any good (Ev — covered later in the track).
Flashcards
Check yourself
1. Your assistant gives a customer a confident, completely wrong answer. Nothing errored. What tells you why?
2. Which open standard now covers LLM calls, agent steps and tool executions?
3. In the RXed audits (62 tools scored), which pair sits at the bottom on Tracing?
4. Roughly what share of GenAI deployments instrument observability at all?
5. What do self-hosted Langfuse or Phoenix cost in licence fees?
Cheat sheet
- A trace = input, model, tool calls, output, latency, cost — one run, end to end.
- Agents fail looking like success; only step-level traces catch that.
- OpenTelemetry GenAI conventions are the standard; MCP tool calls covered since 2026.
- Langfuse and Phoenix: free, self-hostable. Managed tiers ~EUR 0-50/month at SMB volume.
- About 15% of GenAI deployments instrument observability at all. The rest fly blind.
- Buyer's question: when this tool gets it wrong, what exactly can you show me?
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Bland | 9 | Call logs are a genuine debugging surface, not a list. Every call carries full transcript, recording, extracted variables with their originating node, per-turn decision data includ |
| Braintrust | 9 | The core of the product. Typed nested spans for tool calls, reasoning steps, state and memory, SDK wrappers for more than 20 agent frameworks, an OpenTelemetry span processor, and |
| Coval | 9 | OpenTelemetry-native with waterfall and flame-graph viewers, Transition Hotspots across a whole run, natural-language Trace Search, SIP-header linking of production calls to traces |
| LangSmith | 9 | The core product and the best-instrumented one here. Native OTLP ingestion from any OpenTelemetry app with documented attribute mapping for GenAI, TraceLoop, OpenInference and Logf |
| Langfuse | 9 | Both SDKs are built natively on OpenTelemetry, so it works with any language or framework that speaks OTel, plus 100+ direct integrations, agent graphs for multi-step workflows, se |
| Portkey | 9 | The strongest element. OpenTelemetry-compliant tracing, full request and response logs, 21+ metrics, custom metadata tags, filters, a feedback API with weighted scores, per-request |
| Tines | 9 | Every action emits an event; each AI run returns input and output tokens, credits used, model and duration in its payload; audit logs are kept two years and export to S3 every 15 m |
| ASAPP | 8.5 | ASAPP monitors 100% of GenerativeAgent responses in production with human annotation on top. Operators see why each conversation was flagged, can read the agent's thoughts and acti |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| Submagic | 3 | Dashboard shows per-video status and aggregate processing counts only; no run logs, decision traces or per-clip cost attribution — reviewers asking for clip-by-clip reporting can't |
| Udio | 3 | Generation history and Sessions snapshots exist, but there are no run logs, trace trees, cost attribution, or audit trails for operators. |
| Beautiful.ai | 2.5 | Enterprise ships basic audit and change-logs of user actions and version history, but there's no AI-specific run log, trace view, or lineage for what the generator did or why. |
| DeepSeek | 2 | Usage and balance reporting only; no run tracing, no logs surface — observability is entirely bring-your-own. |
Scored on 143 of 144 audited tools. Every score links to the full audit and its reasoning.