Frameworks & harnesses
Agent = Model + Harness. The scaffolding decides whether the agent works.
A model answers questions; an agent does work, and everything between the two is harness — tools, the loop, memory, error handling, permissions. LangChain moved 52.8% to 66.5% on Terminal-Bench 2.0 with the model held fixed, purely by changing prompts, tooling and middleware. That is the whole argument: you rarely have a model problem, you have a harness problem.
You need this when
- You have a loop with memory — multi-step work, tool calls, retries — not a single prompt.
- You need audit trails, permission gates and human approval before an agent touches money or customers.
- More than one person maintains the automation, so the logic must live somewhere other than a prompt in someone's head.
You can skip it when
- It is one prompt in, one answer out. Call the model API directly — a framework adds tokens and dependencies for nothing.
- You are prototyping to see if the idea works at all. Write 200 lines yourself first; port to a framework only if it survives.
- The vendor's 'agent builder' is a drag-and-drop wrapper with no export. Use n8n or a plain SDK so your logic stays portable.
The long version — open when you want the depth
What it is — in one coffee-break
A model answers questions. An agent does work. Everything between those two things is the harness: the tool definitions, the loop that decides which tool to call and when to stop, the memory that survives between steps, the retry logic when an API returns a 500, and the permission gate that stops it emailing 4,000 customers at 3am. RXed writes this as Agent = Model + Harness. The element Fw covers both ends of that scaffolding — the heavy end (SDKs, visual builders, orchestration graphs) and the thin end (three hundred lines you wrote yourself). The applicability test is blunt: does the vendor ship a builder, SDK or harness for composing AI systems? A chat box plus an API key does not count.
How much of the outcome actually lives in the harness? LangChain reported moving from 52.8% to 66.5% on Terminal-Bench 2.0 — a jump of 13.7 points — with the model held constant, purely from changes to system prompts, tooling and middleware (explainX, May 2026). Same brain, better cockpit. That is why swapping to a newer model rarely rescues an agent that keeps failing.
When you actually need it (and when you don't)
The useful rule of thumb from practitioners: use a raw SDK for a chain, use a real harness when there is a loop with memory. One document in, one summary out, no state — call the API directly and skip the dependency tree. A quote agent that reads an email, checks stock, prices three options, asks a human to approve and then writes to your ERP — that is a loop with memory, and hand-rolling all of it means rebuilding retries, tracing and permissions badly.
Be honest about churn. Heavy frameworks age fast. The 12-factor agents argument is that teams hit roughly 70-80% quality quickly, then have to reverse-engineer the framework's own prompts and control flow to get further — and often rewrite. Abstraction is not free either: one review found a hand-written RAG pipeline used about 2.7x fewer tokens than the LangChain equivalent for the same task (agentailor, July 2026). A thin harness plus a good model beats a fat framework more often than vendors admit.
How to recognize good vs bad implementations
Good harnesses make the loop visible and the failures boring. You can see every tool call, its arguments, its result and how many times it retried. Tools are narrow and typed, not one do_anything() function. Permissions are explicit — some actions run automatically, some require a human click. Context is managed, not just appended until it overflows. The Claude Agent SDK is a fair reference shape: built-in tools, hooks, subagents, sessions, MCP connections and a permission layer, all inspectable.
Three questions that expose a weak demo. Show me the full trace of one run — every tool call, every retry, every token. Break a tool on purpose right now: what does the agent do when the CRM returns a 500? Export my agent's logic and show me what I keep if I cancel next month. A vendor who can only replay a happy path, or whose logic cannot leave their UI, is selling you a demo and a lock-in, not a harness.
What this costs
Open-source and self-hosted is genuinely cheap. n8n Community Edition is free software with unlimited executions; a suitable VPS runs roughly $4-7 (about €4-6) per month (No Code MBA, June 2026). You pay in your own sysadmin time and in model tokens. n8n Cloud is $20/month annual for 2,500 executions, $50/month for 10,000, and $667/month for 40,000 — and when you hit the cap, workflows stop immediately. SDKs like the Claude Agent SDK and LangGraph cost nothing as libraries; you pay for tokens and for observability (LangSmith or Langfuse).
The managed end is a different order of magnitude. Sierra runs custom enterprise contracts with annual commitments around $150,000 plus $50,000-$200,000 implementation and three to seven month deployments; Decagon is roughly a $50,000 platform fee plus about $0.50 per resolution; Fin charges $0.99 per resolution with no platform fee; Agentforce charges $2.00 per conversation regardless of outcome (witn, June 2026). For a 30-person company, the honest path is self-hosted n8n or a thin SDK harness at under €50/month all-in, and only move up when volume or compliance forces it.
Where to see it scored
RXed audited 62 tools, 57 of them scored on this element. The top of the table is Sierra (9), Gemini / Google (9) and Claude / Anthropic (9), with n8n (8.5) just behind and by far the cheapest way in. The bottom is where the element does its real work: Cognism (2.5) and Midjourney (2) are fine products that ship almost no composable harness — you cannot build a system on them, only use them. That is not a criticism of the tool, it is a warning about the architecture diagram you were about to draw.
Flashcards
Check yourself
1. LangChain's Terminal-Bench 2.0 jump from 52.8% to 66.5% came from what?
2. You need a single summary generated from one document, once. Best choice?
3. According to the 12-factor agents argument, where do frameworks typically fail?
4. Roughly what does self-hosted n8n Community Edition cost to run?
5. Which is the strongest signal of a bad harness in a vendor demo?
Cheat sheet
- Agent = Model + Harness. Swapping models rarely fixes a broken harness.
- Chain? Raw SDK. Loop with memory? Framework or a deliberate thin harness.
- Frameworks get you to 70-80%; the last 20% means reverse-engineering them.
- Same model, better harness: +13.7 pts on Terminal-Bench 2.0 (LangChain, 2026).
- Heavy abstraction costs tokens — one hand-written RAG used ~2.7x fewer.
- n8n self-hosted: free software, €4-7/mo VPS, unlimited runs. Start there.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Amazon Bedrock AgentCore | 9 | Deliberately framework-agnostic: CrewAI, LangGraph, LlamaIndex, Google ADK, OpenAI Agents SDK, Strands and custom code all deploy, in Python and — for Evaluations since September 2 |
| Bland | 9 | The harness is unusually complete for a voice vendor: REST API, a CLI that treats pathways as YAML with push, pull, diff, validate and watch, Node SDK, remote and local MCP servers |
| Claude / Anthropic | 9 | Agent SDK (Python/TS) with hooks, subagents, skills and plugins, plus the hosted Managed Agents harness — Anthropic's explicit thesis is that the harness is the product. |
| Composio | 9 | TypeScript and Python SDKs under MIT, provider adapters for OpenAI, OpenAI Agents, Anthropic, Claude Agent SDK, Vercel AI SDK, Google GenAI, Google ADK, LangChain, LangGraph, Llama |
| Gemini / Google | 9 | The open-source, model-agnostic Agent Development Kit plus Agent Engine, Agent Garden samples and an Agents CLI — a full build-deploy-govern harness. |
| LangGraph | 9 | It is the harness — nodes, edges, conditional routing, subgraphs, streaming, persistence — with Deep Agents layered on top for planning and subagents. Held back from higher only be |
| LangSmith | 9 | The widest harness surface in this audit set: LangChain, LangGraph and deepagents open source, Python, TypeScript and Java SDKs, a CLI, Studio, the Assistants API, cron scheduling, |
| Promptfoo | 9 | This is a harness by design: CLI, Node library, declarative YAML, CI/CD integrations for GitHub, GitLab and Jenkins, code scanning in pull requests, and Docker/Helm self-hosting. |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| Rilla | 2 | No public API reference, no SDK, no Zapier, Make or n8n connector. CRM coverage (Salesforce, HubSpot, Zoho, Pipedrive, Dynamics) is built on Merge's unified API rather than exposed |
| Superhuman Mail | 2 | No SDK or agent-builder harness beyond the MCP tool surface (scored under Protocols); Superhuman Mail is not a platform other developers build AI systems on. |
| OpenEvidence | 1.5 | No public API, no SDK, no developer portal, no self-serve integration path. Building on OpenEvidence means negotiating an enterprise Epic deployment, which is why only a handful of |
| Freed | 1 | No public API, no SDK, no partner developer platform. The apis.io provider profile lists zero APIs. Freed argues the position openly in its own writing, that the future is not anot |
Scored on 157 of 163 audited tools. Every score links to the full audit and its reasoning.