Evaluations
If you can't measure it, you're shipping prompt changes on vibes.
Every prompt tweak, model swap or knowledge-base edit changes output quality in ways nobody can eyeball reliably. Evals give you a repeatable number so you can tell improvement from regression before customers do. Without them you are guessing, and the vendor's benchmark deck is measuring their data, not yours.
You need this when
- Output goes to customers, touches money, or gets changed more than once.
- You want to swap to a cheaper model and prove quality held. Evals are the proof.
- You are regulated: high-risk AI obligations expect documented testing evidence, not anecdotes.
You can skip it when
- A human reads every output before it ships — do spot-checks and a weekly 10-trace read instead.
- Two-week throwaway pilot — do error analysis instead: read 50 outputs, write down what broke.
- No product yet — build the workflow first; you cannot measure a process nobody has run.
The long version
What it is — in one coffee-break
Evaluations answer one question: did that change make the output better or worse? You edit a prompt, swap a model for a cheaper one, drop ten documents into the knowledge base. Without evals you have an opinion. With evals you have a number you can defend in a meeting.
The discipline rests on one artifact: real cases with known-good answers. Thirty rows is enough. Column A is an input a real customer sent, column B the output you would be happy to ship, column C what the system produced today. Run the set before a change and after, then count the rows that got worse. That is a regression test, pointed at text instead of code.
Four things get lumped in here. Benchmarks are public scoreboards: fine for a shortlist, useless for predicting how a tool handles your invoices. Regression suites re-run your own cases on every change. Red-teaming (folded in at v1.0) is the adversarial half: trying to make the system leak data, insult a customer, or approve a refund it should not. Graders turn "looks fine to me" into pass/fail.
When you actually need it (and when you don't)
You need evals the moment output quality carries a cost: anything customer-facing, anything touching money, anything you intend to change twice. If you cannot say whether last month's version beat this month's, you are flying on vibes. Regulated work adds a second reason: high-risk obligations assume you can produce evidence of testing, and "it seemed good" is not evidence.
Skip formal evals when a human reads every output before it leaves the building; a drafting assistant used by two people is checked by definition. Skip them for a two-week throwaway pilot and spend the time on error analysis instead: read fifty real outputs, write down what went wrong, and you will know what to measure later. And do not build a harness before you have a product; you cannot measure a workflow nobody has run.
How to recognize good vs bad implementations
Good implementations let you bring your own data. You upload your thirty cases, run them against two configurations side by side, and see a per-row diff rather than one aggregate score. They version the dataset and let you promote a production failure into the test set in one click. And they surface disagreement between the automatic grader and human labels, because a model-as-judge nobody checked is a second opinion with the same blind spots.
Bad implementations show one confidence percentage with no way to open the underlying cases, or a wall of generic metrics ("helpfulness 4.2") nobody chose and nobody can act on. Worst is the vendor reporting accuracy on their own curated set and letting you hear it as your accuracy.
Three questions that expose a demo: Can I upload thirty of my own examples right now and see results row by row? When I change the prompt, do you show me a diff of which cases moved, or one number? How do you check that your grader agrees with a human, and what is that agreement rate on my data? A vendor who deflects the third is selling a scoreboard, not a measurement.
What this costs
The spreadsheet path costs time only. Assembling thirty cases from real tickets takes half a day; a human-graded re-run takes two hours. A model-as-judge pass over 30 cases in two configurations costs under €1 on a mid-tier model. Open-source tooling is free to licence: promptfoo runs from a YAML file on your laptop, and Langfuse self-hosts via Docker or Helm at €0 licence plus roughly €20-40 a month for the server.
Hosted tooling starts cheap. Langfuse Cloud's free tier includes 50,000 units a month; Core is $29/month with 100,000 units and $8 per extra 100,000. LangSmith is free for one seat and 5,000 base traces a month, then $39 per seat per month with 10,000 base traces on Plus. Braintrust's Team plan is $249/month. That gap only pays for itself if the pricier tool catches roughly two prompt regressions a quarter. Adversarial testing is the expensive end: outsourced red-teaming engagements average around $16,000, so most operators automate the obvious attacks and buy an engagement only ahead of a high-risk launch.
Where to see it scored
Ev is where the audit spread is widest. Sierra (9.5) leads: eval sets built from your own conversations, regression runs gating every agent change. Decagon (9) sits just behind. Glean (8.5) and Gemini / Google (8.5) score high on repeatable checks you control rather than ones they publish. At the bottom, Descript (2) and DeepSeek (2) give you nothing: you ship a change and hear about it from customers. Across 62 tools audited and 51 scored here, that bottom case is more common than the marketing suggests.
Flashcards
Check yourself
1. What is the minimum viable eval setup for a small operator?
2. LangSmith's Plus plan costs how much per seat per month?
3. Which demo question best exposes a fake eval feature?
4. When is it reasonable to skip a formal eval suite?
5. Roughly what does an outsourced red-teaming engagement cost?
Cheat sheet
- 30 real cases in a spreadsheet beats any vendor benchmark deck.
- Run the same set before and after every prompt or model change.
- Count rows that got worse. That number is your regression test.
- Vendor accuracy is measured on vendor data, never on your invoices.
- Check grader-vs-human agreement, or your judge is just another opinion.
- Free path: promptfoo or self-hosted Langfuse, EUR 0 licence, ~EUR 20-40/mo server.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Sierra | 9.5 | Sierra Research built and maintains τ-bench/τ²-bench, the industry-standard agent evaluation benchmark, on top of an internal Experience Manager that runs daily human-annotated eva |
| Decagon | 9 | A two-phase framework — offline LLM-as-judge scoring against human-labeled ground truth, then staged online A/B testing with real statistical significance — plus a dedicated Experi |
| ElevenLabs | 8.5 | The agent testing framework is best-in-class for the category: scenario tests with LLM evaluators, tool-call tests, multi-turn simulation with tool mocking, probabilistic pass-rate |
| Gemini / Google | 8.5 | Gen AI evaluation service (covering partner models too), offline and simulated agent evaluations, online production monitoring and an automated Agent Optimizer. |
| Glean | 8.5 | Customers get a documented framework for building golden test sets across four failure categories before agent launch, on top of an internal LLM-as-judge pipeline with a stated 79% |
| n8n | 7.5 | Native Evaluations: datasets, ground-truth comparisons, LLM-as-judge metrics and an Evaluations tab for spotting regressions across model or prompt changes — rare at this price poi |
| Clay | 7 | Free test runs on up to 10 real inputs, side-by-side model comparison, saved reusable test suites and version rollback make a real pre-deployment eval loop. |
| Lindy AI | 7 | Offline evals are documented as a first-class feature alongside a step-by-step test panel — genuinely rare at the SMB no-code tier. |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| HeyGen | 2 | No systematic output-quality measurement surface; you eyeball results — thin for a tool pitched at Fortune-100-scale localization. |
| Luma AI | 2 | No evaluation suites, benchmark tools, or systematic quality measurement surfaces; users rely on manual iteration and visual judgment. |
| Make (formerly Integromat) | 2 | No evaluation or regression-testing surface for AI outputs at all — missing but expected for a platform selling production AI agents; you test by running scenarios. |
| Udio | 2 | No systematic evaluation, benchmarking, or regression testing tools; users must subjectively assess output quality with no structured way to measure consistency. |
Scored on 51 of 62 audited tools. Every score links to the full audit and its reasoning.