Evaluations
If you can't measure it, you're shipping prompt changes on vibes.
Every prompt tweak, model swap or knowledge-base edit changes output quality in ways nobody can eyeball reliably. Evals give you a repeatable number so you can tell improvement from regression before customers do. Without them you are guessing, and the vendor's benchmark deck is measuring their data, not yours.
You need this when
- Output goes to customers, touches money, or gets changed more than once.
- You want to swap to a cheaper model and prove quality held. Evals are the proof.
- You are regulated: high-risk AI obligations expect documented testing evidence, not anecdotes.
You can skip it when
- A human reads every output before it ships — do spot-checks and a weekly 10-trace read instead.
- Two-week throwaway pilot — do error analysis instead: read 50 outputs, write down what broke.
- No product yet — build the workflow first; you cannot measure a process nobody has run.
The long version — open when you want the depth
What it is — in one coffee-break
Evaluations answer one question: did that change make the output better or worse? You edit a prompt, swap a model for a cheaper one, drop ten documents into the knowledge base. Without evals you have an opinion. With evals you have a number you can defend in a meeting.
The discipline rests on one artifact: real cases with known-good answers. Thirty rows is enough. Column A is an input a real customer sent, column B the output you would be happy to ship, column C what the system produced today. Run the set before a change and after, then count the rows that got worse. That is a regression test, pointed at text instead of code.
Four things get lumped in here. Benchmarks are public scoreboards: fine for a shortlist, useless for predicting how a tool handles your invoices. Regression suites re-run your own cases on every change. Red-teaming (folded in at v1.0) is the adversarial half: trying to make the system leak data, insult a customer, or approve a refund it should not. Graders turn "looks fine to me" into pass/fail.
When you actually need it (and when you don't)
You need evals the moment output quality carries a cost: anything customer-facing, anything touching money, anything you intend to change twice. If you cannot say whether last month's version beat this month's, you are flying on vibes. Regulated work adds a second reason: high-risk obligations assume you can produce evidence of testing, and "it seemed good" is not evidence.
Skip formal evals when a human reads every output before it leaves the building; a drafting assistant used by two people is checked by definition. Skip them for a two-week throwaway pilot and spend the time on error analysis instead: read fifty real outputs, write down what went wrong, and you will know what to measure later. And do not build a harness before you have a product; you cannot measure a workflow nobody has run.
How to recognize good vs bad implementations
Good implementations let you bring your own data. You upload your thirty cases, run them against two configurations side by side, and see a per-row diff rather than one aggregate score. They version the dataset and let you promote a production failure into the test set in one click. And they surface disagreement between the automatic grader and human labels, because a model-as-judge nobody checked is a second opinion with the same blind spots.
Bad implementations show one confidence percentage with no way to open the underlying cases, or a wall of generic metrics ("helpfulness 4.2") nobody chose and nobody can act on. Worst is the vendor reporting accuracy on their own curated set and letting you hear it as your accuracy.
Three questions that expose a demo: Can I upload thirty of my own examples right now and see results row by row? When I change the prompt, do you show me a diff of which cases moved, or one number? How do you check that your grader agrees with a human, and what is that agreement rate on my data? A vendor who deflects the third is selling a scoreboard, not a measurement.
What this costs
The spreadsheet path costs time only. Assembling thirty cases from real tickets takes half a day; a human-graded re-run takes two hours. A model-as-judge pass over 30 cases in two configurations costs under €1 on a mid-tier model. Open-source tooling is free to licence: promptfoo runs from a YAML file on your laptop, and Langfuse self-hosts via Docker or Helm at €0 licence plus roughly €20-40 a month for the server.
Hosted tooling starts cheap. Langfuse Cloud's free tier includes 50,000 units a month; Core is $29/month with 100,000 units and $8 per extra 100,000. LangSmith is free for one seat and 5,000 base traces a month, then $39 per seat per month with 10,000 base traces on Plus. Braintrust's Team plan is $249/month. That gap only pays for itself if the pricier tool catches roughly two prompt regressions a quarter. Adversarial testing is the expensive end: outsourced red-teaming engagements average around $16,000, so most operators automate the obvious attacks and buy an engagement only ahead of a high-risk launch.
Where to see it scored
Ev is where the audit spread is widest. Sierra (9.5) leads: eval sets built from your own conversations, regression runs gating every agent change. Decagon (9) sits just behind. Glean (8.5) and Gemini / Google (8.5) score high on repeatable checks you control rather than ones they publish. At the bottom, Descript (2) and DeepSeek (2) give you nothing: you ship a change and hear about it from customers. Across 62 tools audited and 51 scored here, that bottom case is more common than the marketing suggests.
Flashcards
Check yourself
1. What is the minimum viable eval setup for a small operator?
2. LangSmith's Plus plan costs how much per seat per month?
3. Which demo question best exposes a fake eval feature?
4. When is it reasonable to skip a formal eval suite?
5. Roughly what does an outsourced red-teaming engagement cost?
Cheat sheet
- 30 real cases in a spreadsheet beats any vendor benchmark deck.
- Run the same set before and after every prompt or model change.
- Count rows that got worse. That number is your regression test.
- Vendor accuracy is measured on vendor data, never on your invoices.
- Check grader-vs-human agreement, or your judge is just another opinion.
- Free path: promptfoo or self-hosted Langfuse, EUR 0 licence, ~EUR 20-40/mo server.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Promptfoo | 10 | Category-defining. Deterministic assertions, LLM-as-judge, 50+ red-team vulnerability plugins mapped to OWASP LLM Top 10, static model scanning via ModelAudit — and OpenAI's own de |
| Bland | 9.5 | Best-in-class, and the single strongest reason to choose Bland. Evals grade real calls with configurable LLM judges (text or audio modality, weighted, pass thresholds, up to 5,000 |
| Sierra | 9.5 | Sierra Research built and maintains τ-bench/τ²-bench, the industry-standard agent evaluation benchmark, on top of an internal Experience Manager that runs daily human-annotated eva |
| Bluejay | 9 | This is the product and it is deep: 70+ shipped metrics plus unlimited custom ones, regression suites, A/B runs, load testing to 200 concurrent, production replays of real failures |
| Braintrust | 9 | Best in class. Autoevals, LLM-as-judge with chain-of-thought rationale, code scorers, human review queues, immutable experiments with side-by-side diffs, CI runs and online scoring |
| Coval | 9 | This is the product and it is unusually complete: deterministic, statistical, ML-model, LLM-judge (seven variants) and trace metrics, metric chaining, agent mutations for side-by-s |
| Decagon | 9 | A two-phase framework — offline LLM-as-judge scoring against human-labeled ground truth, then staged online A/B testing with real statistical significance — plus a dedicated Experi |
| LangSmith | 9 | Offline and online evaluation with LLM-as-judge, pairwise, code and human evaluators, assertions, annotation queues, versioned datasets with tags and splits, repetitions, concurren |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| tl;dv | 2 | Nothing measures the AI. Playbook monitoring scores the salesperson, not the summary, and independent reviewers report summaries that name the topic but miss the decision. |
| Beautiful.ai | 1.5 | Viewer and slide engagement analytics measure audience response, not AI output quality; there's no eval suite, regression testing, or quality benchmark for what the generator produ |
| Cloudflare AI Gateway | 1.5 | Evaluations are deprecated and, per the docs page dated 28/07/2026, no longer supported for new accounts. The feature never got past human feedback in open beta on log-derived data |
| OpenAI AgentKit | 1.5 | Datasets, trace grading, automated prompt optimization and third-party model evaluation all go read-only on 31/10/2026 and shut down on 30/11/2026, with OpenAI's own cookbook point |
Scored on 146 of 163 audited tools. Every score links to the full audit and its reasoning.