Small models
The cheap rung of the model ladder — and the one that runs on your own box.
Small models are how you stop paying flagship prices for work that never needed a flagship: the gap inside a single vendor's catalogue runs from 5x to 50x. They are also the only realistic route to AI on hardware you already own, because quantisation squeezes an 8B model down to about 5 GB. The catch is real — they break first on long-context reasoning and multi-step tool calling, and they break quietly.
You need this when
- High-volume narrow tasks — ticket tagging, invoice field extraction, email triage — where the cheap tier costs 5-50x less with no measurable quality loss.
- Data that cannot leave the building: an open-weight model on your own hardware is the only setup where 'the data never left' is a fact, not a contract clause.
- Predictable spend: fixed hardware plus roughly €60/year of electricity beats a per-token bill that spikes in a busy month.
You can skip it when
- Multi-step agents that call tools — sub-7B models emit malformed calls. Instead run a 27B+ model at Q4_K_M locally, or just pay for the flagship tier.
- Jobs over ~100K tokens needing multi-fact reasoning. Instead chunk and retrieve, or route that one job up the ladder to the big model.
- Buying hardware to save money on a €20/month API bill — payback is about six years. Instead pin the cheap cloud tier and revisit at €200/month.
The long version
What it is — in one coffee-break
Every serious vendor now ships a ladder, not one model. At the top is the flagship from the demos; below it sit smaller siblings trained to do most of the work for a fraction of the price. Google runs from Gemini 3.1 Pro at $2.00 per million input tokens and $12.00 output down to Gemini 3.5 Flash-Lite at $0.30 and $2.50. Mistral runs from Mistral Large at $2/$6 down to Ministral 3B at $0.04 both ways — a fiftyfold spread inside one catalogue.
The second half of this element is the part your accountant cares about: some small models ship as open weights you can download and run on hardware you already own. Quantisation is what makes that practical — the model's numbers are rounded from 16 bits to 4, cutting memory to roughly a quarter for a small accuracy loss. That is why an 8-billion-parameter model is about 5 GB on disk and runs on a laptop.
When you actually need it (and when you don't)
You need the cheap tier when volume is high and each task is narrow: classifying inbound email, pulling fields off invoices, tagging tickets, drafting first-pass replies. Push those down the ladder and the bill falls five- to fiftyfold with no measurable quality loss, because the task never needed frontier reasoning. You need local models when data cannot leave the building — an open-weight model on your own box is the only setup where the data never left is a fact rather than a contract clause.
You do not need them for work that decides something. On 2026 management-task comparisons, local 8B models scored about 2.8-3.3 out of 5 against 4.8 for a frontier model. Agents are worse: benchmarks put Llama 3.3 70B near 97% well-formed tool calls and Gemma 4 27B near 95%, while sub-7B models emit malformed calls on multi-step plans — and at 95% per call an eight-step workflow finishes about 66% of the time. Long context is the other cliff; frontier models already shed 30-60 points of multi-fact retrieval past 200K tokens, and small ones fall off sooner.
How to recognize good vs bad implementations
A good implementation makes the tier a choice you control: a visible model selector, published per-tier pricing, one API shape across tiers so switching is a single string, and at best open weights you can pull down yourself. A bad one hides routing behind a smart toggle, bills a blended rate, and quietly serves the cheap model on your most important prompt.
Ask three things on the demo. Which tier served this exact response, and can you show me that in the response metadata? If I pin the cheapest tier to my highest-volume task, what is my monthly bill — in writing? Do you publish per-tier numbers for tool calling and long-context retrieval, or only for the flagship? Silent routers deflect the first, vendors with no real ladder bundle their way out of the second, and almost nobody survives the third — how they fail there tells you how honest the rest of the pitch is.
What this costs
Cloud, cheap tier: Ministral 3B at $0.04 per million both ways, Mistral Small at $0.10 in and $0.30 out, Gemini 3.5 Flash-Lite at $0.30 and $2.50. A shop pushing 20M input and 5M output tokens a month pays roughly $19 on Flash-Lite against about $100 on the Pro tier for identical traffic; batch endpoints cut another 50%.
Self-hosted: licence €0. A used RTX 3060 12 GB in a PC you already own costs €150-200 and runs a quantised 8-12B model. A Mac mini with 32 GB of unified memory is about €1,200 and gives 55-82 tokens/second on Qwen3 8B at Q4_K_M, or 32-52 on Gemma 3 12B; a 27B model wants 48 GB and a €3,000 Mac Studio. Electricity is the line people forget: a laptop-class machine drawing ~100W for eight hours a day is roughly 200 kWh, or €60 a year at €0.30/kWh, while a 650W GPU rig is nearer €390. Do the arithmetic first — €1,200 of hardware against a €17 monthly API bill takes about six years to pay back. Self-hosting wins on data residency and predictability, not on price.
Where to see it scored
Of 62 tools audited, 39 carry a score here. Gemini / Google (9) leads on the clearest published ladder and per-tier pricing. Mistral AI (8.5) matches it on substance and beats everyone on the self-host path, since its small models ship under Apache 2.0. Decagon (8.5) is the applied case — tier choice exposed inside a vertical product. Krea (8) earns its score for offering real speed and cost tiers instead of one price. At the bottom, Gamma (5) and Base44 (5) give you one opaque model and one bill — fine until your volume triples. Check any vendor's cost story against Google's published price list.
Flashcards
Check yourself
1. Gemini 3.5 Flash-Lite input is $0.30/M. Gemini 3.1 Pro input is $2.00/M. Roughly how much cheaper?
2. What does quantisation do to a model?
3. A model makes well-formed tool calls 95% of the time. What is the end-to-end success rate over eight steps?
4. You buy a €1,200 Mac mini to replace a €17/month Flash-Lite bill. Payback?
5. Which vendor scores highest on this element in the RXed audit?
Cheat sheet
- Flagship $2/$12 per M tokens; Flash-Lite $0.30/$2.50; Ministral 3B $0.04/$0.04.
- Quantisation: 16-bit weights rounded to 4-bit, ~quarter the memory, small accuracy hit.
- Q4_K_M is the production floor — Q3/Q2 breaks tool calling before it breaks chat.
- Qwen3 8B at Q4: 55 tok/s on an M2 16GB, 82 on M4 Pro. 8B model = ~5 GB on disk.
- Sub-7B models emit malformed tool calls on multi-step plans. Triage duty only.
- Self-host: €0 licence, €150 used RTX 3060 or €1,200 Mac mini, €60-390/yr power.
Who actually does this well
| Best on this element | Score | Why it scored that |
|---|---|---|
| Gemini / Google | 9 | Flash and Flash-Lite tiers (3.1 Flash-Lite at $0.25/1M input), thinking-level controls for cost, plus the open-weight Gemma family and on-device Nano. |
| Decagon | 8.5 | Small, fast, fine-tuned models are the explicit architecture strategy; custom speculative-decoding cut p95 voice latency under 400ms, and compact fine-tuned models are stated to be |
| Mistral AI | 8.5 | Small models are Mistral's DNA: Ministral 3 at 3B/8B/14B and Mistral Small 4 give a genuine cost/edge ladder, most of it under open weights. |
| ChatGPT / OpenAI Platform | 8 | mini/nano tiers are genuinely cheap and capable; clear cost ladder. |
| Claude / Anthropic | 8 | Haiku 4.5 is a genuinely cheap, capable small tier and Sonnet a strong mid-tier; a clear cost ladder, though no open-weight or edge options. |
| DeepSeek | 8 | V4-Flash at $0.28/1M output plus the open R1-Distill ladder down to single-GPU sizes give a genuine cost/size menu few labs match. |
| Krea | 8 | Krea 2 Turbo is distilled to ~2-second generations in 8 steps on consumer GPUs; the realtime latent-consistency stack is the product's foundation. |
| Genspark | 7.5 | Lighter/cheaper models (e.g. GPT-5.4 Mini) sit in the routing pool, and Claw's Model Optimizer skill automatically assigns the most cost-effective model per scheduled task rather t |
And the other end of the same column:
| Weakest | Score | Why it scored that |
|---|---|---|
| Gamma | 5 | A standard-vs-premium model ladder exists (standard models included on paid plans, premium models cost credits, image models selectable via API), but it is a cost meter rather than |
| HeyGen | 5 | Avatar engines at different credit prices form a de-facto quality/cost ladder, though not a small-model menu in the LLM sense. |
| Manus | 5 | Three model tiers (1.6 Max, 1.6, 1.6 Light) with different credit costs and capabilities — a genuine cost/quality ladder, but all are proprietary routing, not user-deployable small |
| Microsoft 365 Copilot | 5 | Auto-routing optimizes cost invisibly and Studio exposes some model choice, but there is no explicit cheap-tier product choice inside M365 Copilot — cost ladders live in Azure. |
Scored on 39 of 62 audited tools. Every score links to the full audit and its reasoning.