Every few weeks a new AI model arrives with a claim to be the best. For a business, "best" isn't the useful question. The useful question is: which model is good enough for this job, at the lowest cost and time?
The only way I've found to answer that is to test. Here's a test I ran recently, what it showed, and the routing rules that came out of it.
How the test worked
I built a suite of twelve real-world tasks: the kind of bounded, checkable work I do for clients, like building and fixing pages to a spec, extracting fields from documents and producing structured outputs. Three things made it a fair test:
- Frozen cases. The tasks and inputs were fixed before any model ran, so nothing could be tuned to favour one model.
- Graders written first. For each task, I wrote the pass/fail check before seeing any output. A task passed only if it met the check on the first try.
- Same conditions. Every model got the same instructions, context and tools.
I ran three tiers: a top-tier model (the largest and most expensive), a mid-tier model at roughly half the price, and a small model that costs a few cents per run. Costs below are at API prices.
The results
| Model tier | Passed first try | Cost for all twelve |
|---|---|---|
| Top tier | 12 of 12 | about $2.59 |
| Mid tier | 12 of 12 | about $1.36 |
| Small tier | 7 of 12 | about $0.10 |
Two findings stood out.
The mid tier matched the top tier on bounded work. Same pass rate, about half the cost. When a task has a clear spec and a check, paying for the biggest model buys very little.
The small model was cheap but missed five. Seven out of twelve isn't good enough on its own. But then I tried something else: I sent each of the five misses to the mid-tier model for one repair attempt. All five were fixed, in one attempt each.
So a small model doing the first pass, with a grader catching misses and a mid-tier model repairing them, got to twelve out of twelve at a fraction of the cost of running everything on the top tier.
The routing rules I use now
That test and others like it turned into a simple routing table:
| Work type | Routed to | Why |
|---|---|---|
| Writing, design, judgment calls | Top tier | Quality is the product, and there's no simple pass/fail check |
| Bounded build work with checks | Mid tier | Same pass rate, about half the cost |
| Extraction with full validators | Small tier | Cents per run when a grader catches misses |
| Anything the small model misses | Mid-tier repair | Fixed in one attempt in this test |
The key phrase in that table is "with checks". Cheaper models are safe to use when something verifies their work. Without a check, you're gambling, and the expensive model is the safer bet.
Why this matters for a small business
Most businesses pick one AI tool and use it for everything. That's like paying a senior accountant to do data entry: it works, but you're overpaying for most of the hours, and you're relying on them never to have an off day.
Matching the model to the job, and putting a check on everything that can be checked, gets you two benefits at once: lower cost, and more reliable output, because the grader catches mistakes no matter which model made them.
How to apply this without running a lab
You don't need your own test suite to use the idea.
- Separate judgment work from checkable work. Customer-facing writing and decisions go to your best model. Extraction, formatting and data checks can go to a cheaper one.
- Add a check wherever the answer can be verified. Totals that must add up, fields that must match a source document, dates that must fall in a range.
- Escalate failures. When the cheap path fails the check, send it up a tier rather than to a person first.
- Re-test when models change. Results shift with every release. What I route to a given tier today may move next quarter, and that's the point of testing rather than assuming.
The models will keep changing. A habit of testing on your own real work, with the graders written first, is what keeps your choices honest.