Latest in AI
How to Test LLM Routing Before Switching to a Cheaper Model
A practical way to compare model cascades, catch expensive mistakes and check whether lower token prices actually reduce the cost of useful work.
To test LLM routing, compare a cheaper model, a stronger model and a routing policy on the same labeled tasks. Measure the errors each produces, the total cost of both calls when a task escalates, and the time users wait. Token price alone will give you the wrong answer.
A model cascade sends a request to one model first, then asks another model to handle cases the first stage cannot confidently resolve. This can be useful when most requests are straightforward and a smaller group needs more capable reasoning. The difficult part is recognizing that smaller group before an incorrect answer reaches the user.
That is where I would spend the evaluation effort. A cheap model that knows when to stop can be useful. A cheap model that sounds certain about the wrong answer can make an apparently efficient system expensive to operate.
What the Jev and Opus routing experiment actually shows
Hightouch's November 4 identity-resolution experiment is a useful starting point because it includes a harder test alongside the attractive headline result. The task was deciding whether records grouped into one identity actually belonged to different people.
On its easier synthetic dataset, Jev escalating to Opus achieved 98.8% accuracy at $0.88 per 1,000 cases, compared with Opus alone at 99% and $3.86. On harder data, the cascade achieved 95.3% at $2.94, while Opus achieved 95.1% at $5.39.
The cost advantage therefore fell from roughly 4.4 times to 1.8 times. These are results from a particular synthetic classification setup, not a general price comparison or evidence that your workload will behave similarly. The harder test is the useful lesson: routing economics depend on the work that reaches the router.
Start with one decision you can judge
Choose a bounded task before comparing models. For example, a support system might classify a ticket into a queue. A document workflow might extract a renewal date. A customer-data system might flag a possible duplicate record for review.
For this guide, consider a system that checks whether a support ticket needs a specialist. Sending an ordinary ticket to a specialist wastes time, but failing to escalate an account-access problem can leave a customer stuck. Those errors should have separate acceptance criteria, even if your dashboard also reports overall accuracy.
Write down what counts as a correct result, which mistakes are unacceptable and when a human must review the answer. If two reviewers regularly disagree about the labels, resolve that ambiguity before treating a model score as meaningful. You cannot evaluate routing reliably against a target that keeps moving.
Build separate validation and test sets
Use representative, appropriately handled examples from the workflow, with labels checked independently of the model outputs. Include ordinary requests and the awkward cases that matter operationally, such as incomplete descriptions, overlapping categories and requests containing conflicting instructions.
Keep one set for developing the policy and another untouched set for the final comparison. Choose prompts, escalation thresholds and other routing rules using the development set. Once those choices are fixed, run the held-out test without adjusting the policy to improve its score.
If examples are closely related, keep them together when splitting the data. Ten lightly rewritten versions of one ticket spread across both sets can make the test look more independent than it is. For a changing product, a later time period can also help expose new kinds of requests.
Report results for difficult cases separately from the representative mix. You need to know both how the system handles normal traffic and where it breaks; an artificially difficult dataset alone will not tell you your expected production bill.
Compare three policies on the same requests
Run a cheaper-model-only baseline, a stronger-model-only baseline and the proposed cascade. Keep the task definition and grading rules consistent. Record exact model versions, prompts and configuration so a later change does not quietly invalidate the comparison.
For the cascade, specify what triggers escalation. It might be a failed schema check, missing required information, disagreement between checks, or a score that falls into a defined range. A malformed answer is straightforward to detect. A plausible but incorrect answer is much harder.
Do not assume that asking a language model to give its confidence produces a calibrated probability. A score of 90% is useful only if its relationship to correctness has been checked on the relevant task. Research on calibration-aware cascades explicitly separates model calibration from selecting the deployment policy using validation data.
Also define what the second model sees. Giving it the original ticket is a different policy from giving it the ticket plus the first model's answer and explanation. Test the configuration you intend to deploy, including that extra context and its cost.
Calculate the full cost of escalation
The following is an illustrative calculation, not a measured benchmark or current provider pricing. Suppose a cheap call costs $0.001 per request, a strong call costs $0.01, and the cheap model escalates 20% of requests. Assume equal request sizes and no additional routing, retry or infrastructure costs.
| Policy for 1,000 requests | Cheap calls | Strong calls | Illustrative model cost |
|---|---|---|---|
| Cheap model only | 1,000 | 0 | $1 |
| Strong model only | 0 | 1,000 | $10 |
| Cascade with 20% escalation | 1,000 | 200 | $3 |
| Cascade with 80% escalation | 1,000 | 800 | $9 |
At 20% escalation, the model-call bill is 70% lower than the strong-only baseline. At 80%, it is only 10% lower. Both stages are paid for on an escalated request, so counting only the final model understates the cost.
Under these simplified assumptions, the cascade stops saving money at a 90% escalation rate. In a real workflow, longer escalated prompts, retries or a separate routing model can change that point. Use actual token usage and charges from your evaluation rather than multiplying one convenient example request by your traffic volume.
Inspect the answers the cheap model keeps
A common evaluation mistake is spending all the review effort on escalated requests. Those are the cases the system already recognized as difficult. The more revealing sample is often the requests it accepted without escalation.
Measure correctness separately for accepted cheap-model answers and escalated answers. For the support example, also count specialist-needed tickets that were incorrectly kept in the ordinary queue. A low escalation rate looks efficient until you discover it is caused by missed specialist cases.
Compare error counts as well as percentages and record the sample size. Two policies separated by a handful of correct answers in a small test do not establish a dependable winner. Review the disagreements and collect more evidence when the difference would change the deployment decision.
Choose the cheapest policy that meets your quality requirements, rather than choosing the cheapest model and finding a metric that makes it look acceptable.
Check waiting time and operational cost
An escalated request may wait for the first model and then the second. Measure end-to-end latency, including validation, retries and tool calls, and inspect the slow end of the distribution alongside the average. A lower average can hide a worse experience for the users with difficult problems.
Track manual corrections too. If a cheaper policy creates more specialist reviews, its model savings may be smaller than the extra work it creates. Keep model charges and human-review cost visible separately so assumptions about staff time do not disappear inside one impressive savings percentage.
Before switching live traffic, run the candidate in a controlled shadow evaluation where it cannot take external actions. Then introduce it gradually with explicit rollback conditions. Continue reviewing a sample of accepted cheap-model answers and recheck the policy when the task, model version or traffic mix changes.
The useful question is how much you can reduce the bill while preserving the decisions your product needs to get right. Once that is measurable, routing becomes an engineering choice you can defend rather than a discount you hope will survive contact with users.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical founders turn product decisions and engineering work into clear stories that developers and customers can understand.