Strands Decider vs Sonnet: How to Evaluate Cheaper Ticket Classification

Amazon's November 9 case study reports a 98% cost reduction after moving an internal ticket-labeling workload from Claude Sonnet 4.6 to Strands Decider E4B. The system chooses among 20 categories. Amazon reports average call time falling from 4,215 ms to 897 ms, with costs around $500 rather than $22,200 per million tickets. These are Amazon's workload-specific results, not an independent benchmark or a new model release. Read the case study.

For an engineering team, the useful question is whether an existing classification step can move to a smaller model without making expensive mistakes elsewhere. This guide describes how I would evaluate that change. I have not run these models or reproduced Amazon's measurements.

Start with the decision your application actually needs

Strands Decider accepts a state and a typed question. Its interface supports yes-or-no probabilities through noul, selection among options through choice, and ordered ratings through score. It does not generate an explanation or a customer response. The inference guide documents the CLI and HTTP interface.

That makes a support queue a useful starting point. Separate “which team should receive this?” from “write a helpful reply.” Evaluate the first operation on its own. Keep the reply-generation path unchanged until there is a separate reason to modify it.

Write down the cost of a wrong route before setting an accuracy target. Sending a billing question to general support may add a small delay. Losing an urgent security escalation can be much more serious. A single overall accuracy score will hide that difference. Track recall for important categories, the proportion sent to review, and the time until a ticket reaches the correct owner.

Try E4B with a small, explicit label set

The October 15 release added E2B, E4B, 12B and 26B-A4B variants based on Gemma 4. They share the question interface, so model selection does not require redesigning every request. The release guide specifies Python 3.10 or later and CLI version 0.2.0 or later.

Its documented installation and question syntax can be adapted for a local routing experiment:

pip install "strands-decider>=0.2.0"

strands-decider ask StrandsAgents/strands-decider-E4B-gemma4-v1-2610 \
  --state "My invoice includes two charges for the same order." \
  --choice "Which queue should receive this ticket?=billing,technical support,sales,needs review"

This is an illustrative input, not a recorded test result. One documentation wrinkle matters: the E4B model card still says Gemma 4 support requires the repository's main branch, while the release guide specifies PyPI 0.2.0. Check the installed version and its model support before troubleshooting hardware. Pin the working package and checkpoint revisions in your evaluation record.

The model card also explains that the checkpoint contains an adapter and readout heads, with base weights downloaded separately. Account for that initial download when preparing an offline environment. The documented local server has no authentication, so keep it local during experiments rather than exposing its port publicly.

Compare preprocessing as well as models

Build a labeled evaluation set from representative historical tickets. Include short requests, long discussions, ambiguous categories and messages that belong in none of the normal queues. Keep a separate set for choosing thresholds so that the final evaluation does not reward decisions tuned to its answers.

Run two comparisons. First, give both models equivalent useful information and concise category definitions. Then compare the complete proposed production paths, including truncation, retries and fallback behavior. Otherwise, an improvement from cleaner inputs can be mistaken for an improvement from the model itself.

The Gemma 4 release warns that changing the question can sometimes leave answers unchanged when state and options stay fixed. It also cautions that confidence calibration comes from short classification tasks. Test these behaviors directly on your categories rather than treating a confidence value as a universal guarantee.

Calculate break-even volume before renting an endpoint

Amazon's case study prices its SageMaker instance at $1.408 per hour. Using that rate, one continuously running instance costs about $1,014 over a 30-day month. Dividing by the reported Sonnet cost of $0.0222 per ticket gives a rough compute-only break-even point of 45,700 tickets monthly. This is my calculation from the reported prices, not Amazon's deployment threshold.

Your calculation should include redundancy, idle capacity, monitoring and engineering time. Also compare against a shorter Sonnet prompt or another hosted model. A dedicated endpoint can be attractive at steady volume while remaining unnecessarily expensive for a small queue with occasional bursts.

Roll out with an explicit fallback

I would begin with shadow predictions that cannot change ticket ownership. Have reviewers adjudicate disagreements, including cases where both models agree but the existing label is wrong. Then enable a small traffic slice, with low-confidence and high-consequence cases sent to the established path or human review.

Keep the selected checkpoint, preprocessing version, options and threshold alongside each decision. Make rollback a configuration change. The best migration outcome is a lower total cost for correctly handled tickets, with enough evidence to know which requests should still use the more capable path.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help teams explain technical products through clear technical writing and developer storytelling.

Book a 30-minute call