The Cheapest Token Is the One Your Agent Never Writes
Amazon swapped Claude Sonnet for a decision model that can't generate text on a ticket-labelling job. The unit of AI cost is moving from the model to the decision, and narrow agents are cheaper and easier to trust.

- Models
- AI Agents
A ticket lands in a queue. A general-purpose agent reads it, rereads a long system prompt describing twenty categories, reasons about which one fits, and writes a paragraph explaining its choice.
Nobody reads the paragraph. The queue needed one label, and it pays for the whole performance on every ticket.
Two Amazon engineers who work on one of the company’s largest internal ticketing systems ran close to that setup until recently. In November they and the Strands Agents team described replacing Claude Sonnet 4.6 on the labelling job with a decision model that, in their words, “can’t generate text”. It returns a probability for each label plus a confidence score. It belongs to the same family as TypeSafe’s Jev, which I looked at in September: models built to return a decision and a probability instead of prose.
That swap previews where enterprise AI is going. The unit of AI cost is moving from the model to the decision. Break an agent’s work into the decisions it makes, and give each one the smallest model that can make it reliably. Domain-specific agents built this way, for one bounded job with a known set of answers, spend fewer tokens and are easier to trust, and both benefits come from the same source: a smaller job.
Where the tokens actually went
Sonnet produced about 170 output tokens per ticket. The decision model, Strands Decider, produces one. The input shrank too, because the team rewrote each production label description as a one- or two-sentence criterion so the whole catalogue fitted a 4,096-token window. Most of a general agent’s spend on this task was instructions it had to reread and reasoning nobody needed.
Using token list prices for Sonnet and an estimated instance cost for its own model, the team puts the new path at “about 44 times cheaper than Sonnet, a 98% reduction”. Those are two different cost models, so treat the multiple as an estimate. The Decider side assumes busy hardware with some headroom, the comparison is Amazon’s own evaluation of an AWS open-source project, and the post says the smaller model “performed just as well” without publishing an accuracy figure. One widely shared retelling has since attached a dollar total and a different architecture that appear nowhere in the original, which promises “millions” a year and stops there.
The authors also say where the approach stops. The swap worked, they write, because “the task is a closed choice from a known catalog, the volume is high, and the latency of the old path mattered”. If the labels need explanations, the catalogue changes weekly or the useful context runs past a few thousand tokens, they expect to keep the LLM. Those conditions make a workable definition of a domain-specific agent.
None of this retires the general model. The Decider repository recommends letting “the LLM make the hard decisions and a decider make the rote ones”, and a position paper on small language models in agentic systems, revised this September, calls “heterogeneous agentic systems (i.e., agents invoking multiple different models)” the natural choice. Anthropic put it more plainly in 2024 when it told builders to look for “the simplest solution possible, and only increasing complexity when needed”.
Trust grows when the job shrinks
Cost gets the headlines, but trust is the bigger reason this pattern will spread.
A model that picks one of twenty-one options cannot email a customer, call a tool or invent a twenty-second category. When it fails, it produces a wrong label, which is a narrower failure than a wrong action. That makes errors easier to detect, contain and fix, provided nothing downstream acts on a label without checks of its own. A risk team reviewing it is approving a classifier with a measurable error rate.
The confidence score gives the team a practical escalation signal. On the repository’s own evaluation, answers given with confidence of 0.9 or higher on short classification tasks the model had not seen were right about 95% of the time, and its guidance for anything lower is to “confirm or ask a person”. That calibration belongs to that benchmark and needs rechecking on your own data, the same gap I flagged with Jev, whose calibration nobody outside the company had yet tested. Once rechecked, it is a rule you can write into a process document, and a general agent’s fluent paragraph offers nothing equivalent.
The Amazon team added guardrails around the model as well. Its endpoint runs with outbound network access blocked. A three-way switch lets the system run Sonnet alone, both models side by side on live traffic, or the Decider alone, and the team says that comparing both on real tickets “told us more than any offline eval would have”. Rolling back is a configuration change.
This is where model selection becomes a security decision. The less agency a component has, the smaller the blast radius when it fails. Excessive Agency is now number three in OWASP’s 2026 Top 10 for LLM Applications, and the project’s Q1 exploit round-up asks platforms to “make least-privilege and least-agency defaults mandatory”. A domain-specific agent gets least agency by design, because nothing outside its job is wired in.
There’s a catch. Narrow models fail in narrow, strange ways. The same post reports that sending identical content with the keys in a different order moved output probabilities by up to 0.19, so the team now sorts keys before every call. A smaller scope makes failures easier to find, and somebody still has to own finding them.
What changes for enterprise AI
If cost follows the decision, the place to look is the agent’s traces. Most are full of small, repeated choices: which queue, which template, is this a duplicate, does this need a human. Many of them are closed choices from a known list, which makes them candidates for a cheap, auditable decider while the general model keeps the work that needs language.
That moves the hard work somewhere less glamorous. Someone has to own the catalogue of options, because the Amazon result depended on rewriting labels into crisp criteria. Someone has to own the confidence threshold and the side-by-side run against live traffic. Teams that already treat prompts, evals and routing as owned assets will find this easy. Teams that bought one big agent and pointed it at everything will need to take it apart first.
It also changes what to ask vendors. Price per million tokens matters less once many of your calls need a single output token. A better question is whether your platform lets you route a decision to the smallest model that can make it, show the confidence, and switch back quickly.
For the next planning cycle, start with your agent’s traces and ask how many of its decisions need a model that can write at all.