The thirty-thousand-token tax on asking what time it is

The same twenty lines, written four times, one of them wrong

Symptom

The toggle had to be honoured at four separate chat entry points: start a conversation, send a message, and both of those again with attachments, each with its own copy of the tool validation block. I drafted the change as four edits.

Diagnosis

Three of the four were right. The fourth diverged: it had been written against the wrong baseline, so it also introduced a log line that the two attachment paths had never had, and the branch structure did not match what was actually there. Reviewing my own four blocks side by side was the only reason it was caught, and it was caught by comparing them to each other rather than by reading any one of them.

The deeper problem is not the typo. Four hand-maintained copies of a security-relevant decision, may this request use tools at all, is a structure that guarantees a future divergence. Whoever adds the fifth entry point copies whichever block they happen to open.

Decision

The toggle went in as four edits, because that matched what was there and I did not want to refactor the streaming handlers in the same change. But when the relevance stage arrived shortly after, it went in as one helper called from four places, and the tool decision consolidated there. A three-state preference is what made this worth getting right: the field is a nullable boolean, where unset means “never expressed an opinion” and must behave as on, while false is an explicit refusal. Four copies of a nil-check that has to distinguish absent from false is exactly the shape of a bug that reaches production as “the toggle doesn’t stick”.

Not a clever fix, and I have included it because the honest version of this story is that I wrote the same block four times and one copy was wrong. The lesson I took is narrower than “don’t repeat yourself”: a decision that must hold on every path should be reachable from one place, and permission-shaped decisions especially.

Measuring before choosing, and choosing the boring option

Symptom

A trivial question, the current time in a given city, was arriving at the model behind roughly thirty thousand tokens of tool schemas. Nothing was broken. It was simply the case that the platform’s fixed overhead had grown quietly, and no measurement had ever been attached to it.

Diagnosis

Two findings changed the shape of the fix. There was no caching anywhere in the path: the tool set was read out of the database on every single request. And the expense was in the schemas, not in the retrieval, so making the database read faster would have saved nothing that mattered. Whatever went in had to reduce the number of tool descriptions serialised, and everything else was noise.

Decision

I compared four approaches on latency, cost, accuracy and, the criterion that decided it, what new infrastructure each would require, and I wrote the numbers down before choosing rather than after.

A dedicated reranking model scored highest on accuracy, at ~200–300 ms and 20× the cost per call, and introduced a new vendor integration. A small language model acting as a router was ~800–1500 ms, five to ten times slower, at roughly 1500× the cost, plus a new provider dependency. Keyword matching was under 10 ms and free, and unacceptably crude for paraphrase, which is most of what real messages contain. Embedding similarity against the model already running in production for knowledge-base retrieval came in at ~100–150 ms and needed only a similarity function and a cache.

Then I budgeted the latency I was adding, because this sits in front of every request: a 150 ms timeout that falls back to the full tool set rather than failing, cosine similarity at under 5 ms, cache lookup at under 1 ms, and a p95 under 200 ms as the number that would decide whether it stayed.

I chose the one that added nothing. It is not the most accurate option available and I would not claim otherwise, but “we now depend on a second vendor to answer what time it is” is a real cost that does not appear in an accuracy column, and at 1500× the per-call price the most expensive option was being asked to justify itself against a shortlist it had already lost on latency.

Every tool described on every request

tool schemas serialised   all permitted tools
source                    database read, per request, uncached
input tokens              ≈ 30k
tools actually called     1

Top K by relevance, cached embeddings

tool schemas serialised   10
source                    in-memory cache, refreshed periodically
input tokens              ≈ 2.2k
tools actually called     1   ← same answer, same tool

⚙

tool_selectionbefore the model call · every request

Tools: On

“What’s the time in Tokyo?”

Schemas sent

10

Input tokens

≈2.2k

Permitted, unfiltered

27

Top 10 of 27 permitted tools, ranked by embedding similarity to the message. The rest never reach the model.

Reconstruction of the measured before/after in this case study. The permitted-tool count is per user; 27 is the account this was measured on.

Observed reductions across a handful of real messages ranged from roughly a third to over ninety per cent, the spread depending entirely on how tool-relevant the message was. These are individual observations from a test environment rather than an average over production traffic.

I did the annual arithmetic at the time, and it is worth saying what it produced, because the reason it is not on this site is the interesting part. At the measured 60k-to-12.2k reduction and published input-token pricing, the per-interaction saving is about fourteen cents, which lands at ~$4,300 a month and ~$52,000 a year at a thousand interactions a day, and ~$523,000 a year at ten thousand. Real numbers, multiplied correctly.

None of them are anywhere on this site, including the front page, because I never measured the message rate. Every figure above is the token measurement I did make, multiplied by a volume I guessed, so the guess is carrying the entire result, and picking a different one moves the answer by a factor of ten. Quoting $523,000 would also invite a reader to discount the $1,299 a month elsewhere on this site that is an actual invoice. The measurement is 93%; the dollars are somebody else’s to calculate once they know their own traffic.

I built an intent detector, measured it, and deleted it

Symptom

A specific fear, and a legitimate one. Six of the tools are state-changing, covering sending messages, creating calendar events and replying to mail, and each is protected by a human confirmation gate. If relevance ranking dropped one of those tools when the user had plainly asked for it, the model could not propose the action, so the confirmation gate would never fire and the user would be told the assistant lacked a capability it had. An optimisation silently disabling a safety feature’s subject is the worst outcome available here.

Diagnosis

So I built a guard: detect that the message expresses intent to take an external action, and if so force all permitted state-changing tools into the set regardless of their similarity score. The first design was a keyword list. I rejected it on my own review, because it fires on “send me a summary” and “create a list of bugs”, which are not external actions at all, and paying the full gated-tool overhead on every message containing the word “send” gives back much of the saving.

The second design was better and is what shipped: a handful of action phrases embedded once at startup, cosine similarity from the message to each, and a threshold above which intent is declared. It handles paraphrase, and “send me” genuinely does sit far from “send to someone” in that space.

Decision

Then I instrumented it, and in real use the threshold never fired. Not rarely. Never. The reason is slightly humbling: when a user asks to send a message, the ranking already puts the send tools at the top, because that is precisely what similarity to the message measures. The guard was protecting against a failure the primary mechanism does not have.

I deleted it, and reduced K from fifteen to ten with the headroom it had been reserving. What made that safe to do is a property of the existing design rather than of my change: the confirmation gate keys on the tool’s identity after the model has chosen it, not on how the tool came to be offered. Selection cannot bypass the gate, because the gate does not consult selection.

Designed for a failure the ranking doesn’t have

action phrases embedded at startup      7
threshold crossings in real use         0
gated tools force-included             all permitted, on crossing
top K                                  15   headroom for the forced set
send tools’ rank without the layer      already top 3

Deleted, K reduced

action-intent layer                    removed
top K                                  10
confirmation gate                      unchanged. keys on tool identity,
                                       checked after the model chooses

The favourite thing I did on this project, and it consists of removing my own work. A guard that never fires is not free: it is a component that must be maintained, a threshold that must be tuned, and a reserved token budget that costs real money on every request. I only knew it never fired because I had instrumented it before trusting it.

The metric moved, which is not the same as the mechanism working

Symptom

Local testing looked like a success and proved much less than it appeared to. The logs reported a healthy token reduction, and alongside it a selection of twenty-seven tools out of twenty-seven available.

Diagnosis

My own account had access to twenty-seven tools, not the full permitted catalogue, so locally the selection cap sat above the input size and the stage was barely selecting at all. The token drop was real but was coming mostly from a shorter serialisation path, not from ranking. The ratio the whole feature rests on was untestable on my own account.

Worse, the log printed counts and names, not positions. A count going down is consistent with correct ranking and equally consistent with the stage discarding tools arbitrarily. The only evidence that ranking was any good was whether the right tool survived the cut and was then actually invoked, so I went to the execution logs and confirmed the datetime tool had run for a time-of-day question, rather than inferring it from the count.

Decision

Ship behind a flag, default off, and validate against a set large enough for ranking quality to matter, which meant an environment with a wider tool grant than my own account had. On that environment the behaviour held up: the reductions were much larger, the tool that should have ranked first for a given question did, and the confirmation gate still fired on a calendar request. The flag is the honest acknowledgement that a local pass on twenty-seven tools was not evidence about a wider grant.

I nearly wrote this up as validated on the strength of the token graph. The number was correct and the conclusion I was about to draw from it was not. The graph would have moved the same way if the ranking had been random.