The optimisation that would have bought one per cent

The endpoint that returned in two milliseconds

Symptom

A bid detail page taking seconds to appear locally, with no obvious culprit: the database holds two documents of under a kilobyte, fifty lookups run in forty-one milliseconds, every container is under two per cent CPU, and the page shell itself serves in two milliseconds.

Diagnosis

One endpoint in the table was the answer. Listing the assignable users for a bid returned in 2 ms at p50 while its neighbours ran 174 to 304 ms at p50 and up to 6.8 seconds at p99. It shares the authentication chain, the permission middleware and the database with every other row. The only thing it does differently is that it does not decrypt anything.

That made it a control variable rather than a hypothesis, and it ruled out the whole middleware stack in one reading. A round trip to the key service from that machine measures about 150 ms: 41 ms to establish the connection and 100 ms for the handshake, which is precisely the floor sitting under every other row.

Then the part that decided the fix. The shared gate decrypts exactly one field. I enumerated all thirty-nine call sites: twenty-six discard the loaded record entirely, wanting only an existence-and-authorisation check plus an identifier, and the thirteen that keep it read only plaintext columns. Not one of them reads the decrypted field. Nor does anything else: its only other appearances are the crypto functions themselves and one write path. Seven of the thirteen then make a second round trip on the same key.

A further twenty call sites bypassed that gate altogether, and two of them were on the page’s hot path. One used a full decrypting read as a bare existence check and then re-read the same record to fetch its key, paying two database reads and two key round trips per request. A page load spent roughly twenty-eight key round trips, nearly all of them to produce a string nobody looks at.

Decision

Not a key cache. A time-limited cache of plaintext data keys would have fixed this everywhere in about thirty lines, and I wrote down why I was not doing it: it holds key material in process memory past the request that needed it, it removes the per-decrypt audit record from the cloud trail, and it defers key revocation by the length of the cache lifetime. When almost every call turns out to be unnecessary, not asking is both faster and tighter than caching the answer.

So: a non-decrypting read for handlers that touch only plaintext fields, a non-decrypting existence gate for the twenty-six that discard the record, and the redundant gates in front of key fetches downgraded to the presence check that already existed. No new mechanism, no route changes, no response-shape changes.

Cache the answer

mechanism   TTL cache of plaintext data keys
cost        ~30 lines, fixes every call site
key material lives past its request
audit trail loses the per-decrypt record
revocation  delayed by the TTL
and every needless decrypt is still there

Stop asking the question

mechanism   three reads that do not decrypt
  discards the record  → existence gate
  plaintext fields     → non-decrypting read
  needs the key        → unchanged
key material never outlives the request
audit trail intact · revocation immediate
encrypted field returned BLANK, not as bytes

Two details carry more weight than the speedup. The non-decrypting read blanks the encrypted field rather than passing the ciphertext through, because serving encrypted bytes labelled as prose is a silent failure and an empty field is a loud one. And the invariant: nothing from a non-decrypting read may reach the whole-document write path, which re-encrypts that field and replaces the document, so an empty value arriving there is data loss rather than a no-op. The two read-modify-write callers keep the decrypting read, deliberately, and a test asserts both halves: that the raw read blanks the encrypted field, and that it does not blank the plaintext columns its callers exist to read.

Output-bound, not input-bound, so the obvious optimisation was dropped

Symptom

A 301-question tab answered in eight batches of forty, two batches in flight, about 6.4 minutes per wave and roughly 26 minutes for the tab. Tabs run strictly one at a time, so a fifteen-tab workbook is 70 to 80 minutes. The prompt for one batch is around 245 KB. The obvious read is that the prompt is enormous, so cache its stable prefix.

Diagnosis

A batch is one model call. Its input prefills in tens of seconds; its output is forty answers of prose, 10 to 16 thousand tokens, and that generation is five to six of the 6.4 minutes. The call is output-bound.

That single sentence reorders everything, because prompt caching acts on the input side. Two independent counts finished it off.

The byte-stable prefix is about 4–5% of the prompt. Roughly 10–12 KB of system prompt and preamble against 245 KB, because the retrieved material differs per batch by design. Discounting 4% of a prefill that is itself the minority of an output-bound call is on the order of 1% of wall clock. And the prefix is not even strictly stable: parts of it are conditional on what a given batch contains.

The cache lifetime is shorter than the gap between waves. The available cache entries expire in five minutes and the SDK version in use offers exactly one lifetime setting, so every wave after the first would find the entry expired and pay the write premium again. The mechanism would run, cost more, and produce nothing, which is the worst of the available outcomes, because it yields cost with no signal.

Decision

Dropped, with both counts written down and a specific condition for revisiting it: if per-batch latency ever falls under five minutes, waves fall inside the cache lifetime and it becomes worth its 4%, as a cost optimisation rather than a speed one.

What the output-bound finding leaves is exactly two levers: more calls in flight, or fewer output tokens per call. And it rules a third one out on the record. Shortening the answers would cut the dominant cost directly, and that is precisely the accuracy trade this exercise existed to avoid. Raising the batch size is also out: 64% of the per-call timeout is already consumed, and on an output-bound call a bigger batch buys timeout risk in direct proportion.

Instrument first, and not as diligence theatre. It answered two of the four open questions by observation instead of argument. The per-call token counts were already being returned by the SDK and discarded, and rate-limit rejections had no counter, so are we being throttled today and is the binding limit our concurrency or the account’s output quota were both unanswerable. One day’s work, no behavioural risk. The one wrinkle found while scoping it: the usage struct is serialised to the browser verbatim by an unrelated chat feature, so adding fields to it is a wire change rather than an internal one, which decides how they are named, or whether a local type is used instead.

An un-retried throttle is not a slowdown. It is a silent quality loss.

Symptom

Concurrency was capped low. The comment on the cap explained why: rate limits exist and this service has no retry or backoff, so a throttled batch is a degradation rather than a failure, its questions fall back to needs-manual-entry, and a conservative cap therefore costs a few manual questions at worst.

Diagnosis

Read that again as a user experience. A throttled batch does not fail the run. The run completes, reports as complete, and is quietly forty questions thinner, and nothing counts that. This is not a throughput problem that also has a quality wrinkle; it is a silent correctness problem being described as a rate-limit accommodation.

Which reverses the ordering of the whole plan. Backoff is not a prerequisite for going faster; it is the accuracy fix, and it happens to also be the prerequisite. It must land, and be observed working, before any concurrency number moves.

Then the part I would have got wrong by reading fast: retry is not absent, it is thin. A single bounded retry on throttling had landed shortly before, and the comment on the concurrency cap still asserts there is none. Two comments, in fact, both arguing from a premise that was no longer true, including one whose entire justification for halving the cap is a retry gap that has since been partly closed.

Decision

Rewrite the comments in the same change that moves the number. This sounds like tidying and is not: the alternative is a constant guarded by an adjacent rationale that argues against the edit you just made, which is how a number gets reverted by the next person to read it. And the cap steps one at a time, with the throttle counter read between steps, because “we raised it and nothing broke” is not a reading when the failure mode is invisible by construction.

The premise that had expired

“…this service has no retry/backoff on
  throttling, so concurrency is capped
  rather than unbounded.”

“A throttled batch still degrades
  gracefully … a conservative cap costs a
  few extra manual-entry questions at
  worst, not a broken run.”

false since the retry landed, and the
reassurance IS the defect

What the comment has to say instead

one bounded retry on throttling exists;
that is what lets this number move.

an un-retried throttle is not slowness.
it is a whole batch of answers lost, on a
run that still reports success. hence the
throttle count and retry outcome in the log.

step the cap one value at a time and read
that counter between steps.

bounded by the silent interval the watchdog
tolerates. derivation written down here.

One constant, three deadlines, and a watchdog derivation that was not the mechanism

Symptom

Every knob involved (batch size, concurrency, retrieval budgets, retry budget) is a package-level constant. The work was framed as tuning the returnable-tab path, which is where the hour-long run was observed.

Diagnosis

There are three callers of the answering routine and only one of them has tabs. Response-grid generation runs under a single hard deadline for the entire run, keyed off when the run started, which no progress write can extend. Requirement validation runs synchronously inside a live HTTP request, where backoff means a person watching a request hang and a proxy timeout that will cut it long before any retry budget of ours expires.

So the blast radius is not returnables, and a single compiled-in retry budget cannot be correct: the three deadlines differ in kind, not in size. The first vetting pass of this plan reasoned as though the tab path were the only caller. A second pass found it, along with a test the first pass had declared unaffected, which would have shipped broken.

The watchdog reasoning had the same shape of error, in the codebase rather than the plan. A comment derives worst-case tab silence from the call timeout plus a retry, arriving at about twenty minutes, and the resumer’s staleness window is sized against that. It is a sound derivation of a bound the code does not rely on: progress is written after every batch, and that write unconditionally touches the very timestamp the staleness filter matches on. Legitimate silence is therefore one batch, not a whole tab. The genuinely exposed cases are narrow and specific: a single-batch tab, and the last batch of any tab, where there is no subsequent batch to touch the clock.

Decision

Make the retry budget caller-supplied, and write down which deadline each value was derived from. Rewrite the watchdog comment around the per-batch touch that is the actual mechanism, because otherwise the rewrite re-enshrines a plausible fiction, and the next person sizes a timeout against it. Test the single-batch tab specifically, because it is the only case where a retry produces real silence. And in the validation path, ensure the retry loop aborts on a cancelled request rather than sleeping through one.

The fail-open lane one layer over is the trap I would most expect to be missed. Raising answering concurrency also doubles the concurrent retrieval calls behind each batch, and that lane fails open by design, against a different service quota. So the answering-side throttle counter can read clean while retrieval context quietly thins: the exact silent loss this whole plan exists to close, reintroduced one lane over, and invisible to the instrument built to watch for it. Hence a separate per-batch count of retrieval failures, in place before the cap moves.

Halving the batch is not halving the input

Symptom

The remaining lever is fewer output tokens per call, which means fewer questions per batch. That looks like a one-constant change: halve the batch size, per-call output halves, per-call latency roughly halves, and batch count doubles so the concurrency has something to work on for small tabs too.

Diagnosis

Halving the batch alone changes what every question sees, in two directions.

The retrieved question-answer block is always filled to its character budget: the merge takes the best per question up to a cap, then backfills unspent budget with the next best of what is left, zero-scoring pairs included last. Halve the questions and leave the budget alone and the block does not shrink. It renders the same volume over half as many questions, with the tail made of material that lost. That is a pure increase in near-miss context, and near-miss context degrades answers rather than diluting them harmlessly.

The library lane failed for a different reason than the plan claimed. The original argument was that the budget was starving the per-question allocation. The run says otherwise: 1 to 1.8 chunks admitted per question against a ceiling of three, so the admission threshold is what binds, not the character budget, and halving a budget cannot restore what a threshold already rejected. Worse, the risk direction reverses: if real rendered size per chunk exceeds a computable figure, the halved budget starts binding and per-question volume falls silently. Nobody measures rendered characters today, only counts. That is one of the things the instrumentation adds, and it has to be read before this change ships.

Decision

Halve the batch size and both character budgets together, so each question’s material stays approximately as it is, and the change buys only throughput. State the part that is not neutral rather than claiming it is: fewer questions per batch means less cross-question deduplication, so the exact rendered set shifts slightly even with per-question volume preserved. Verified by comparing retrieved-material-per-question before and after, plus a spot-check against the one corpus item that has human-written ground truth.

And do not claim the latency halves. A per-batch fixed cost does not: the retrieval index is rebuilt over the whole corpus on every batch, so doubling the batch count doubles that work rather than halving it. The net is still a win because the embedding fan-out does halve, but the instrumentation has to separate retrieval from generation or the result reads as an under-delivery when what actually happened is a fixed cost being paid twice as often.

One variable per run, and one sequence: instrument, make the loss visible, step the concurrency, then reshape the batch, on a separate run. Two of these changed together would leave neither result readable. Two more things stayed out on purpose: moving the answering model to a smaller one, which is the one change here that genuinely trades accuracy for speed and was a deliberate earlier decision, and parallelising tabs, where the run claim, the per-tab claim, the staleness window and the tab deadline are all built around one tab at a time. That is reworking the claim model for a win that only shows up on large workbooks.