Case study · AI systems engineering

The cache that had never served an answer

Cost measurement and cache-correctness work on the answering path of a bid-response module.

Go
MongoDB
AWS Bedrock
Retrieval-augmented generation
Cost measurement

The answering path was built cheapest-first: an exact match, then a semantic match, then a full model call. Counted against the database, the store behind the first two tiers held zero rows against 658 logged attempts. Every question the system had ever answered had gone the expensive way, at roughly six tenths of a cent and a little over three minutes each. Nothing was broken enough to notice.

0 of 658logged answering attempts served by a cache that was wired up, logging, and had never once hit

The problem

The system answers tender questionnaires. A workbook can carry several hundred questions, and each one is answered from the company’s own library of past responses. Answering is the expensive part, so the path was built cheapest-first: try an exact match on a normalised question, then a semantic match over previously cached answers, and only then retrieve documents and generate an answer with a full model call.

That design is correct and it was built. The page on the answering path covers how, including an earlier encryption bug that would have made every cross-questionnaire lookup miss. That bug was found and fixed. What nobody had done afterwards was check whether the cache had started working. It had not, and two more defects were why.

Every answered question writes an audit event recording which tier served it. Querying that collection against the store behind tiers one and two produced the two numbers this page exists for: 658 events, and zero stored answers. Not a poor hit rate but an empty store. The cheap path had been available, wired in, and logging its own attempts since it shipped, and it had never once answered a question.

The failure mode of a cache is silence

A miss is a normal event. A cache that misses every single time behaves exactly like a cache that is merely cold, and produces correct answers throughout, just slowly and at full price. There is no error to see and no alert to fire.

The tests could not have caught it

Unit tests seed the store themselves, so they exercise the read path against data they wrote. The defect was on the write path, in production, under a condition no fixture reproduces: an answer with nothing to match against.

Two defects, one on each side

Even once the store filled, every hit would have been rewritten by a model rather than served exactly as stored. That was a second, independent bug on the read path, invisible while the write path kept the store empty.

The expensive tier is expensive in two currencies

Roughly six tenths of a cent per question is small. A little over three minutes per question, in a workbook that can carry several hundred, is not. Both were measured on the same sampled event.

What I built

Already existed

The three-tier answering path itself: hash lookup, semantic lookup, retrieve-and-generate, with its thresholds and its audit event on every answer
The store behind the cheap tiers, with a repository, an envelope-encryption option, and a documented schema
A separate response step that decides whether a cached answer is served exactly as stored or rewritten to fit the question
A classification config assigning each content class to a verbatim or a narrative bucket

What I added

The measurement that found it: the tier ledger queried against the store, which is what turned "the cache seems quiet" into two numbers
The write-path fix. The admission gate now reads a confidence the answer actually has, rather than a match score that is zero by definition when nothing matched
A classifier and a backfill for the content class the verbatim branch keys on, which was empty on every live pair
Both authored as a single change, because they edit the same four functions and either one alone leaves the store holding rows the other cannot route
An end-to-end test designed to isolate the routing decision from the corpus entirely, because the corpus could not exercise it
Two claims removed from my own plan after checking them: the partial payoff, and a guard protecting against a state nothing was in

A cache with two empty tracks, and the two independent reasons it had never served an answer

Four problems worth describing

The gate that read "nothing matched" as "not confident"

Symptom

None. Correct answers, produced slowly, at full price, indefinitely. The finding came from reading the audit trail rather than from anything going wrong.

Diagnosis

Write-back is gated on a confidence threshold, which is the right instinct. A shared cache should not be seeded with answers the system was unsure about, because a bad entry gets served to every future question that hashes to it.

The number being handed to that gate was a question-match score, meaning how well the incoming question matched an existing library pair, and its own documentation says it is zero when there is no match.

But the answers most worth caching are exactly the ones generated from scratch, which by definition matched no existing pair. So they arrived at the gate carrying zero, the gate read zero as no confidence, and refused them. Every time. The threshold was doing precisely what it was written to do, to a number that did not mean what the name suggested.

Two different quantities had collapsed into one word. Confidence in this answer and confidence that this question matched something are unrelated, and one of them has a hard-coded zero for the single most valuable case.

Decision

Feed the gate a confidence that belongs to the answer, and mark entries that were generated rather than matched with their own source flag so the read path can treat them differently.

Generated entries therefore enter the cache with a confidence of zero recorded honestly, and are admitted on the basis of their source rather than on a score they cannot have. That has a consequence I made sure held. The interface shows a confidence badge, and it must stay hidden on these entries rather than displaying a truthful-but-meaningless zero to a bid manager. A zero on screen would read as “this answer is bad”, which is not what it means.

The reusable part is the naming, not the fix. A field called confidence that is sometimes an answer’s confidence and sometimes a match’s confidence will eventually be read as the wrong one, and the compiler cannot help because both are the same type in the same range. The gate was reviewed, merged and correct-looking, and it was the only thing standing between the system and its own cheap path.

A payoff blocked behind a second bug, on the other side of the cache

Symptom

Discovered while checking what would actually happen once the first fix let the store fill, so before shipping it rather than after.

Diagnosis

A cache hit does not simply return its stored text. A separate step decides between two responses: serve the answer exactly as stored, which is correct for a certificate number, an insurance limit or an accreditation date, or hand it to a model to be rewritten to fit the phrasing of the new question, which is correct for narrative prose about methodology.

That choice keys off a content classification on the entry. The classification was empty on all 136 live answer pairs. So the verbatim test evaluated false every time, and every hit the system would ever have served would have taken the rewriting branch.

The two branches fail differently, which is what makes that bad. Rewriting narrative prose is fine. Rewriting a certificate number is a model paraphrasing a fact that had exactly one correct form. The cheap, exact, free branch was not merely unused. It was unreachable, and the branch that replaced it was the one that can corrupt hard facts.

Decision

Write the classifier, backfill the existing pairs, and, the part that mattered for sequencing, author both fixes as one change.

The two touch the same four functions, and the failure mode of doing them independently is specific. Fix the write path alone and the store fills with entries carrying no classification, every one of which routes to the rewriting branch, so the cache starts quietly rewording facts at scale. Fix the read path alone and there is nothing in the store to route. Landing them separately would have converted a dormant bug into an active one for however long the gap lasted.

I also verified the classifier against the live config rather than a fixture, confirming that the classes carrying hard facts resolved to verbatim and the topical ones to narrative. The whole defect was a classification that looked present and was empty.

What a cache hit actually did

Both sides broken: empty store, unreachable verbatim branch

write  gate wants   answer confidence ≥ threshold
       gate is fed  question-match score
       generated    matched nothing → score 0
       result       REFUSED, every time
store  rows 0 · logged attempts 658
read   verbatim needs a content class
       class present on 0 of 136 pairs
       result       always the rewrite branch
                    → a model rewords a cert number

One change, both sides

write  gate is fed  the answer’s own confidence
       generated    admitted on source flag
                    confidence recorded as 0, honestly
                    badge hidden, not shown as zero
read   classifier   hard facts → verbatim, $0
                    prose      → rewritten, as intended
landed both sides in one change, not two

A test that had to be isolated from the corpus, because the corpus could not exercise it

Symptom

Unit tests passed. The classifier was tested, the backfill was tested, and the live config was checked. None of that proved the routing decision fired end to end on a real request, and I said so in writing rather than calling it done.

Diagnosis

The obvious end-to-end test is to run a real questionnaire and observe a verbatim hit. It cannot be done, for a circular reason: the store was still empty, and the corpus contained no pairs in a verbatim content class to fill it with. The only way to reach the branch through the normal path was to first fix the thing the test was meant to verify.

A test that ran the whole import-and-answer pipeline would also be testing the pipeline. If it failed, the cause could be extraction, classification, hashing, matching, or the routing decision, and only the last one was under examination.

Decision

Seed the store directly with a single entry, in a verbatim content class, carrying a known distinctive answer string. Upload a real questionnaire containing that exact question, process it, and assert the returned answer is byte-identical to what was seeded and that the audit event records no model cost.

Then change one field on that same seeded entry, the content class, from verbatim to narrative, and run the same question again through a fresh questionnaire. The answer should now come back reworded, still carrying the fact, with a model cost recorded against it.

One variable moves. The question text, the stored answer, the hash, the upload path and the corpus are all held identical between the two runs, so a difference in the output can only have come from the routing decision. It isolates the branch from the import mechanics completely, which is the only reason it proves anything.

Reproducing the hash was its own small task. The key is a normalisation that lowercases, drops every non-alphanumeric character with no substitution, collapses whitespace and trims. Get one step of that wrong and the seeded row is simply never found, and the test fails looking exactly like a routing failure.

The note I left myself in the plan is the part I would defend hardest. The audit event’s tier field records the lookup tier, which is the same in both runs, so it cannot distinguish the two branches. The cost field is what does. So the instruction was to check which field actually differs between the two runs before declaring a pass, rather than assuming. A test that asserts on the wrong field passes for the wrong reason, and a green test asserting nothing is worse than no test, because it stops anyone looking again.

Two claims I took out of my own plan

Symptom

Both were in a document I had written and was about to work from.

Diagnosis

The first was the headline. Fixing the admission gate reads as “cache starts working, hits become free”, and that is not what it buys. The response step selects verbatim versus rewritten on the shape of the answer, not on the tier that served it. A generated entry has no content class, so it routes to the rewriting branch and a hit still costs a model call. It is cheaper than the full retrieve-and-generate path, not free. My estimate of the partial win was about a third, and I wrote it down as an estimate.

Behind that sits the real ceiling, which I also wrote down rather than buried. The reference linking a cached entry back to the library pair it came from was null on all 658 events. Until matching works, the verbatim branch has nothing to inherit a classification from, so the backfill and the classifier are groundwork rather than an immediate saving. Everyone’s plan, including mine, had deferred investigating that, and it is the thing that decides how much any of this is worth.

The second was a guard. A bid can carry a house tone of voice, and an answer written in one bid’s tone should not be cached and served to a differently-toned bid. Correct in principle, and already planned. I checked: no bid sets a tone. The guard protects against a state nothing in the system was in.

Decision

State the partial figure as a third rather than as free, name the null reference as the real ceiling on the whole idea, and drop the tone work to last. The merge rule is recorded for whoever picks it up, because the ordering is the whole bug: cache the untoned answer first, then apply tone to the copy being returned, or the poisoning it prevents comes straight back.

Deferring the tone guard was the easy call. Downgrading my own headline from “free hits” to “about a third, and here is what caps it” was the useful one, because a plan that oversells its payoff gets sequenced against other work on a number that isn’t real.

A thing I keep relearning: the interesting question about a fix is not whether it works but how much it is worth, and those have different answers surprisingly often. Three separate defects stacked on this path, each hiding the next. Fixing the first two was correct, and the honest description of the result is still “the cheap path can now run”, not “the cheap path is now cheap”.

Where it landed

0 of 658logged answering attempts served by either cheap tier, across the cache’s entire life

136live answer pairs carrying the content class the verbatim branch keys on. None of them

≈193 sper question on the expensive tier, measured on one sampled event, alongside ≈$0.006

≈1/3of the payoff the write-path fix actually buys. My own estimate, stated instead of “free”

Verified

The admission gate reads a confidence that belongs to the answer, so generated answers can enter the cache for the first time
Generated entries record a confidence of zero honestly and are admitted on a source flag, with the interface badge suppressed rather than showing a meaningless zero
A content classifier plus a backfill, verified against the live configuration rather than a fixture, so the verbatim branch is reachable
Both fixes landed as one change, because either alone leaves the store holding rows the other cannot route correctly
An end-to-end test designed to move one field between two otherwise identical runs, isolating the routing decision from import and corpus mechanics
The tone guard deferred on the measured basis that no bid sets a tone, with the ordering rule recorded for whoever implements it

What this doesn't solve

The reference linking a cached entry to the library pair it derived from is null on all 658 events. Until that matching works, the classifier and backfill are groundwork rather than a saving, and this is the real ceiling on the cache’s value. It was deferred, by me as much as anyone, and naming it is not the same as fixing it.
The end-to-end test is a documented manual procedure against a running stack, not an automated test file. That matches the existing pattern for live-only checks in this area and it is still worse than a test that runs itself.
The per-question cost and duration were measured on a single sampled event. One sample is enough to establish that the expensive tier is expensive; it is not a distribution, and nothing on this site multiplies it by a question count.
The write-path fix means generated answers now enter a global cache. That makes the existing write-back guards load-bearing in a way they were not while nothing was being written at all. A cache with no entries cannot leak one bid’s content into another.
Whether hit quality holds as the store fills is unmeasured, because until this work it could not be measured. The ledger that found the problem is also the instrument for that, and it is now worth reading regularly rather than once.

What I'd do differently

Read the ledger on day one, not months later. The system had been logging exactly the number that proved this the whole time. Two queries, how many rows are in the store and how many events name each tier, turned an unexamined assumption into a finding, and either could have been run the week the feature shipped. A cache is one of the few things that reports its own effectiveness, and the reporting is worthless if nobody looks.

Never let two different quantities share a name. Confidence in this answer and confidence that this question matched something met at one gate under one word, and the collision is the entire bug. The fix is a rename and a separate field, which is cheap at design time and, once production data exists on both sides of the confusion, is not.

Check the payoff before sequencing the work, not after. I found the null reference that caps this whole idea while writing up the plan to fix the gate, late enough that “revive the cache” had already been argued for on the strength of free hits. The question “what does this actually buy, and what has to be true for that” belongs at the top of the plan, where it can still change which plan gets written.