Case study · AI systems engineering

Answering tender questionnaires from a knowledge base that contradicted itself

Retrieval, caching and measurement work on a bid-response module.

Go
MongoDB
AWS Bedrock
OpenSearch
Embeddings
Retrieval-augmented generation

Answering a tender questionnaire is the same work every time: several hundred questions, most of which the company has answered before, in a document nobody can find. The obvious build is retrieval over past answers. The part I did not expect is that most of the engineering went into distrusting the corpus, not into retrieving from it. A retrieval system inherits the correctness of what it retrieves, and this one was quietly answering the same regulatory question both ways.

$1,299/mothe subscription tender-response product the in-house system displaced, gross, as billed

The problem

A bid team answering a tender questionnaire works through several hundred questions, and has answered most of them before: in a previous submission, in a spreadsheet, in a document nobody can now locate. The company had accumulated years of that material. None of it was reachable at the moment it was needed.

Retrieval over past answers is the obvious build. Four things made it harder than that, and only the first is a retrieval problem.

The same question is never asked the same way twice

Two buyers asking about the same control word it differently, number it differently, and split it across a different number of questions. Exact matching finds almost nothing.

The corpus is not a source of truth

It is an accumulation of things the company once said, some superseded, some contradictory. Retrieval treats all of it as equally true.

A wrong answer here is a submitted document

It goes to a buyer under the company name, and some of these questions are legal or regulatory. A hallucinated capability claim is a much worse outcome than a blank.

Cost scales with questions, not with bids

Several hundred questions per questionnaire, each potentially a retrieval and a generation. The naive shape is expensive per bid and gets no cheaper as the corpus grows.

The last one is what the caching is for. The middle two are what most of this page is about.

What it replaced

There was already a commercial product doing this. A subscription tender-response tool, billed at $1,299 a month, which the in-house system displaced. That is the one number on this site where annualising is legitimate: a fixed subscription is $15,588 a year without any assumption about volume, unlike a token count, which needs a message rate I do not have.

Three honest qualifications, because this is the kind of figure that gets repeated without them:

Gross, not net

The replacement is not free to run. It has infrastructure and per-call model costs that are real and recurring, and I do not have that figure. $1,299 a month is the subscription that stopped, not the amount the company is better off by.

I built the system, not the business case

Dropping a vendor is a procurement decision made by people above this work. I can attest to what was built and to the price of what it replaced; I would not claim the decision, and the saving is the company’s outcome rather than a personal one.

Displacement is not feature parity

A mature commercial product does things this does not. What made the trade worth making is not that the in-house version is better in general. It is that it can be shaped to how this company actually bids, which no subscription can be.

That last point is the real argument for building rather than buying, and it is the part a monthly figure obscures. Every one of these exists because the bid team needed it, and none of them is expressible as configuration in a general-purpose tool:

Answers gated on the specific product stack of that bid, so a questionnaire for one product set cannot be answered out of documentation for another
Buyer history carried across tenders: who this buyer is, what was submitted to them before, and whether it won
Customer references hand-pinned or excluded per bid, outranking every automatic ranking axis, because who you name as a referee is a relationship decision and not a retrieval result
Per-bid content-admission classes, set once by a human and applied to every row
Report sections routed per environment, and the sections no honest generator can source left as empty boxes rather than invented

A subscription product optimises for the median bid across all its customers. The value here is that it does not have to.

What I built

A three-tier answering path in front of an existing retrieval pipeline, plus the measurement work that told me what the corpus actually contained.

Already existed

Document extraction and chunking
Vector index and retrieval
One batched generation call per questionnaire
Exact-duplicate collapse on import
A near-duplicate detector, wired to the admin import path

What I added

An exact-match and semantic-match cache in front of retrieval, as an optional stage that degrades to the original path when unwired
The write-back guards that make a global cache safe to share across bids
A measured audit of the corpus, and the contradiction surfacing that came out of it
One shared question normaliser, so the dedup key and the cache key stop disagreeing

Three tiers, cheapest first.
An exact hash match, then a semantic match by cosine similarity, then the original retrieval-and-generation path for whatever is left. Misses are batched into a single generation call and slotted back into the order they were asked.
A shared cache, with guards on what may enter it.
High-confidence generated answers are written back, so the next questionnaire gets them for free. Three categories of answer are refused, and those refusals are the whole reason the cache is safe.
An audit of the corpus against the live database,
rather than against an assumption. That is what found the contradictions, and what established that the visible problem people were complaining about was not the real one.
Contradiction surfacing for a human, not automatic resolution.
Deciding what the company’s actual position is on a regulatory question is a bid-team call. The system’s job is to stop pretending there is one answer when there are two.

Architecture

Answering a questionnaire: three tiers, and the guards on the write-back

Two details do more work than they look like they do.

The tier-2 corpus is loaded once per questionnaire, not once per question. With several hundred questions per run that is the difference between a workable stage and one that costs more than the retrieval it was meant to avoid. Cosine similarity runs in memory over that preloaded set rather than as a second vector index. The cached corpus is small, and a second index is a second thing to keep consistent.

The cache is global and keyed on the question alone. That is what makes it valuable: an answer written during one bid is available to every later bid, which is the entire point of a reusable answer library. It is also what makes it dangerous, and both of the first two exhibits below are consequences of that single design choice.

Four problems worth describing

A cache that would have worked perfectly and never once hit

Symptom: None, and that is the point. Every test passed. The cache wrote entries, reported healthy, and would have returned a hit rate of zero forever in production.
Diagnosis: Cached answers were being encrypted with the per-questionnaire data key, because that is the key every other record in that flow uses and reaching for it was the consistent-looking choice. But the cache is a global corpus whose whole purpose is to be read by a different questionnaire later, and a different questionnaire has a different key. Every entry would have been undecryptable by every reader except the one that wrote it. Within a single questionnaire, which is what the tests exercised, it works flawlessly.
Decision: This was caught in code review, not by me, and the fix was a decision rather than a patch: store the shared corpus unencrypted, matching the imported answers it derives from, and record in the code why. The repository still supports envelope encryption, and the note says plainly that if these entries ever need it they need a single shared key, never the per-questionnaire one. The wrong choice here is invisible in every test that does not span two questionnaires, so the comment is the control.

The bug I find most instructive of the four. It has no symptom, it passes review by consistency, since everything else in that flow is encrypted that way, and the tests that would catch it are exactly the ones nobody writes, because they span two runs.

Two correct features that leaked one buyer's name to the next

Symptom

Answers were coming out of the library with literal template placeholders still in them: {{customer}} and friends, visible in a draft heading for a buyer. Measured across the corpus, 264 of roughly 3,000 answers carried one.

Diagnosis

The placeholder vocabulary turned out to be tiny and dominated by one token, so this was a dictionary problem, not a model problem. Filling them from the bid’s own context is straightforward, and filling them before the retrieved text reaches the generation prompt is the half that matters, because given real prose the model adapts it and given {{customer}} it copies the placeholder through or silently drops the clause.

Then the interaction. Filling makes an answer bid-specific. Write-back makes answers global. Each is correct alone; together they take an answer now containing one buyer’s name and serve it, on a question-hash match, to every future bid. Nothing was wrong with either feature. The defect only exists at the join.

Decision

Write-back now refuses any answer containing the bid’s customer name, a per-answer check, because filling is per answer. That sits alongside two guards of the same shape already there for the same reason: an answer written in a non-default tone, and an answer framed by one tender’s evaluation criteria, are both refused for the cache. Placeholders naming a person or a third party are deliberately not filled at all: no value we hold answers them, so they stay visible and trip the existing export block rather than being guessed at. That left 261 of 264 resolving cleanly and 3 needing a human, which is the correct split.

The lesson I actually took: a global cache needs an explicit rule about what may enter it, and that rule has to be re-checked every time a feature makes answers more specific. Three independent guards on the same arrow is not duplication. It is the same hazard arriving from three directions.

The library answered a compliance question both ways

Symptom

A remark from the bid team, not an error: there are a lot of duplicate answers in here. I went to measure how bad the duplication was.

Diagnosis

The duplication was mostly legitimate and should not be collapsed. Near-identical questions genuinely asked different things, and merging them would have destroyed information. So the reported problem was not a problem. Underneath it was one that was.

Clustering the corpus by topic surfaced a group of nine answers to what is substantively the same yes/no regulatory-compliance question. They did not agree. Several said yes, one said no, and the one that said no had the highest usage count in the group, so it was the answer most likely to be retrieved and submitted. A buyer can check some of these claims against a public register. Whichever answer won retrieval was the one that shipped, and nothing in the system had an opinion about which was right.

Diagnosis, continued

Then the part that changed how I read the rest of the codebase. A near-duplicate detector already existed, with a full review-and-merge flow behind it, and it had never run on this corpus, because it was wired only to the administrative import path, while the bid-answering path reached the database by a different route. Zero flags across 3,873 pairs. The answer-quality checks that did run flag thin and off-target answers and have no concept of two answers disagreeing with each other.

A second finding made this worse: the merged status the review flow writes was read by nothing. The query that assembles the retrieval corpus filtered on other fields entirely, so every merge an administrator had ever approved was cosmetic, and a superseded duplicate could still be served into a live bid.

Decision

Surface, do not resolve. Deciding the company’s actual regulatory position is a bid-team judgement and not a thing to infer from usage counts. The work was to make the disagreement visible where a human is already looking, make the existing detector run on the path that matters, and make the merged status actually remove a row from the retrieval corpus, because until it does, every other fix here is cosmetic too.

Corpus audit and contradiction detection

Measured against the live corpus

qa_pairs                        3,873
topic cluster (one yes/no reg. question)     9 answers
  answers asserting yes                      several
  answers asserting no                       1   ← highest usage in group
near-duplicate flags set                     0
  detector wired to                          admin import path only
answer-quality flags                         158   (thin / off-target only)
merged status read by corpus query           no    ← merges were cosmetic

After: flagged, and merges finally mean something

detector runs on                             the answering path
cluster surfaced                             needs human review
merged status                                filtered by corpus query
resolution                                   bid team, not usage count

The reported problem was not the real problem, and the real one was invisible because a control that would have caught it existed, passed review, and was connected to the wrong path. That is a more common failure than a missing control, and much harder to see.

A dedup key too literal to work, and a scan nobody had noticed

Symptom

Exact deduplication reported perfect results: zero hash collisions, nothing missing a hash. The corpus still visibly contained pairs of identical answers.

Diagnosis

The dedup key normalised case and whitespace, but not punctuation. Re-hashing the corpus under punctuation-stripping normalisation collapsed 145 groups, merging 152 documents away, 3.9% of the corpus. The clearest case was two rows extracted from the same source document, with byte-identical answers, split apart because a word processor had autocorrected one pair of quotes into smart quotes. Under the literal key they are two separate answers forever.

The normaliser I needed already existed, as the cache’s tier-1 key: Unicode-aware, with punctuation folded and digits deliberately kept, so numbered questions like 3.1 and 3.2 stay distinct. Two components were normalising questions for the same purpose to two different standards.

While measuring, a second thing: the import upsert filtered on a field pair with no index behind it. Every imported row scanned the full collection, on the order of 11 million document examinations for a three-thousand-row import.

Decision

Extract the existing normaliser into a leaf package both sides can import, rather than reimplementing or copying it. The package layering only allowed one direction, which is what had produced two implementations in the first place. The hard constraint: output must stay byte-identical, because several hundred cached entries are keyed on that hash and would all have silently missed tier 1 otherwise. A pure move, verified as one. Plus the missing index, which was two lines.

Not a clever fix, but the constraint is the interesting part: a refactor that changes a hash function’s output by one byte invalidates a cache without failing anything. “Pure move” was a requirement, not a description.

3,873answers audited against the live database

152cosmetic duplicates found (3.9%)

60groups needing a human decision

0contradiction flags before the work

Where it landed

Verified

A three-tier answering path in front of the existing pipeline, degrading to exactly the original behaviour when the cache stage is unwired
Tier-1 and tier-2 hits serve at no model cost, with the candidate corpus loaded once per questionnaire rather than once per question
A cross-questionnaire cache that *can* return hits. The encryption choice that would have silently prevented that was caught and documented. Whether it did is a separate question, answered on another page and answered badly; see the residual note below
Three independent write-back guards, so no bid-specific answer enters the shared corpus
261 of 264 placeholder-carrying answers resolving cleanly, with the remaining 3 held for a human rather than guessed
A measured audit of 3,873 answers, replacing an assumption about the corpus with figures
The existing near-duplicate detector running on the path that serves live bids

Properties of the problem, not of the fix

A retrieval system inherits the correctness of its corpus. Nothing here makes a wrong stored answer right; it can only surface that two stored answers disagree.
Whether the company is right about a regulatory question is not a technical judgement, and no amount of ranking makes it one.
The cache is global by design, which is what makes it useful across bids and what makes every write-back rule load-bearing rather than defensive.
Fixing the encryption-key bug in exhibit 1 made cross-questionnaire hits *possible*. It did not make them happen. Counted against the database later, the store held zero rows against 658 logged answering attempts, because two further defects sat behind that one: an admission gate reading a match score as a confidence, and an empty content class that made the free response branch unreachable. That is [its own case study](/work/canonical-cache), and it is the correction to the most natural misreading of this page. Three separate bugs stood between this design and its first cache hit, and exhibit 1 only removed the first.
Cost and latency figures in the telemetry are estimates, not measured billing, and are labelled that way in the code.
Semantic matching is only as good as the embedding’s notion of “the same question”, and two tender questions can differ by one clause that changes the answer entirely.

What I'd do differently

Audit the corpus before building retrieval over it, not after. Every interesting finding on this page came from querying the live database and comparing it to what the code assumed. I did that because someone mentioned duplicates in passing. It should have been the first task rather than a follow-up. The retrieval quality ceiling was set by the corpus, and I spent time tuning underneath a ceiling I hadn’t measured.

Treat “is this control actually wired to this path?” as a standing question. A detector that exists, passes review and runs on the wrong path is worse than no detector, because it reads as covered. I now check what calls a safeguard before trusting that it runs.

Write the rule for what may enter a shared cache once, explicitly. Three separate guards arrived one at a time, each after a feature made answers more specific. They should have been one documented predicate from the start, so the fourth such feature has an obvious place to declare itself instead of quietly poisoning the corpus.