The only answer whose value is the customer's name

The corpus was not thin. It was shattered.

Symptom

Recent bids returned the wrong references. Two hypotheses were on the table: too few relevant references, or broken matching.

Diagnosis

Neither. I measured against the running database rather than reasoning about the code, and the numbers settled it: 108 reference-shaped answers existed, behind 108 distinct question wordings, with at most one reference reachable under any single wording.

That last figure is the whole story. The importer deduplicates on the pair of source and question hash, deliberately discarding which file a row came from, and the last occurrence wins. The corpus stores one reference per row against a repeating question, which is exactly the shape question-hash dedup cannot survive: it treats “three references” as “one answer, imported three times”.

I replayed the real import file through that logic to see the size of it: 283 of 3,141 answers discarded, eight of them reference-shaped. Three references answering one repeating question collapse to whichever row happened to be last.

So “give me three references” was unanswerable by construction. There was no question you could ask that had three. And to reach one particular reference you had to ask the past bid’s question near-verbatim.

Decision

Stop trying to serve references from the question-keyed store, and stop trying to guard it. The mismatch was not a prediction about caching to be mitigated. It was already deleting the corpus at import time, silently, with the survivor chosen by row order.

The corpus check also sized the solution, which is why it was worth doing before designing anything: on the order of twenty distinct customers, tens of references rather than hundreds. That answers “curation or machinery” in favour of curation. And thirteen of the reference rows in the export had been hand-added directly rather than extracted from any bid, so somebody was already curating references by hand. The habit existed; it had no home in the system.

Keyed on the question

three references → three rows, one question hash
dedup key       (source, question hash)
source file     deliberately dropped from the key
winner          the last row. by row order.
replayed        283 of 3,141 answers discarded
                8 of them reference-shaped
retrievable     1 reference per question wording
so “three refs” unanswerable by construction

Keyed on the engagement

one engagement  → one structured record
three refs      → three records. nothing to
                  collapse them onto.
selection       once per bid, over the register
ranked by       product overlap → sector → recency
pins/exclusions outrank every ranking axis
the test        three DISTINCT customers across
                the three blocks  ← fails if
                selection goes per-cell again

A ranker with nothing to rank on, and a lane that had never executed

Symptom

Which reference came back correlated with nothing about the bid being answered.

Diagnosis

Two separate causes, and neither was a bug in the sense of a mistake.

The ranker had nothing to rank on. Retrieval scores the question field at twice the weight of the answer field, deliberately, because it is matching question-to-question. The informative tokens in a references question are provide, references, work, relevant, service. None of them name a product, a sector or a scale. So the winner is whichever past bid phrased its references question most similarly, a quantity uncorrelated with the bid being answered. The ranker was working; the input carried no signal.

The lane built to fix this had never run. A content class for customer references existed, and so did supersession by engagement date and a product-restricted retrieval pass. Of 114 library documents, zero were classed as a customer reference, and zero carried a customer, product or date. With no product values anywhere, the distinct-product lookup returned empty, the product matcher matched nothing, and the restricted pass had never fired for any question in any bid. The prior record described that lane as un-validated. It was dormant.

And the sting: populating the data would not have been sufficient. The product terms that open that pass are derived from the question text, and a references question names no product. After a full migration, reference selection would still have been embedding proximity, which puts two competing vendors’ equivalent products side by side.

Decision

The product and sector intent has to come from the bid, set explicitly, never inferred from the question, because on a references tab there is nothing to infer from. The cells read description of services, phone number, email.

I had found the same thing one layer away, in a different feature, on a different document class: an admission signal keyed on question wording, against questions written to be vendor-neutral. Two independent instances of one mistake is a pattern worth naming rather than fixing twice. If the signal you need is a property of the engagement, do not try to read it out of the sentence.

Selection therefore runs in Go and Mongo before any search, which is also forced rather than preferred: the register is encrypted, so it can supply neither a search-index vocabulary nor a searchable index of its contact fields. Ranking is product overlap, then sector, then recency.

The requirement said selection must be steerable by product and sector. Measured, the service tag was populated on 36 of 49 reference rows and the industry tag on 2. So sector matching was aspirational, and the honest version of the requirement is product-and-recency until somebody captures sector going forward. Writing that down was cheaper than shipping a ranking axis with no data behind it and letting it look like it worked.

Twenty-one blanks, and the two that were filled were the dangerous ones

Symptom

On a real completed export, the references tab came back with every cell still showing the template’s placeholder text, three reference blocks of seven fields and twenty-one blanks, while the questions on either side of it received long, detailed answers.

Diagnosis

Extraction was not the failure. All twenty-one cells had been extracted as questions, and all twenty-one were marked needs manual entry, pending, with an empty answer. That is the model behaving correctly.

A cell reading phone number, stripped of its Reference 2 parent block and of the section preamble asking for three examples of similar scope, is genuinely unanswerable. Nothing told the answerer that those seven fields describe one engagement, that there are three of them, or that reference two must name a different customer from reference one. That last one is cross-row consistency, and there was no mechanism for it anywhere in the answering path. Even a perfect reference pool could not have helped.

Two cells had been answered, from the company-profile block, meaning the export shipped the buyer a referee phone number and email belonging to the supplier. A buyer ringing that number to check a reference reaches us. Blank is a visible gap; a plausible wrong number is not.

Decision

Ship the safety fix first, on its own, before any of the register work. It needed none of it.

It is a post-check in Go, not a suppression, because the profile block is injected once per batch into a shared preamble. There is no per-question place to withhold it from, and rebuilding the preamble per row to hide four fields would make every row in the run pay to protect a handful. The prompt states the rule as well, worded as leave it blank rather than be careful, because a model told to be careful with a value it has been handed still uses it. But an instruction is not a guarantee: the defect being fixed is precisely a model doing something a prompt could have told it not to.

The check refuses rather than corrects, and there is no right value to substitute. If the register holds the referee’s real number then the reference path has already filled the cell, and if it does not, then an empty cell asking for a person’s attention is the honest state. It matches on every configured legal entity’s contact values rather than just the one the bid is submitted under, since the failure mode is a model reaching for a number from the block. Values below a minimum length are dropped, because a two-character “fact” would match half the answers on the sheet. And a missing profile or a failed lookup disables the check with a log line rather than failing the run, because it is a backstop on top of the prompt rule and losing it must not cost the bid its answers.

This exhibit is the reason the whole feature got built rather than deferred. A tab that is entirely blank is embarrassing and obvious; a tab that is blank except for two cells confidently naming the wrong person’s phone number is a failed reference check, and the bid team would not have caught it, because a filled cell looks like a working one.

The ceiling nobody had computed

Symptom

Every plan so far, including one of mine, had treated this as a retrieval problem: get the matching to pick the right reference prose.

Diagnosis

I checked how many of the tab’s seven fields per reference existed anywhere in the corpus at all. Three: the services description, how long the relationship had run, and the customer’s name. The other four, being the engagement lead, the referee’s name and position, their phone number and their email, existed nowhere. The corpus said contact details available on request, which is the company’s own settled position: naming the customer is disclosable, naming the referee is not.

So the ceiling for every retrieval-side plan on that returnable was nine of twenty-one blanks, and the remaining twelve needed data the system did not hold and a decision it had not made. That reframes the work as a data-and-selection problem rather than a retrieval problem, which is what reversed the earlier decision to reject a separate reference store. That rejection had assumed a reference is prose to be cited, and a grid disproves it. Chunk retrieval returns passages; a grid needs discrete field values, one of which is a phone number.

Decision

A structured, encrypted register of engagements with a global editing surface, and four rules that each exist because of a specific way this fails.

Filling is copying, not generating. Contact values are copied verbatim or left blank. Only prose fields may be adapted to a word limit, and a cell that maps to no field is left for a human rather than filled from a near miss.

Selection runs once per bid, not per cell. Otherwise the three blocks disagree about which customer they describe, which is the exact defect that made the earlier approach unusable.

It offers rather than fills, as a swappable shortlist on the bid page. The team’s review is what keeps the register trustworthy, and nothing in code can know that a buyer excluded a competitor’s customer.

A block label must carry a number. An unnumbered References heading identifies a section, not one engagement inside it, and treating it as a block collapses every reference on the tab into one, which is the original defect: three identical customer-name rows all receiving the first customer. The same care distinguishes a field label from a real question that happens to sit inside a references section. Measured field labels run six to forty-five characters, and a genuine question routed down the copying path would come back blank, because that path answers only what it can copy.

One detail with a longer shadow than its size: the register’s editable fields deliberately do not omit empty values when they serialise. The update is built from the struct’s stored form, so an omitted empty field leaves the old value in place, and the field becomes settable but never clearable. Clearing a departed referee’s phone number is exactly that operation, and it is the single most likely edit anyone will ever make to this collection.