The half of the report I refused to generate

Twelve slides, and only one of them a missing capability

Symptom

A request, not a bug: make the generated report match the kickoff deck the bids team produces by hand. Twelve slides of it.

Diagnosis

I graded the deck against what the system already did, slide by slide, and it was not twelve gaps.

Most of it was already covered by the existing report structure. Two of the remaining slides were already extracted. The buyer’s objectives and the weighted evaluation criteria were both being parsed out of the tender and stored as structured records, and no screen displayed either of them. That is why the deck read as a gap: the capability existed and was invisible. Exactly one slide was a genuine retrieval gap, covering contract vehicle, pricing basis and evaluation period.

And about two fifths of the deck was not in the tender at any price: relationship history, decision makers by name, win strategy, a SWOT, next steps with owners. The tender does not contain the sentence “a competitor already has a footprint here”, and it never will. That half is what makes the deck a kickoff rather than a summary.

There was also a trap in the “already extracted” rows. The structured criteria were being injected into the report prompt as framing under an explicit instruction that they must never carry a citation marker, because the marker parser drops markers it cannot resolve to a retrieved passage. So a new criterion asking for the evaluation weightings could not be grounded on the extraction that already held them, weighted and structured. The model would have to re-find them in raw passages, or return nothing.

Decision

Three calls, and the cheapest one first.

Show them the feature before building anything. The report templates were already user-editable through an existing settings screen, so sections could be added and reordered without code. That may dissolve half of a request, and reshaping the compiled-in default is still worth doing afterwards because it is what every new bid gets before anyone edits anything. But the urgency of everything else depends on that conversation.

Reshape the existing report; do not add a third type. A new type needs generate, fetch and status handlers, a template mapping, a repository case, a card and a route, and it leaves the old type in place as a near-duplicate, meaning two model calls per bid and two section sets to keep in step. Reshaping is an edit to one data structure. Related: rename the user-facing label in the five places it appears, and never the persisted type string, which is stored on saved templates and on every generated report and filtered on. Changing it orphans both silently.

Leave the unsourceable half as empty boxes. Hand-authored slots, following the shape of the non-AI field the lane already had, with a pencil affordance instead of Regenerate, and the model call never touching them. Rather than re-derive the extracted criteria with a large model, surface the structured records that already existed: they are weighted and exact, and re-deriving them costs a criterion’s share of the passage budget to produce something worse.

I wrote down the risk the finished set carries, because it is not a code problem and would otherwise surface as disappointment. Done in full, the kickoff ends up spread over three screens: generated points, hand-authored assessment, and a timeline in the schedule that already owns dates and owners. There is no report export of any kind. Each placement is individually right, and duplicating dates onto a report is the worse bug. But nobody can present the result, so the deck may still get rebuilt by hand. If the deck itself is the deliverable rather than its content, that is a separate piece of work and it should be priced before the hand-authored half is built, not after.

A grader that had already seen the mark scheme

Symptom

The obvious objection to the most-requested check in the whole set. Buyers publish weighted evaluation criteria; the system reads them; a report that scores the answers against those criteria is flattering by construction, because the model that wrote the answers had the criteria in front of it.

Diagnosis

True, and much narrower than it sounds, which is only knowable by tracing where the rubric actually reaches the model. It is set in exactly one place and reaches generation through one preamble and one system-prompt sentence. So:

the questionnaire and returnable lane never sees the rubric at all; and even in the lane that does, only rows written by full retrieval see it. Cache-served rows, rows flagged as needing a document, reference rows and anything a human edited never enter that prompt. Contamination applies to one tier of one lane. Everywhere else, assessing an answer against the rubric is genuinely new information.

So the report should carry the distinction rather than pretend it away, and state how many of the rows it assessed were written with the rubric in view. A reader who knows a dozen of three hundred rows were rubric-framed can weigh the verdict; one handed a bare grade cannot.

That count was not derivable from what was stored. My own first draft of the plan said the row’s answer-source field already had it, and that did not survive contact with the code. The source field returns the same value for a cache-served row and a retrieval-written one, so it cannot separate them. The row carried no tier at all; the tier existed as transient telemetry that never landed. And the event log that did record the tier minted a synthetic identifier for the row and held no bid reference, so its events could not be joined back to rows or filtered to a bid.

Decision

Stamp the tier on the row at answer time. Additive, omitted when absent, so every pre-existing row decodes as unset and is reported as unknown provenance rather than folded into either side.

Rejected: inferring the tier from the shape of a row’s citations, which is a second answer that goes silently wrong the first time the citation ordering changes; and widening the event log, which is a cross-tenant audit record with readers outside this feature.

Then two commitments whose whole purpose is to make flattery expensive rather than to hope it away.

Rank and band, never grade. The output is the weakest answers against the highest-weighted criteria, and why, with a band per criterion. No number out of a hundred. A scale that always returns seventy-eight is worse than no scale, a ranking survives miscalibration in a way an absolute grade does not, and a ranking is the only form of this a bid manager can act on with a day left.

Every adverse verdict must quote the answer it indicts, and the quote is substring-checked against the row’s stored text in code, with the verdict dropped if it does not match. That is the load-bearing part: it makes a fabricated criticism exactly as costly as a fabricated citation, and it is checkable without a second model call. Combined with an adversarial prompt, telling the model it is the buyer’s panel looking for reasons to score this down rather than our own reviewer looking for reassurance, the failure mode inverts usefully. An uncontaminated grader that cannot substantiate a criticism produces fewer findings, not flattering ones.

A score

output          one number out of 100
calibration     none established
contamination   unstated. some rows were
                written with the rubric in prompt
criticism       free. costs the model nothing
reader can ask  “is this a mirror?”  → no answer

A ranking that has to show its working

output          weakest-first ranking + a band
                per criterion. no number.
criticism       must quote the row it indicts
                → quote substring-checked in Go,
                  unmatched verdict dropped
contamination   counted and stated: N of M rows
                rubric-framed; pre-field rows
                reported as unknown provenance
framing         the buyer’s panel, not our own
                reviewer → fewer findings, not
                friendlier ones

A report whose sources are rows, not passages

Symptom

Every existing report on the lane was retrieval-shaped: build a pool of passages from the search index, run one model call, map the markers in its output back onto citations. This report’s input is thousands of database rows. The question was how much of that lane could honestly be reused.

Diagnosis

Split evenly, and not along the line I expected.

The citation machinery reuses unchanged, because the pool entry type turned out to have no dependency on the search engine at all. It is a plain struct. So input assembly can build a synthetic pool: one entry per answer row, content being the question plus its answer, identifiers being the row’s own, under a new citation scope for answer rows. A precedent already existed for a scope that resolves to a database object rather than an indexed passage, so this was following a pattern rather than inventing one. Every judgement finding then resolves to the exact row it is about, through code nobody had to fork.

The enclosing generate function does not reuse, because it owns four things this report does not want: it retrieves from the search index for you, it hard-fails when the pool comes back empty, it runs exactly one model call with one system prompt, and it applies the wrong staleness rule. Its prompt also hard-codes “source passages from the tender documents” and “only use information present in these passages”, wrong on both counts for a pool of answer rows. So it takes a seam: hoist the pool to a parameter and pass the source-block wording through.

Three sharp edges around the citation choice. Markers are per-call, so with one call per check [1] means a different row in each. They have to be renumbered onto a single counter on merge, the way another lane already did for its two citation sources. The scope needs a branch in four resolve call sites. And the interesting one: the interface’s citation panel calls the whole-document route unconditionally, not the passage route. Branch only the passage endpoint and every citation renders “source removed” in the app while the endpoint you tested by hand works perfectly.

Decision

Reuse the citation machinery, seam the generator, and write down which of the two it was for each piece, because “reuse the report lane” was the sentence in my own first draft, and it was too broad to build from.

Two checks stayed deterministic against a suggestion to have a model do them: readability, and scanning for a past customer’s name left in an answer. Both are arithmetic and string matching; a model adds cost, variance and no accuracy. Symmetrically, the deterministic tier’s findings are fed into the judgement prompts as stated facts, so a model asked about coverage is not also counting rows.

The output shapes were reused too, and one of them cost more than it looked. The export-notice shape was ideal for deterministic findings, with a stable kind, a severity and a subject list the interface turns into links, but it had only ever been a transient return value from an export call. So it had no persistence and, more importantly, the report encryption path did not cover it. Its text and every subject entry carry question labels and quoted answer spans, which is customer content, and it would have been written to the database in the clear. Extending the encrypt and decrypt paths over the new field was part of shipping it, not a follow-up, because the document-level encryption flag means “every content field on this document is encrypted”. A half-covered list breaks the flag’s meaning rather than one field’s confidentiality.

The staleness rule is the neatest example of an inherited concept meaning the wrong thing. On this lane, stale means a tender document was added or removed. A review of the answers goes stale when an answer changes, a different and far more frequent event. Choosing a new trigger was not enough: two existing paths stamp reports stale on document activity regardless, one of them an untyped bulk update called from the indexing worker, so the new type needed actively excluding from both. A third path was safe only by accident, filtering on a field this report’s citations do not populate. I wrote the accident down rather than relying on it silently.

Adding six sections makes the other thirteen worse

Symptom

None yet. This is the cost I found while pricing the reshape in exhibit 1, and it would have shipped as an invisible quality regression on the sections nobody asked me to touch.

Diagnosis

The passage pool handed to the model is capped for the whole report, and the allocator round-robins one passage per section per pass until the cap is reached. So the cap divided by the number of sections is what each section actually gets.

At thirteen sections each criterion got about three distinct passages. At twenty, with the cap unchanged, it gets exactly two. There is a second constant that bounds how many passages each query fetches, and it is easy to mistake for an allowance. It bounds what is fetched, not what survives allocation.

Which means adding six criteria silently dilutes every criterion that already worked, and the symptom would be a thinner answer on an unrelated section that someone would naturally blame on query wording. The pool is a hard cap, not a per-section budget.

Decision

Raise the cap to hold parity at three passages per section, and treat that as part of the change rather than a tuning follow-up. The constant is shared with the other report type, so that report’s prompt lengthens too. Worth stating, and comfortably inside the model’s context, so the cost is tokens and latency rather than truncation.

Then the validation item that actually protects the thirteen: no existing criterion loses citations relative to a pre-change run on the same tender. That is the regression the new sections can cause invisibly, and it needs a before-run to exist, which means capturing it before shipping rather than reconstructing it afterwards.

Two related costs written down rather than discovered. Retrieval is a plain sequential loop, so thirteen to twenty sections is fifty per cent more round trips to the search index. When the after-measurement comes in slower, that is half the reason, and attributing all of it to the model call would send the next person optimising the wrong thing. And an empty section has three possible causes in order of likelihood: the passage budget, the query wording, or a tender that genuinely does not say. Check the pool size before rewording anything.

A cluster of smaller traps from the same work, all of the same family: a count that means something other than what it looks like. An empty primary answer does not mean unanswered, because a dropdown-column row stores its values in cells and leaves the primary field blank, so a coverage check that counts empty answers reports false gaps across every choice-column tab. Cache-served rows have prose and no cells, so cell-level and row-level counts legitimately differ and neither can be derived from the other. Normalising a question for hashing strips punctuation and case, so every “Comments” and “Please provide details” cell across six tabs hashes identically. A duplicate-answer check without a minimum length and a boilerplate skip list produces findings that are technically correct and useless, in volume. And a readability score penalises exactly the register the technical tone setting asks for, so it is reported next to the tone it is fighting, and gates nothing.