The gate that would have been a deletion

A gate that could only subtract

Symptom

No symptom. This one was caught while checking the design against the corpus before writing the filter, which is the only reason it is a paragraph here rather than a quarter of the library going quietly unreachable.

Diagnosis

The design was the obvious one: exclude vendor documentation from the unrestricted retrieval pass, and admit it through the restricted pass when the question names a product.

Two facts, checked separately, killed it.

First, the tag the restricted pass filters on was present on no document and no chunk in the library. Not sparse, absent. So the exclusion would have removed a large block of chunks from the corpus and the admission path would have matched nothing, ever. Backfilling the tags fixes the arithmetic and not the design.

Second, and worse: I read the actual questions. Buyers write requirements to be vendor-neutral, so the product names live in the answer, not the question. Across a real returnable of several hundred rows the number of rows naming a product the company resells was zero, and the closest near-hit named the buyer’s own internal tooling, which would have opened the gate to the wrong vendor’s documentation entirely.

And none of it would have surfaced. The unrestricted pass always runs, so a restricted pass that matches nothing produces no error and no empty result, just a marginally worse answer, with no signal that a lane was closed.

Decision

Move the admission signal from the question to the bid. Whether a bid proposes a given product is a fact about the bid, known before a single row is answered, and it is one human input that covers every row rather than several hundred failed inferences.

The field for it already existed on the bid, with a doc comment making exactly this argument, and nothing wrote to it. So the change had to include the control that populates it. Otherwise the filter keys off an empty field and the same silent failure recurs one level up, which is precisely how the tag ended up on nothing in the first place. Anything read as a gate needs a visible, intentional writer.

The question-text signal stayed, unioned with the bid’s stack rather than replacing it. On the rare row that does name a product it is correct, and as a bonus rather than the mechanism it is harmless.

The reusable part is not the fix, it’s the check. “Which existing records satisfy this predicate, and which real inputs trigger this branch” is two queries and a read of the input data, and it is the cheapest possible time to find out that a filter has no domain. Both answers here were zero.

Two vocabularies that could never match, and an admin screen that said they did

Symptom

A tag applied through the interface and visibly present on a document contributed nothing to retrieval.

Diagnosis

Three mismatches stacked on one field.

The stored tag is one canonical token per product line. The names people actually use are surface forms: a vendor’s name, a product’s marketing name, an abbreviation, a module name that shares no substring with any of them. Vendor product naming is deliberately not systematic, so there is no rule to derive one from the other.

The retrieval filter is a whole-term, case-sensitive keyword match. A tag differing only in case is not a near miss; it is a non-match.

And the database query behind the admin list is a case-insensitive regular expression. So the screen you would use to confirm your tag had worked matches loosely, finds the document, and shows you a success, while the retrieval path that matches strictly finds nothing. The verification surface disagreed with the enforcement surface, in the direction that hides the bug.

Decision

One many-to-one alias map, resolving any surface form to the canonical tag, applied at every boundary that reads or writes a tag: the ingest path, the retag path, the admission set built from the bid, and the question-text scan. A resolver used on three of four paths is a resolver that produces inconsistent data on the fourth.

Normalisation happens on the way in, not on the way out. Fixing this at query time by lowercasing the filter would have left the index holding whatever spellings had already been written, and the next path added would have to remember to lowercase too.

Three surfaces, three vocabularies

stored tag      whatever spelling the writer used
retrieval       whole-term, case-SENSITIVE keyword
                → differing case is a non-match
admin list      case-insensitive regex
                → finds it, reports success  ← the lie
surface forms   vendor ≠ product ≠ abbreviation
                (no derivable rule between them)

One vocabulary, normalised on write

alias map       many surface forms → one token
applied at      ingest · retag · bid stack ·
                question scan   (all four, or none)
normalise on    write, not read
                → index cannot hold a spelling that
                  only some query paths can match
verification    now agrees with enforcement

An indexing worker that stamped nothing, and a corpus that looked healthy

Symptom

The library’s classification coverage was complete. Every document classed, nothing untagged. The index looked exactly like an index whose tagging works.

Diagnosis

The worker that extracts, chunks and indexes runs as a compiled binary in its own container with no source mounted, so it does not pick up code changes until the image is rebuilt. The running image predated the commit that added tag stamping to the indexing path. Every tag in the index had arrived through the separate re-stamping endpoint, which is an update over already-indexed chunks, not through indexing. Past re-stamps are exactly what made the coverage look complete.

So a newly ingested document would be indexed with no tags at all, and that is the worst possible state for this design, because it is wrong in both directions at once. An untagged chunk passes straight through an exclusion filter, so the filter fails open, and is simultaneously invisible to the positive filter that admits it deliberately. Untagged means “always allowed and never admitted”.

Decision

Rebuild the worker image as a prerequisite of the change, not a follow-up check, and verify in the search index rather than in the database. The database row is written by a different code path from the indexed chunk, so a correct-looking record proves nothing about what retrieval will filter on. The check that matters is: ingest one document, then query the index for a chunk carrying the tag.

I wrote the deployment note as a hard ordering, image first and then anything that depends on stamping, because this class of bug is invisible from inside the repository. The code was correct and merged. It just wasn’t running.

Two ingest guards came out of the same read-through, both of the same shape: a failure that writes plausible data. Extraction returns a handful of sentinel strings alongside a nil error for files it cannot read, and the library branch checked only the error, so a scanned image-only document would be chunked, embedded, indexed and marked complete with an error message as its entire content. Separately, deriving the file-type field from the object key rather than the content type made the extractor prepend an “unsupported type” note to plain-text documents, which lands in chunk zero of every one of them, while every other extension kept working, so the ingested tree looked correct.

A thousand keys, and a step that deletes

Symptom

Caught in review of my own plan for the reconciler, before it ran on anything.

Diagnosis

The reconciler compares the vendor tree in object storage against what the library has indexed, so it needs to list a prefix. No listing method existed on the storage service. The nearest prior art, a method that walks a prefix for a different feature, has two defects: it issues a single request and never follows the continuation token, so it silently stops at the API’s thousand-key page limit, and it reads a bucket configured for a different purpose.

Copying that, as one does, into a reconciler whose job includes acting on keys the library does not have and keys the library has that the store does not, makes every object past the first page read as absent. In a step that deletes, “absent” means deleted. That is not a coverage gap, it is data loss, and the vendor tree is comfortably large enough to reach the boundary.

Decision

Write the paginated listing properly, and assert the page boundary in a test, because a pagination bug is invisible on any fixture smaller than a page, which is every fixture anyone writes by hand.

Then remove the capability that made it dangerous: the reconciler reports orphaned library entries and never purges them. The caller is handed the identifiers and deletes them deliberately if it wants to, following a pattern already used elsewhere in the codebase. A reconciler is a diff tool; the moment it is also a delete tool, every bug in its comparison is a destructive bug, and the comparison depends on a listing, a tag, and an alias map, three things this piece of work had already caught being wrong.

The related decision I argued against was bulk-ingesting the entire vendor tree. Two reasons, one of them arithmetic: the cost that scales badly is per-page document extraction, not embedding, and the tree is orders of magnitude larger than the library. The other is retrieval quality. A much bigger corpus of highly similar near-miss documents degrades generated answers even when it never contains the answer, so the ingest would have to be justified per vendor rather than assumed. I sized a first tranche and left the rest out.