Case study · AI systems engineering

Measuring an agentic SOC triage pipeline before improving it

Threat-intelligence enrichment, model grounding and evaluation work on an agentic security-operations triage platform.

Go
MongoDB
Valkey
AWS Bedrock
MITRE ATT&CK

An agent that investigates security alerts was answering inconclusive on most of the alerts I fed it. That looked like the problem to fix. It was not. The test alerts carried nothing to corroborate, and inconclusive was the honest answer. Which is the whole difficulty with this kind of system: without a labelled baseline you cannot tell a correct refusal from a bad one, and every improvement you then make is unmeasurable.

$0committed to a paid feed, against four candidates at $5k–$12k a year each. Every source optional, the pipeline correct with none

The problem

A platform that triages security alerts for multiple client organisations already had an investigation agent: it read an alert, called tools, reasoned, and produced a verdict with a confidence score. What it mostly lacked was anything to reason with. One source of address intelligence, cached in the memory of a single process, and no reputation, registration or malware-family context at all.

Adding sources is the easy part of that sentence. Four things made it not easy.

Caching threat intelligence is a multi-tenancy question

Every client benefits from a shared cache, because what is known about a public indicator is a global fact. But a cache shared across tenants is exactly the shape of an accidental cross-tenant data leak, so the design has to make the leak impossible rather than unlikely.

Enrichment sources return personal data

Registration records, geolocation and campaign context carry names, contact details and network specifics. All of it was headed for a cache, a model prompt and an investigation transcript.

A model will attribute confidently from memory

Asked to map a vulnerability to attacker techniques, a model returns real-looking technique identifiers whether or not it has read a definition. The output is plausible, well-formatted, and wrong in a way nothing downstream can detect.

You cannot improve what you have not measured

Verdict quality on ambiguous alerts is a judgement, so "did that change help" needs human labels. Without them, a shift in the verdict distribution is not evidence of anything.

The last one is why the most useful thing I did on this project produced no user-visible feature at all.

What I built

An enrichment pipeline, a grounding layer, and the evaluation harness that made the other two assessable.

Already existed

An investigation agent with a tool-calling loop, verdicts and confidence scores
Automatic tool registration, so a capability becomes an agent tool without touching the loop
One address-intelligence source, cached in process memory
Vulnerability enrichment from public catalogues, and a model layer to call
A configuration file defining priorities, verdicts, stop rules and cache lifetimes, written first, as the contract

What I added

A cache-aside indicator enrichment service over a shared cache, deliberately org-agnostic, fanning out to several optional sources
A model reconciliation step that resolves disagreement between sources into one interpretation with its reasoning recorded
Grounding and validation of attack-technique mapping against a synced authoritative catalogue, replacing the model’s recall
A labelled ground-truth baseline and the before-and-after comparison protocol built on it
Enforcement of the stop rules the configuration file had specified and the code had never read

A cross-tenant cache that cannot be handed tenant data.
The enrichment service receives an indicator type and value and nothing else. No organisation, no user, no alert. It therefore cannot key an entry to a tenant, which is a stronger guarantee than remembering not to.
Personal data distilled out before the cache, not after.
Sources that return personal detail are reduced to fixed non-personal fact structures before anything is written or shown to a model, with a test per source asserting nothing leaks.
Technique attribution grounded in a real catalogue and then validated against it.
The model selects from the actual active technique list rather than from memory, and every identifier it returns is looked up afterwards, corrected if known, demoted if not.
A ground truth, built before the changes it was meant to judge.
A sampled, human-labelled set of past investigations, with the sampling reason recorded per row and a documented protocol for re-running the same alerts afterwards and comparing by investigation identity.

Architecture

Indicator enrichment: a cross-tenant cache that cannot be given tenant data

Three decisions in that path are worth stating as decisions rather than as steps.

The shared cache is a design position, not a shortcut. Intelligence about a public indicator is not any client’s data, so caching it per tenant would multiply cost and slow every investigation for no privacy gain. The privacy work is elsewhere: the service is given no tenant context, so there is no cross-tenant linkage available to store, and the distillation step guarantees the values hold no personal data either. Both halves are needed. Either alone is a liability.

Cache values are deliberately not encrypted in the application. They contain public intelligence and no secret material; at-rest encryption is handled with a per-environment managed key at the infrastructure layer, which is transparent here. Encrypting each value in code would add cost and latency to every cache hit, which is to say to the fast path the cache exists to create, in exchange for nothing.

Everything degrades rather than fails. A source with no credential is not offered. A source that errors is skipped. A cache error fetches live instead of failing the investigation. The reconciliation call is skipped when there is nothing to reconcile, and returns unknown rather than a guess when its own output will not parse. An investigation is the last thing that should fail because a third party is having a bad day.

Four problems worth describing

The failure that was the correct answer

Symptom

Running test alerts through the agent, most investigations came back inconclusive with middling confidence: 62% of them, and on one batch 8 of 10. On the face of it, an agent that mostly cannot decide is not much of an agent.

Diagnosis

The alerts carried no indicators, no addresses, no domains, no hashes, so there was nothing for the agent to corroborate against. Its own instructions explicitly license inconclusive when there is nothing to check, and it was following them. Checking the batch of ten individually, 8 of the 8 inconclusive verdicts were the correct call. On one, the agent declined to call a scheduled-task technique benign because the technique is used just as often by administrators as by attackers, and it had no baseline for the account involved and no directory data to establish whether that person was IT staff. It settled at 42% confidence and inconclusive, which is the correct output there. Higher confidence would have meant guessing: with the directory context it lacked, the same alert is a 90%-confidence false positive.

So the 62% was not a quality signal at all. It was a property of my test data. And I had been about to start tuning against it.

Decision

Stop, and build a ground truth before changing anything. I exported all 406 investigations on record as a raw baseline, then sampled the ~150 rows where a human label carries the most information: every true positive, every false positive, and the 40 most confident verdicts, tagging each row with the reason it was sampled so the sample’s own bias is visible. That went into a labelling worksheet with a written guide, so each row gets a human disposition, a judgement of whether the agent was right, and a note. Alongside it: the baseline metrics as they stood, the 62% recorded explicitly as the before number, and the protocol for re-running the same alerts afterwards and comparing them by investigation identity rather than in aggregate.

None of that shipped anything. It is what makes the next three exhibits assessable instead of merely plausible.

The trap here is specific to agents that are allowed to say “I don’t know”. Their honest refusals and their failures produce the same-looking metric, and the instinct is to tune the metric down. I would have improved the number and made the system worse.

A model attributing attack techniques from memory

Symptom

Mapping a well-known critical vulnerability to attacker techniques produced identifiers that all existed and read convincingly. One was wrong. The step describing a malicious Java class being fetched and executed was labelled as a command-and-scripting-interpreter technique, which is not what that is. The behaviour was already covered by the exploitation technique in the previous step.

Diagnosis

The model was pattern-matching the phrase “code execution” to a technique it knew by identifier, rather than reasoning from that technique’s actual definition. It could not do otherwise: the authoritative catalogue was being synced into the database on a schedule and was then never passed to the model. The mapping call sent vulnerability data and nothing else, so every attribution came out of training recall, which drifts, is unversioned, and fails silently because a wrong-but-real identifier passes every check a downstream system can perform.

Decision

Two layers, because grounding alone is a request and validation alone is a filter.

Grounding: build a reference block once per process from the active catalogue (identifier, canonical name, tactics, one line each, deprecated techniques excluded at the source rather than filtered afterwards) and send it as a static system block so the model selects from real techniques. It benefits from prompt caching precisely because it never varies.

Validation: look up every identifier the model returns. Known ones are kept with their name, tactics and description overwritten from the catalogue, which is authoritative. Unknown ones are dropped from the trusted index but retained in the stored metadata flagged as unvalidated, because thrown away silently is worse than kept and marked.

Both layers fail open. An empty or unsynced catalogue switches them off and passes the model’s output through unchanged, because the alternative is that a sync failure quietly reduces every mapping to nothing.

Technique attribution for one critical vulnerability

Attribution from training recall

sent to the model    vulnerability data only
catalogue in the db  synced, and never read
identifiers returned all real ← which is why nothing caught it
one step             “java class fetched and executed”
                     → scripting-interpreter technique   wrong
detectable by        nothing downstream. it validates as real.

Grounded, then validated against the catalogue

system block         active catalogue: id, name, tactics
                     deprecated excluded at the source
                     static → benefits from prompt caching
every id returned    looked up: known → canonical values applied
                     unknown → dropped from index, kept + flagged
live run             all returned ids resolved in the catalogue
stability            confirmed core steps identical across runs;
                     only unconfirmed follow-ons varied

One bug I caught in my own validator: a transient database error while looking up an identifier would have silently dropped a possibly-valid technique, making an infrastructure blip indistinguishable from a hallucination. The validator now returns the identifier and a keep decision, so a genuine miss drops and a lookup failure keeps it uncorrected. “I could not check” is not the same answer as “I checked and it is not real”.

Three numbers for one limit, and none of them agreed

Symptom

An investigation reported nine steps against a documented limit of six, having been cut off by a hard-coded eight.

Diagnosis

All three numbers were real and in different places. The configuration file, written first and deliberately as the contract both the prompt builder and the enforcement code were meant to read, specified a maximum iteration count, a maximum tool-call count and a wall-clock limit. The loader for that file had been deferred to a later phase. So the code enforced none of it, except one stale constant predating the contract entirely. And the counter it guarded incremented per tool call while the constant was written as a per-turn limit, so any turn calling several tools overshot the cap and reported a number that matched neither document.

Decision

Make the code match the contract it already had, rather than adjust the contract to the code. The single ambiguous constant became two limits with distinct jobs: a hard tool-call ceiling matching the specified figure, and a soft enrichment budget which, once spent, injects a one-time nudge to conclude, and only when the agent is not mid-pivot. A hard cut in the middle of a productive line of enquiry throws away the investigation’s most valuable steps; a nudge lets it finish the thought. The terminal fallback that forces a conclusion stayed as the last resort it always was.

I also went back and marked the documentation contradiction as resolved instead of leaving a note saying two figures disagreed. A known-and-recorded inconsistency is still an inconsistency.

Writing the configuration first as “the source of truth” was the right call and it created its own hazard: for several phases there was an authoritative-looking document that nothing read. A contract with no reader is not a contract, and it is more misleading than no document at all, because the next person believes it.

A better design I argued myself out of building

Symptom

A real modelling problem. A vulnerability description often establishes the class of attacker behaviour without establishing the specific method. Forcing the model to pick one specific sub-technique manufactures precision the source does not support; forcing it up to the parent throws away detail that was genuinely established.

Diagnosis

The attractive answer is to represent the uncertainty honestly: one primary technique per step, plus a bounded set of candidate sub-techniques when the method is left open. It is faithful to the evidence, and it is more defensible under audit rather than less. “The description establishes command execution but not which interpreter, so these are the candidates” is easier to justify than a single guess.

Then the costs, which I made myself write down. It is a determinism regression: two runs would return slightly different candidate sets, so I would be trading “sometimes picks the wrong one” for “picks a different set”. It needs a hard cap or it decays into hedging, and a chain where every step lists six possibilities is noise. And it is a schema, storage and validation change rather than a prompt change: one broad parent technique has more than a dozen sub-techniques, so a short chain could balloon from three attributions to twenty.

Decision

Don’t build it. The question I had actually been asking was never one technique or several. It was does the model choose well when a parent has a dozen children. If it does, there is no problem to solve. If it does not, the cheapest available fix is one line of prompt instructing it to emit each plausible sub-technique as its own step rather than guessing between them, which gets most of the benefit for none of the schema change.

Two related calls did get made. Sub-techniques stay in the catalogue: removing them is a lossy one-way door and the grounding and validation layers already prevent invented identifiers, so over-specificity is a prompt concern and not a data-model one. And the sync job was filtering sub-techniques out while the prompt was written expecting them to exist, a disagreement between code and prompt that would have quietly capped attribution detail with no error anywhere.

Included because it is the shape of decision I most often see made badly: a design that is genuinely more correct, cheap to describe, and not worth its cost. Writing the determinism regression down explicitly is what settled it. Until then it read as a pure improvement.

What it costs, and what it was allowed to cost

Two of the design decisions on this pipeline were made on cost rather than on correctness, and both of them are more interesting than a saving would have been, because in each case the cost was structural: a question of how many times you pay, not of how big the thing is.

What a large prompt actually costs, and why one optimisation got two opposite verdicts

The reference block is large and that is not the same as expensive. Grounding the attack-technique mapping means injecting the whole active catalogue, 709 identifiers and names, into the prompt, at roughly ten to thirteen thousand tokens per call. My first reaction was that this was obviously too expensive to ship, and it was the wrong reaction. Each vulnerability is mapped once, ever: the result is written to permanent storage and never re-derived, so the block is paid once per vulnerability rather than once per investigation. Prompt caching then does the rest, because a static catalogue block is exactly the byte-stable prefix that caching exists for.

I did consider the cheaper prompt, keyword-matching the description first and injecting only the twenty or so plausibly relevant techniques, and rejected it. Not because it wouldn’t work, but because the permanent-storage property had already bounded the total spend, so the remaining saving bought complexity in the one place where a wrong narrowing is invisible: a technique dropped before the model ever sees it cannot be recovered by the validator downstream.

The same optimisation, opposite verdicts. Prompt caching was adopted here and dropped on the bid-answering run, and the two decisions are consistent. This call is input-dominated with a large byte-stable block. That one is output-dominated, with five to six of its six-and-a-half minutes spent generating tokens, and only about four per cent of its prompt byte-stable, which caps caching at roughly one per cent of wall clock. The technique is not good or bad; the workload’s shape decides, and the shape is measurable before anything is built.

The commercial constraint was spend not yet committed. The enrichment sources are third-party feeds, and I priced the four commercial candidates before writing anything against them: an address-reputation feed and a scan-noise classifier at around ten thousand a year each, an address-context service at a thousand a month, and a malware-sample service at around five thousand for its commercial tier. The best-known multi-scanner’s free tier is non-commercial only, so using it properly is an enterprise licence as well. Roughly $5,000 to $12,000 a year per source, and a full set is a five-figure annual commitment for data the pipeline had not yet proven it needed.

The engineering consequence, and the reason this belongs in a case study rather than a purchase order: every source is optional, and the pipeline is correct with none of them. Sources are configured independently, skipped cleanly when absent or failing, and the pipeline activates only when the cache is reachable and at least one source exists. That is what let the work proceed against free and freemium tiers, one of which allows a thousand lookups a day, with no spend committed and no code written against a contract nobody had signed yet. Two bounds keep usage inside those tiers by design rather than by luck: a hard cap on enrichment calls per investigation with a softer nudge to conclude well before it, and a shared cache with a day-long expiry, so a repeated indicator is a cache hit rather than a second paid lookup.

709catalogue entries injected per mapping call, ≈10–13k tokens, paid once per vulnerability rather than once per investigation

≈1%of wall clock the same optimisation was worth on an output-bound workload, which is why it shipped here and not there

$5k–$12kper year, per source, across the four commercially licensed candidates. List prices at the time, none of them committed to

$0committed to a paid feed: every source optional, the pipeline correct with none configured

What I am not claiming: any of this converted into an annual figure. The token cost per mapping call is measured and the licence prices are list prices, but the spend depends on how many vulnerabilities get mapped and how many alerts arrive, and I have neither number. The saving I can defend is structural: a cost that recurs per vulnerability instead of per investigation, and a pipeline that does not require a purchase to be correct.

Nor is there an analyst-time figure here, which is the number a reader most wants. Measuring it honestly needs a before: how long a person took to reach the same verdict on the same alerts. That baseline was never captured. What I built instead was the labelled ground-truth set and the before-and-after protocol that would let someone capture it, which is the prerequisite for the claim rather than the claim.

Where it landed

Verified

A cache-aside enrichment pipeline over a shared cache, feature-flagged and off by default, activating only when the cache is reachable and at least one source is configured
Multiple optional sources across address, domain and file-hash indicators, each fetched on a miss, each skipped cleanly when unconfigured or failing
Cache keys carrying no organisation identifier, enforced by the service never receiving one, so it is verified by construction rather than by review
Personal data distilled to fixed non-personal structures before any cache write or model prompt, with a per-source test asserting no leak
Private, loopback and link-local addresses excluded from lookup and caching, which doubles as an outbound request-forgery guard
Model reconciliation of disagreeing sources into a single interpretation carrying its confidence, the conflicts resolved and its reasoning
Exposure to the agent as a capability, with no change to the investigation loop, and only for indicator types a configured source can answer
Attack-technique attribution grounded in the synced active catalogue and validated against it, with a live run in which every returned identifier resolved
Stop rules enforced to match the figures the configuration file had always specified
A sampled, human-labelled ground-truth set: 406 investigations exported as the raw baseline, ~150 sampled for labelling, the 62% inconclusive rate recorded as the before number, with its sampling bias recorded and a written before-and-after comparison protocol
A full end-to-end run captured as evidence rather than asserted: 181 seconds, a 2,388-line transcript of every model input and output, cache probes and per-call, per-stage and total token counts, kept so the result is auditable rather than remembered
Unit tests throughout, integration tests against a real cache instance proving the miss contract and real expiry, and integration coverage of the grounded mapping against a real catalogue and a real model call

What this doesn't solve

Verdict quality on genuinely ambiguous alerts is a human judgement. The evaluation harness makes changes comparable; it does not make correctness automatic.
An agent permitted to answer “inconclusive” will produce refusals that look identical to failures in any aggregate metric. That is a property of allowing honesty, and the mitigation is labelled data rather than a better metric.
Grounding constrains the model to real techniques; it does not guarantee the right one. Confirmed steps were stable across runs, unconfirmed follow-on steps varied, and that variation is inherent to reasoning at a non-zero temperature.
The reference block costs real tokens on every mapping call. It is affordable because each vulnerability is mapped once and the result is stored permanently, which is a property of the caching rather than of the prompt.
Four of the candidate sources are commercially licensed, at list prices between roughly $5,000 and $12,000 a year each, so which sources exist in a given environment is a budget decision that changes the pipeline’s output. The free tiers the pipeline runs on today are rate-limited, which is a capacity ceiling rather than a correctness one.
The verdict distribution I measured came from largely synthetic alerts. It is a baseline for comparison, not a statement about performance on real traffic.
That end-to-end run surfaced a defect I have not fixed: only three of the four priority bands can ever fire, because an alert the agent calls a true positive at low severity maps to the lowest band rather than the second. The mapping is a table, so it is a small change, but it means the priority output has been quietly coarser than its own specification, and every band distribution measured before it is corrected is measuring three buckets, not four.

What I'd do differently

Build the ground truth first, not at the phase boundary. I stopped to build it because I noticed I was about to tune against a metric I could not interpret. That was luck arriving as instinct. The labelling harness cost a day and every judgement after it got cheaper, so it should have been the first task of the project rather than the last task before the third phase.

Never write a contract before its reader. Putting the rules in configuration first was right; deferring the loader created a document that looked authoritative, was cited in reviews, and was enforced nowhere. If the loader has to wait, the contract file should say in its own text that nothing reads it yet.

Ask what a model is reasoning from before assessing what it reasons to. The attribution output was well-formed, plausible and internally consistent, and the actual defect was one layer back: the authoritative data existed, was synced, and was never in the prompt. I now check what is in the context window before I evaluate the output.