Case study · AI systems engineering

Two vendor integrations, and the least privilege neither token could express

Integration and deployment work adding third-party security platforms as tool servers on an internal AI assistant.

Python
Go
MCP
OAuth 2.0
Docker
Kubernetes
AWS Secrets Manager

Two integrations that look like the same task and share almost no engineering. One vendor would only issue a token representing the whole organisation; the other would only issue tokens representing a person, with permissions so coarse that least privilege could not be written into the token at all. In both cases the interesting work was deciding where the access boundary went once it became clear the credential would not carry it.

0 writesoperations either proxy exposes. Least privilege is enforced in the proxy, because neither vendor's token could express it

The problem

The assistant already had tools for the company’s own documents, chat, mail and calendars. The next requests were for third-party platforms the company actually runs on: a compliance-automation platform holding controls, frameworks and audit findings, and a device-management platform holding the fleet inventory. Both were asked for as “connect this to the assistant”, which is not what either turned out to be.

The vendor decides the credential model

One platform issues organisation-level machine credentials and treats per-user consent as something you build for your own customers, not for yourself. The other issues per-user tokens only. Neither offers the alternative, so the architecture is chosen for you.

Read-only is a claim until it is a construction

Both integrations were meant to be read-only. A tool list that happens to contain no write tools is a claim about the current code; a client that can only issue read requests is a property of it.

This is security data, in a prompt

Fleet inventory carries device serials, hardware identifiers, user principal names and network details. Compliance data maps the organisation’s own control gaps. All of it would land in model context, conversation history and logs.

Read-only is not the same as harmless

Read access across a management plane and a compliance posture is a reconnaissance surface. Acceptable for an internal audit aid; not something to be casual about who can reach.

The third and fourth constraints are the ones that made “who is allowed to see these tools” a security control rather than a configuration detail.

What I built

Two tool servers with almost nothing in common but their shape.

Already existed

A tool-server pattern with several existing servers to mirror, including shared helpers for auth, log redaction and result limits
Per-server OAuth templates in the backend, and header-based token forwarding to tool servers
A generic OAuth flow supporting authorisation code with PKCE and refresh
Secret retrieval from a managed secret store, already used by one other server
Role-based access control over which users may use which tools

What I added

A read-only proxy to the compliance platform, self-authenticating with an organisation credential, after the intended design turned out to be impossible
A per-server scope set for the device integration, so its consent could not leak into every other integration
A two-layer tool gate: a static allowlist plus a write-verb denylist checked against both names and descriptions, enforced before any network call
The deployment path: container, build pipeline, environment-scoped secrets, cluster manifests and the IAM to read them

Read-only by construction, not by inventory.
The compliance client issues read requests only. There is no code path through it that performs a write, so the guarantee does not depend on anyone maintaining a list correctly. The allowlist and denylist sit on top of that as defence in depth, not as the guarantee.
An allowlist that fails closed, checked before the network.
An unknown tool name is refused, not passed through. The denylist matches write verbs against the tool’s description as well as its name, and wins over the allowlist, so a benign name with a remediating description is still blocked.
Per-server consent, so one integration cannot widen another.
The device integration requests its own explicit scope set rather than inheriting the shared default, which is what stopped its permissions being granted to every user who had ever connected the provider for any reason.
Field filtering at the tool boundary,
because the honest answer to “should device serials and hardware identifiers flow into a language model’s context” is usually no, and the tool layer is the last place you can decide that before they do.

Architecture

Two integrations, two credential models, and where least privilege ends up in each

The row that took the longest to accept is the third one. For the device platform, the read permissions the tools need have no resource-scoped variant available to a delegated application. The only form on offer grants read across the whole tenant. There is no scope string that expresses “this user’s own team’s devices”, so no amount of care in the OAuth configuration produces least privilege.

That has a concrete consequence: the narrowing has to happen in the vendor’s own role model instead, which means the correct recommendation for anyone connecting the integration is a narrow read-only operator role limited by scope tags, not a broad directory-wide reader role, which is the one people reach for because it sounds passive. The OAuth scope is a ceiling; the vendor role is the actual boundary. Documenting which is which was part of the deliverable.

For the compliance platform the same conclusion arrives from the opposite direction. An organisation credential cannot express per-user anything, so every permitted user sees identical org-wide data. The only remaining granularity is who may see the tools at all, which is exactly why that was configured deliberately and restricted to a small group, rather than left at the default.

Four problems worth describing

The design I got approved was impossible, and the discovery took one call

Symptom

The plan was clean and reviewed: a thin proxy that authenticates to the vendor’s own hosted tool endpoint with an organisation credential and forwards filtered read-only calls. Wiring it up produced an error at the authorisation step, and then a second error on refresh.

Diagnosis

Two independent problems, and the second one killed the design.

The backend’s generic OAuth support is hard-wired to authorisation code with PKCE and a refresh token. A machine-to-machine credential has neither an interactive authorisation step nor a refresh token, so it failed at the first and then failed again when the platform tried to refresh something that does not exist. That was fixable in principle.

The decisive finding was one step earlier, and I only got it by testing the credential directly against the vendor rather than through the platform. The credential is valid: it mints a token successfully. The vendor’s hosted tool endpoint then rejects that token because it carries no user identity. The identical token works against the vendor’s ordinary REST API. The endpoint I had designed the proxy around structurally cannot accept the only credential type the vendor would give us for our own organisation.

Decision

Keep every goal, change the transport. The proxy self-authenticates with the organisation credential and calls the REST API instead, with a client that issues read requests only, which turns “read-only” from a filtering promise into a property of the client. Twelve allowlisted tools map onto read endpoints. No backend changes were needed, no per-user OAuth was needed, and the vendor’s one-token-per-application rule is respected by a single process-wide token that re-mints shortly before expiry.

The token manager and the guardrail layer survived the rewrite unchanged. What was thrown away was the assumption that the vendor’s own tool endpoint was the right thing to talk to.

Authenticating as the organisation

Proxy → vendor’s hosted tool endpoint

platform oauth      expects authorisation code + PKCE + refresh
machine credential  has neither  → error at authorise, 400 on refresh
token mint          ok  ← the credential itself is valid
hosted tool endpoint  401  “token missing user identity”
same token → REST API   200
verdict             the designed transport cannot accept this credential type

Proxy → vendor’s REST API, read requests only

proxy self-authenticates, caches one token process-wide,
remints shortly before the 1h expiry, no refresh token
client                read requests only → read-only by construction
tools                 12, allowlisted, mapped to read endpoints
backend changes       none
guardrail layer       unchanged from the original plan

The lesson is about the order I tested things in. I validated the credential through our own platform first, which produced a confusing platform-shaped error and sent me looking at our OAuth code. One direct call to the vendor, made first, would have told me the design was dead before it was written. Test the assumption furthest outside your control before the code that depends on it.

A scope set for one integration was about to be granted to everyone

Symptom

None yet. This was caught while reading the configuration rather than from a failure. The device integration needed two read permissions, and the natural place to add them appeared to be the shared default scope string that the provider’s integrations use.

Diagnosis

That string is a fallback. Each tool server carries its own OAuth template, and a template with no scopes of its own inherits the shared default. Because most of the existing servers had been left to inherit, they all requested the same broad bundle, which reads exactly like a deliberately shared template and is not one.

So appending to the fallback would have meant every user who connected the provider for any reason, whether mail, files or chat, consented to device-inventory read as a side effect. The blast radius of a two-permission change was every user of every integration on that provider, and the change would have looked entirely local.

Decision

Give the device server its own explicit scope set, including the base identity scopes, so it never falls back, and leave the shared default untouched so no other integration moves. Per-server isolation, with each user still authenticating individually, so a user’s token carries only the permissions of the server they connected.

Worth being precise about what this does not achieve: the scope set is bound to the server, not chosen per user at request time. Everyone who connects that server gets those scopes. It contains the blast radius across integrations; it does not create per-user granularity within one.

Inheritance that produces uniform behaviour is indistinguishable from a shared definition until you change one thing. This is the same failure I hit in another project from the other direction, where a control was wired to one path and assumed to run on all of them. Both are questions about what is actually connected to what, and neither is answerable by reading the file you are editing.

One status code, three unrelated causes, and a colleague’s access expiring at 5pm

Symptom

The device API returned an authorisation failure. It kept returning it after a colleague with the necessary administrative access granted my test account a role, and their ability to grant that role was time-boxed to the end of the day.

Diagnosis

The failure is genuinely ambiguous: that platform returns the same status whether the token lacks the required scopes, the account lacks the required role, or the account has no licence for the product at all. The status code cannot distinguish them, and each has a different owner. The first is mine, the other two are not.

Earlier in the day I had run a short diagnostic that decoded the token’s own claims and proved it carried both required scopes and the correct audience. That put the failure on the authorisation side rather than the token side. But by this point I had rebuilt on a freshly-switched branch and re-consented, and a different template could in principle have issued a token without those scopes. The evidence I was relying on was stale, and the signature of both causes is identical.

Decision

Re-run the claims diagnostic rather than send the colleague looking. It cost about a minute and converted “probably a role-propagation delay” into a fact, because the alternative was spending someone else’s expiring access on a guess, and if the re-consent had dropped the scopes then the fix was mine and asking them at all would have been wrong.

The related decision was made before any of this: because my own account had no licence for the product, I defined in advance what success looked like for me. A clean authorisation refusal from the vendor was the passing result, since it proves the server connected, the token forwarded and the vendor was reached. Full end-to-end verification with real device data was always going to be the licensed colleague’s test, not mine.

Deciding what “working” looks like for a tester who cannot possibly succeed is worth doing before you start. Without it, every run looks like a failure and there is no way to tell progress from a wall.

The bug that only appears when you scale up, and two deployment conventions I broke

Symptom

None in production, because I found it by reading the vendor’s constraint against our own deployment model rather than by triggering it.

Diagnosis

The vendor permits one active token per application. Minting a new one revokes the previous one. The proxy caches its token process-wide, which is correct for a single instance and hostile the moment there are two: each replica mints its own token, each mint revokes the other’s, and both replicas alternate between working and returning authorisation failures indefinitely. It would present as intermittent flakiness with no failing component.

Reviewing my own deployment work turned up two more things, both of which I had got wrong by not looking hard enough at the sibling servers. I had used a floating image tag with an always-pull policy; every other server in the cluster pins a full image digest, bumped by hand per deploy. That is not a style difference. The pinned digest is how a build is promoted from test to production, and a floating tag means you cannot tell which build is running or reproduce it. And the credential was reaching the container by environment variable, when the established pattern is retrieval from a managed secret store at startup.

Decision

There is no correct in-process fix for the token constraint, since a shared upstream session would break token isolation and a distributed lock is absurd for this, so it is a documented deployment constraint: one replica, recorded where someone scaling the service will read it. Naming a limitation is the fix when the limitation is the vendor’s.

The rest was conformance: pin the digest like every sibling, move the credential into the managed secret store, and, at the platform lead’s request, make the secret name carry the environment with the IAM policy scoped by a matching prefix so a new environment needs no policy change. That last one is theirs, not mine, and it is better than what I had proposed.

Honest framing of the effort here: I estimated this as a proxy plus a registration call. It became a rewritten transport, an environment-scoped secret, IAM, cluster manifests and a build pipeline. The integration was the small half.

Where it landed

Verified

The compliance proxy running in both the test and production environments, self-authenticating with an organisation credential held in a managed secret store
Read-only by construction: the client issues read requests only, with an allowlist and a write-verb denylist over the top as defence in depth rather than as the guarantee
The allowlist failing closed on unknown tool names, and the denylist matched against tool descriptions as well as names, so a benign name with a write-shaped description is refused
Every mapped tool returning real data, confirmed by a live end-to-end run against the vendor API
The device integration requesting its own explicit scope set, so no other integration on that provider had its consent widened
End-to-end device data verified by a licensed colleague, with my own unlicensed account producing the authorisation refusal defined in advance as the passing plumbing result
A measured reduction in tokens on one verbose compliance query after trimming the returned fields, from roughly 40k to 11k

What this doesn't solve

An organisation-level credential gives every permitted user identical org-wide access. The only granularity available is who may see the tools, which makes that setting a security control and not a convenience.
Read-only across a management plane and a compliance posture is still a reconnaissance surface. It is appropriate for an internal audit aid and would not be appropriate to expose loosely.
The tenant-wide read scopes have no resource-scoped delegated variant, so least privilege is enforceable only in the vendor’s own role model. That is a recommendation to whoever configures it, not something the integration can guarantee.
One active token per application is a vendor constraint, so single-replica operation is a documented ceiling rather than something the code solves.
Validation of the rewritten proxy was byte-compilation, an offline logic harness over the tool-to-endpoint mappings, and a live API run, not the unit test suite, which I could not execute in that environment. I would rather say that than imply a green test run.

What I'd do differently

Test the thing furthest outside my control first. One direct call to the vendor, before any proxy code existed, would have shown that the endpoint I had designed around could not accept the only credential type available to us. Instead I validated through our own platform, got a platform-shaped error, and spent time in the wrong repository. The cheapest possible experiment was also the one that invalidated the design.

Read the siblings for conventions, not just for structure. I copied the shape of the existing tool servers accurately and missed two conventions that were not visible in any single file: pinned image digests as the promotion mechanism, and secrets from a managed store rather than the environment. Structure is easy to see; operational convention has to be looked for.

Ask what a permission grants to people who are not the requester. The scope change I nearly made was two lines and would have widened consent for every user of every integration on that provider. I now treat “who else inherits this” as a required question whenever a default or fallback is being edited, because the diff never shows it.