The Failure Mode Is Not Bad Prompting

Your enterprise Copilot isn’t hallucinating. It’s following orders — it just has too many bosses and no chain of command.

Engineering teams treat agent instructions like folklore: passed down in a Slack thread, copied into a new repo, half-remembered by the time anyone needs them. They should be treating them like configuration. A modern agent harness stacks organisational constraints, repository rules, workspace skills, custom agent personas, and local files into a single context window before the model ever sees a user prompt. Each layer gets written by a different person, at a different time, for a different reason. None of them can see the others.

When an agent acts unpredictably, the reflex is to blame the prompt — rewrite it, add capital letters, repeat “IMPORTANT” three times. Wrong diagnosis. The real failure mode is two valid instructions disagreeing with no recorded winner. Global policy says mask PII; a local file says dump the payload for debugging. Both are correct in their own scope. The model has no scope awareness, though, so it picks one — and not necessarily the same one twice.

Its arbiter of last resort turns out to be proximity, not business logic. Liu et al.’s 2023 paper “Lost in the Middle: How Language Models Use Long Contexts” found that retrieval accuracy degrades measurably when relevant information sits in the middle of a long context, with performance strongest at the beginning and end. That study measured retrieval in long-document question-answering, not instruction-following in agent harnesses — the extrapolation isn’t proven, but the mechanism generalises cleanly enough. Map the finding onto an instruction stack: org policy loads first, local .agents.md loads last, and the user prompt gets appended after both. In an instruction stack, that finding is a warning sign rather than proof: placement can bias what the model uses, so file load order can become an accidental tie-breaker when the harness has no explicit precedence rule. That’s not a policy decision. It’s an accident of file load order. And it fails silently: the agent never says “I resolved a conflict, here’s why.” It just does one thing today and something else next week, and everyone assumes the prompt changed when the architecture never did.

The SME Clash: Enterprise Guidelines Versus Repository Files

We ran into this on a multi-agent deployment for an SME client under a strict compliance regime. The organisational layer carried a system instruction to mask all personally identifiable information in any logs the agents wrote — standard stuff, non-negotiable, the kind of line a compliance officer signs off once and never revisits.

One repository in that client’s stack, an analytics microservice, had its own .agents.md. A developer had added it months earlier while chasing an integration bug, instructing the crew to dump raw API payloads to logs so the team could trace field mismatches. Reasonable, in isolation. Nobody deleted it once the bug was fixed, because nobody thought of it as a standing instruction. It was just a note.

Here’s the part that should worry every engineering lead running Copilot or a similar harness at scale: the agent didn’t consistently violate one rule or the other. It flipped. Same repo, same two instructions loaded, different outcome depending on how that day’s prompt happened to be phrased — sometimes masking correctly, sometimes not. The only variable that changed between a masked run and a leaking run was token positioning in the assembled context — nothing about the underlying rules had shifted. The result was intermittent PII leakage into development logs, a genuinely bad outcome for a compliance-sensitive client, and no systemic way to inspect which instruction had “won” on any given run. There was no log line reading “local override applied.” There was just an output, correct or not, and no way to know which one you’d get until it landed.

Adding more adjectives to the global prompt didn’t fix it. We tried. Stronger language changed the outcome on some runs but never stopped the leak from recurring — because the underlying problem was never the wording. It was that two instructions occupied the same authority tier with no arbitration mechanism between them.

The wGrow Harness: Local Skills Rewriting Stable APIs

We hit our own version of this building the internal wGrow harness. Our baseline project instruction was explicit: never modify core API routing logic. That file predates most of the skills we’ve since layered on top of it.

A developer later added a local skill to scaffold CRUD endpoints quickly — the kind of productivity shortcut every team eventually writes. Its instructions told the agent to update the main router automatically whenever a new endpoint was scaffolded, which is genuinely useful when you’re cranking out boilerplate. Nobody flagged the collision, because on paper the two instructions looked like they were about different things: one a constraint on architecture, the other a workflow convenience.

They weren’t different things. They were competing claims on the same file, and the local skill — loaded closer to the point of generation — won. The agent rewrote stable routing logic to wire in the new endpoints and broke the build. Nothing exotic happened. No jailbreak, no adversarial prompt. Two instructions we’d each individually approved simply outranked each other in an order nobody had chosen.

That incident is the whole argument in miniature: system prompts aren’t documentation, they’re competing executable directives, and a four-layer instruction stack without an explicit precedence rule will eventually let the wrong layer win. It’s not a question of if. It’s a question of which build it breaks first.

WaterDoctor Guardrails and Deterministic Pre-Flight Routing

For WaterDoctor, the deep-tech investee we work with, this couldn’t be a “we’ll tune the prompt” problem. Database migrations there run against production schemas that control monitoring hardware calibration data. A deny-by-default rule against destructive schema operations was non-negotiable — we weren’t willing to bet that outcome on which instruction the model happened to attend to on a given run.

So we didn’t. We built an orchestration layer that sits outside the LLM entirely and rejects unapproved actions before they execute, regardless of what any instruction layer told the agent to do. It works in two parts: a deterministic SQL parser converts every generated SQL payload into an abstract syntax tree and rejects destructive statement types — DROP, ALTER, TRUNCATE — outside a small allow-listed migration path, while a pre-flight LLM pass acts as a semantic router, classifying intent before anything runs. If the AST parser finds restricted syntax outside the approved migration path, the orchestrator rejects the request immediately — that check alone is sufficient to block execution, no second opinion required. The semantic router runs in parallel to catch destructive intent earlier, before SQL generation even completes, but it has no authority to overrule the parser in either direction: it can’t wave through what the parser blocks, and a router miss doesn’t let anything past the parser either. No partial execution, no negotiation, no “are you sure.”

The point isn’t that the LLM got smarter about not dropping tables. It’s that we stopped asking it to be the enforcement layer for the one class of action where being wrong is expensive to reverse. There’s a real trade-off here worth naming: the AST parser only catches what its keyword and allow-list rules anticipate, so extending it to a new class of destructive action means updating the parser, not just the prompt. That’s a maintenance cost. But it’s a cost we can audit — unlike a prompt’s phrasing, which you’re mostly just hoping about. Alignment language and capitalised warnings are not a substitute for a deterministic gate that sits above the model’s reasoning.

Deploying the Precedence Matrix

Firewalls don’t ask a packet to explain itself. They evaluate it against rules in a strict, ordered sequence and accept or reject deterministically. Agent harnesses running org instructions, repo rules, skills, and local files need that same discipline, because most of them right now behave like a firewall with no rule order at all: every rule fires, and whichever one the hardware happens to process last is the one that wins.

The fix is a precedence matrix — and it has to live as a deployment artifact, not a paragraph in an onboarding doc. Something as blunt as Org Compliance > Project Architecture > Local Skill > User Prompt, shipped as a YAML file alongside the repository and versioned like code. The orchestrator parses it at runtime to build the context window on purpose: append higher-precedence instructions in the positions the model actually attends to, filter out lower-tier rules that would contradict a higher tag, and stop hoping proximity sorts it out for you.

From what I’ve seen across these deployments, the teams that get this right stop debugging the phrasing of their prompts and start debugging their precedence configurations instead. That’s the whole shift. Enterprise Copilot rollouts scale past pilot stage when an instruction conflict trips a predictable, loggable error at build or deploy time — not when it produces a plausible-looking answer that happens to be wrong in production, three weeks after everyone stopped watching for it.