The Failure Mode Is Not Bad Prompting

Your enterprise Copilot isn’t hallucinating. It’s following orders — it just has too many bosses and no chain of command.

Engineering teams treat agent instructions like folklore: passed down in a Slack thread, copied into a new repo, half-remembered by the time anyone needs them. They should be treating them like configuration. A modern agent harness stacks organisational constraints, repository rules, workspace skills, custom agent personas, and local files into a single context window before the model ever sees a user prompt. Each layer gets written by a different person, at a different time, for a different reason. None of them can see the others.

When an agent acts unpredictably, the reflex is to blame the prompt — rewrite it, add capital letters, repeat “IMPORTANT” three times. Wrong diagnosis. The real failure mode is two valid instructions disagreeing with no recorded winner. Global policy says mask PII; a local file says dump the payload for debugging. Both are correct in their own scope. The model has no scope awareness, though, so it picks one — and not necessarily the same one twice.

Its arbiter of last resort turns out to be proximity, not business logic. Liu et al.’s 2023 paper “Lost in the Middle: How Language Models Use Long Contexts” found that retrieval accuracy degrades measurably when relevant information sits in the middle of a long context, with performance strongest at the beginning and end. That study measured retrieval in long-document question-answering, not instruction-following in agent harnesses — the extrapolation isn’t proven, but the mechanism generalises cleanly enough. Map the finding onto an instruction stack: org policy loads first, local .agents.md loads last, and the user prompt gets appended after both. In an instruction stack, that finding is a warning sign rather than proof: placement can bias what the model uses, so file load order can become an accidental tie-breaker when the harness has no explicit precedence rule. That’s not a policy decision. It’s an accident of file load order. And it fails silently: the agent never says “I resolved a conflict, here’s why.” It just does one thing today and something else next week, and everyone assumes the prompt changed when the architecture never did.

Enterprise Guidelines Versus Repository Files

Suppose a team runs a multi-agent deployment under a strict compliance regime. The organisational layer carries a system instruction to mask all personally identifiable information in any logs the agents write — standard stuff, non-negotiable, the kind of line a compliance officer signs off once and never revisits.

One repository in the stack, an analytics microservice, has its own .agents.md. A developer added it months earlier while chasing an integration bug, instructing the crew to dump raw API payloads to logs so the team could trace field mismatches. Reasonable, in isolation. Nobody deletes it once the bug is fixed, because nobody thinks of it as a standing instruction. It’s just a note.

Here’s the part that should worry every engineering lead running Copilot or a similar harness at scale: the agent wouldn’t consistently violate one rule or the other. It would flip. Same repo, same two instructions loaded, different outcome depending on how that day’s prompt happened to be phrased — sometimes masking correctly, sometimes not. The only variable that changes between a masked run and a leaking run is token positioning in the assembled context — nothing about the underlying rules has shifted. The result is intermittent PII leakage into development logs, a genuinely bad outcome in a compliance-sensitive setting, and no systemic way to inspect which instruction had “won” on any given run. There would be no log line reading “local override applied.” There would be just an output, correct or not, and no way to know which one you’d get until it landed.

Adding more adjectives to the global prompt doesn’t fix it. Stronger language may change the outcome on some runs but won’t stop the leak from recurring — because the underlying problem is never the wording. It is that two instructions occupy the same authority tier with no arbitration mechanism between them.

A Second Collision: Local Skills Rewriting Stable APIs

A baseline project instruction can be explicit: never modify core API routing logic. That file can predate most of the skills layered on top of it.

If a developer later adds a local skill to scaffold CRUD endpoints quickly, its instructions can tell the agent to update the main router automatically whenever a new endpoint is scaffolded, which is genuinely useful when you’re cranking out boilerplate. The collision can go unflagged, because on paper the two instructions look like they are about different things: one a constraint on architecture, the other a workflow convenience.

They aren’t different things. They are competing claims on the same file, and the local skill — loaded closer to the point of generation — can win. The agent can then rewrite stable routing logic to wire in the new endpoints and break the build. Nothing exotic has to happen. No jailbreak, no adversarial prompt. Two instructions that were each individually approved simply outrank each other in an order nobody chose.

That scenario is the whole argument in miniature: system prompts aren’t documentation, they’re competing executable directives, and a four-layer instruction stack without an explicit precedence rule will eventually let the wrong layer win. It’s not a question of if. It’s a question of which build it breaks first.

Guardrails and Deterministic Pre-Flight Routing

Where the stakes are high, this can’t be a “we’ll tune the prompt” problem. If database migrations run against production schemas that control monitoring hardware calibration data, a deny-by-default rule against destructive schema operations is non-negotiable — nobody should bet that outcome on which instruction the model happens to attend to on a given run.

So don’t. Build an orchestration layer that sits outside the LLM entirely and rejects unapproved actions before they execute, regardless of what any instruction layer told the agent to do. It works in two parts: a deterministic SQL parser converts every generated SQL payload into an abstract syntax tree and rejects destructive statement types — DROP, ALTER, TRUNCATE — outside a small allow-listed migration path, while a pre-flight LLM pass acts as a semantic router, classifying intent before anything runs. If the AST parser finds restricted syntax outside the approved migration path, the orchestrator rejects the request immediately — that check alone is sufficient to block execution, no second opinion required. The semantic router runs in parallel to catch destructive intent earlier, before SQL generation even completes, but it has no authority to overrule the parser in either direction: it can’t wave through what the parser blocks, and a router miss doesn’t let anything past the parser either. No partial execution, no negotiation, no “are you sure.”

The point isn’t that the LLM got smarter about not dropping tables. It’s that you stop asking it to be the enforcement layer for the one class of action where being wrong is expensive to reverse. There’s a real trade-off here worth naming: the AST parser only catches what its keyword and allow-list rules anticipate, so extending it to a new class of destructive action means updating the parser, not just the prompt. That’s a maintenance cost. But it’s a cost you can audit — unlike a prompt’s phrasing, which you’re mostly just hoping about. Alignment language and capitalised warnings are not a substitute for a deterministic gate that sits above the model’s reasoning.

Deploying the Precedence Matrix

Firewalls don’t ask a packet to explain itself. They evaluate it against rules in a strict, ordered sequence and accept or reject deterministically. Agent harnesses running org instructions, repo rules, skills, and local files need that same discipline, because a harness with no precedence rule behaves like a firewall with no rule order at all: every rule fires, and whichever one the hardware happens to process last is the one that wins.

The fix is a precedence matrix — and it has to live as a deployment artifact, not a paragraph in an onboarding doc. Something as blunt as Org Compliance > Project Architecture > Local Skill > User Prompt, shipped as a YAML file alongside the repository and versioned like code. The orchestrator parses it at runtime to build the context window on purpose: append higher-precedence instructions in the positions the model actually attends to, filter out lower-tier rules that would contradict a higher tag, and stop hoping proximity sorts it out for you.

Stop debugging the phrasing of your prompts and start debugging your precedence configurations instead. That’s the whole shift. An instruction conflict should trip a predictable, loggable error at build or deploy time, not produce a plausible-looking answer that happens to be wrong in production, long after everyone stopped watching for it.