The Death of the Static Instruction Contract

If your agent runtime can mutate the system prompt mid-conversation — inject a new instruction set at turn 14 instead of locking it in at turn 0 — the eval contract stops being static. That’s a real architectural lever. It’s also a quiet demolition job on the assumption every eval pipeline was built on.

The old model treats the system prompt like static configuration: it loads once, sits above the transcript, and every turn after that gets graded against it. Eval engineers built entire scoring frameworks on that premise — pull the system prompt, pull the assistant turns, check compliance against one fixed rulebook. Standard observability tooling assumes an append-only transcript, too: user turn, assistant turn, user turn, assistant turn, stacked chronologically, nothing above the first system message ever moving.

That assumption doesn’t survive contact with a contract that can change mid-stream. Drop a system message at turn 14, and turn 15 now has to be judged against different rules than everything before it. If your eval harness doesn’t catch that change, it grades turn 15 against turn 1’s original rulebook — and produces a false negative. The agent looks like it violated a policy. It wasn’t even operating under that policy anymore.

Pattern One: Editorial Verification Gates

We run this pattern in our editorial pipeline. A primary extraction agent pulls structured data out of source content — names, figures, claims, attributions — and it runs permissive, because most extraction work simply doesn’t need friction.

Then it hits a flag term. Our routing logic injects a stricter compliance system message mid-run, and the agent’s posture flips: from “extract broadly” to “verify every attribution before you emit it.”

Read the raw transcript cold, without knowing any of that, and it looks like the agent had a personality change at turn 9 — permissive going in, cautious coming out, no explanation in sight. Nothing in a bare {"role": "assistant", "content": "..."} log tells you why. An eval engineer scoring that run against the original prompt sees an agent second-guessing itself for no visible reason and marks it inconsistent. It isn’t. It’s compliant — with a contract the log never bothered to record.

Pattern Two: WaterDoctor Emergency Protocols

WaterDoctor’s diagnostic handoff pipeline hits the same structural wall, except the stakes are higher. The first agent in the chain does routine symptom checks against water treatment hardware telemetry — flow rates, pressure differentials, filter cycle counts. Low urgency, standard conversational flow.

Then the symptom check surfaces something like pump cavitation, and everything changes. We don’t want that agent working the problem on the same permissive rulebook it uses for a filter-change reminder. So the system prompt gets rewritten mid-run to enforce an emergency mitigation protocol: stop diagnosing, confirm the shutdown sequence, escalate to a technician.

Here’s the hard part: telling whether the agent actually followed that emergency protocol, versus just landing on the right answer by coincidence, requires knowing exactly which turn the rulebook changed. Hide that injection boundary in replay and an eval engineer can’t tell “the agent correctly executed the emergency protocol it was just handed” apart from “the agent got lucky under the old permissive prompt.” Those are different findings. They carry different safety implications. A flat transcript can’t tell them apart, and that’s the whole problem in one sentence.

Structural Markers for Mutable State

The fix isn’t clever — it’s a schema problem, but not the one people assume. In runtimes that model instructions as ordered messages, that array can hold a second {"role": "system", "content": "..."} object at index 14 without any format changes at all — the turn just slots in. Runtimes that instead hoist system instructions into a dedicated field outside the message list don’t get that convenience; they need an explicit mutation event type to record the same thing. Either way, the wire format was never what stood in the way. The obstacle is tooling that pulls the system message out at index 0 and treats it as run-level metadata — loaded once, cached, done — instead of reading the transcript in order and catching the second one when it shows up. Treat every system message as a state mutation logged in place, not configuration hoisted out of the timeline, and the rest follows.

Every mid-run injection should log four things: a timestamp, a turn index, the delta between old and new instruction payload (not the full new text — the delta, because that’s what an auditor actually wants to diff), and the provenance of whatever triggered it. That’s more logging discipline than most teams currently practice. It’s also the floor, not the ceiling, for keeping eval results honest. The provenance field matters more than it looks like it should. “Routing logic injected a new system message” tells you nothing. “Classifier flag-terms-v3 matched on turn 9, triggered policy strict-compliance-v2” tells you everything.

Once that metadata exists, an eval framework can parse the boundary and split the grading: turns 0 through 8 against contract A, turns 9 onward against contract B. Mechanical fix, once the boundary is structurally marked. Skip it, and no amount of careful prompt engineering saves you — the eval is scoring against the wrong contract by construction.

Replayability Dictates Production Readiness

Dynamic rulebooks earn their complexity. They let an agent run a multi-stage workflow — editorial extraction that escalates to compliance review, a diagnostic check that escalates to emergency protocol — without front-loading every possible policy into the first context window. That’s the whole point of shipping the feature in the first place.

But an audit log that can’t show where the contract changed isn’t an audit log. It’s a transcript with a hole in it. Treat every mid-run system message as a state mutation — timestamp, turn index, delta, provenance ID — or don’t ship the pattern to production. There’s no middle version of this.

None of that instrumentation is free. It’s engineering time spent on logging paths that add zero user-facing value, and if your system prompts never mutate mid-run, you don’t need any of it. Skip ahead. But for the teams that do mutate — and more will, as this kind of runtime capability spreads — skipping the instrumentation isn’t a shortcut. It’s a decision to fly blind. If you can’t replay a conversation and point to the exact turn the rulebook changed, you can’t tell an agent that followed the new rules from one that happened to guess right. That’s not something you paper over in a postmortem. It has to be built into the transcript from the first injection, or it doesn’t exist at all.