Agent Gateway Policies Need Simulation Runs

Programmable Control Planes Defeat Static Text

We caught it during a test run, not a production incident: our agent had pulled a prospect’s full customer PII without consent for a data export. That distinction — test versus production — is the only reason this is a case study and not a disclosure letter. Discovering an autonomous agent’s cross-tenant or over-broad data access during a live incident is the expensive version of the same mistake. Reading YAML after the fact does not un-fetch the data.

Google Cloud’s Vertex AI Reasoning Engine and AWS Bedrock Agents have both moved past “attach a policy, hope it holds.” Bedrock Agents runs on a customer-created IAM execution role — typically named AmazonBedrockExecutionRoleForAgents_* — and the permission boundary that actually governs the agent is that role policy, evaluated each time the agent invokes a configured action group or its backing Lambda, not a one-time approval granted when the agent is deployed. Google’s Vertex AI extension model pushes authorization into the same invocation path: an extension call runs through its configured authentication and the downstream API’s own permission checks, rather than being validated only when the extension is registered. Different vendors, different dialects, same underlying shift — access is no longer a static grant checked once at the gateway. It’s a runtime decision, re-evaluated at the points where the agent invokes tools and downstream APIs.

That’s a control plane, not a policy file. And you can’t audit a control plane by reading it. You have to run it.

The combinatorial problem is easy to state and even easier to underestimate. A single agent deployment has a matrix of user role, agent identity, tool selected, data source, and downstream action. Five dimensions, each with a handful of values, and suddenly you’ve got hundreds of reachable states. A human reviewing a JSON policy document is checking whether the grammar looks right — not whether state 214 of 360 quietly permits an unbounded query. Policy text encodes intent. It doesn’t simulate execution. Those are two different documents, and platform teams keep treating them as one.

The PII Over-Fetch in an SME CRM Integration

The project was a scoped sales-assistant agent for an SME client, built on their existing CRM and backed by DynamoDB. The IAM policy attached to the action-group Lambda’s execution role granted dynamodb:BatchGetItem on the customer table — the role that actually runs the downstream query once the Bedrock Agent’s service role hands off invocation to it. On paper, that looked like a normal, bounded read permission — the kind you’d wave through in a five-minute review. It specified no projection expression and restricted no attributes.

Here’s where it went sideways. The agent’s query-generation tool, when asked a broad sales question, constructed a BatchGetItem call with no attribute filter. IAM doesn’t care that the intent was “get deal stage.” It sees a permitted action on a permitted table and hands back everything the row contains, PII fields included. The policy was correct as written. It was wrong as executed. DynamoDB access control at the IAM layer defaults to table-level and action-level grants, but AWS does support finer-grained restriction — condition keys like dynamodb:Attributes can scope a policy down to named fields. This policy didn’t use them, and there was no compensating control on the other side either: the tool layer that should have enforced field-level projection on the query itself wasn’t in the access decision path at all. The gap wasn’t a broken rule. It was a rule with no opinion on a dimension nobody thought to test.

We caught it because we ran the agent against realistic prompts before go-live and logged every tool-call payload. That’s not a code-review practice — it’s a simulation practice. A static reading of the IAM policy, the kind you’d do in a change-approval ticket, would have sailed through. And to be fair, this approach isn’t free either: logging full payloads at scale carries its own storage and review overhead, and it only surfaces the prompts someone thought to test. A narrower prompt set would have missed this just as cleanly as the policy review did. The lesson we took away: nobody can predict dynamic query generation by reading the grant that authorizes it. You have to fire the query and look at what actually comes back.

Actuating Valves and Physical Stakes in WaterDoctor

WaterDoctor raises the stakes from a compliance problem to a physical-safety problem. Diagnostic agents there read telemetry off IoT nodes attached to pump and valve infrastructure. They are explicitly not permitted to actuate anything — close a valve, reset a pump limit — without a human in the loop. That boundary lives inside a zero-trust gateway between the agent’s tool layer and the PLC-facing API.

The failure mode we designed against isn’t “the agent goes rogue.” It’s “the deny rule has a syntax error, or a scope mismatch, or someone updates the tool schema six months later and the new field slips past the old boundary.” A misconfigured allow on an actuation endpoint doesn’t produce a bad customer support answer. It produces a valve-state change with no human ever having reviewed it. That’s not a category of bug you want to discover from the incident log.

So the requirement was a literal dry run: inject a “close valve 4” command through the same tool-calling path a live agent would use, and confirm the gateway rejects it at the network layer, before the request ever reaches the PLC. Not a unit test on the policy document — an actual request, actually blocked, actually logged. We run that same injected-command test against every new telemetry tool added to the agent’s toolset, because each new tool is a new edge in the access graph, and the old deny rule was written against the old graph. Passing the test confirms the rule holds against the payload shapes we’ve exercised so far. It doesn’t prove the rule is unbreakable against a shape nobody has written yet — which is exactly why it has to run on every change, not just at launch.

Executing the Dry-Run Simulation Matrix

User Role	Agent Identity	Selected Tool	Downstream Action	Simulation Result
Sales Rep	Sales Asst	CRM API	Read PII	❌ Denied
Sales Mgr	Sales Asst	CRM API	Read PII	✅ Allowed
Operator	Diagnostic Bot	IoT API	Read Telemetry	✅ Allowed
Operator	Diagnostic Bot	PLC API	Close Valve	❌ Denied
Illustrative — A binary pass/fail matrix evaluated during dry-runs to confirm dynamic resolution paths before deploying an agent.

The fix in both projects was the same: stop treating the policy as documentation and start treating it as executable configuration you run before every deployment, not just the first one.

Concretely, that means building the User Role × Agent Identity × Tool × Data Source × Downstream Action matrix explicitly and evaluating every reachable cell — not just the ones product expects to be used. AWS’s IAM Policy Simulator CLI can evaluate whether a given principal, action, and resource combination is allowed under the current policy set, without executing the action against live infrastructure. For gateways built on Kubernetes-native policy enforcement, Open Policy Agent’s opa eval supports the same dry-run pattern against Rego rules — feed it the exact JSON payload structure the agent would dispatch, and check the decision, not the intent. Neither tool substitutes for judgment. The simulator only checks what you ask it to check, and a matrix built from an incomplete tool inventory will pass cleanly while still missing the one cell that matters.

The output that matters isn’t “policy looks fine.” It’s a pass/fail table: this role, with this agent, calling this tool, against this data source, attempting this action — allowed or blocked. If a cell in that matrix surprises you, you just found the bug before a customer did.

Mandatory Regression Testing for New Connectors

Treat every access policy attached to an agent gateway as code. Code that isn’t tested is a guess with good formatting. The practical rule we now hold every deployment to: no new tool or data connector goes into an agent’s toolset without an automated access regression test that exercises the new edges in the permission graph — run against realistic and adversarial prompts, not just the happy path.

Gateways keep adding dynamic resolution, delegated credentials, and multi-hop authorization as AWS and Google build these out further. Manual policy review doesn’t scale against that. It was already missing failures at five dimensions and a few hundred states. It’s only going to miss more as the graph grows. Simulation isn’t a nice-to-have step before launch — at this level of complexity, it’s the most reliable way to know what an agent’s permissions actually allow, rather than what someone wrote them to allow. If you can’t simulate a permission, don’t ship the agent that depends on it.