AI Data Agents Need Metric Contracts, Not Raw SQL

ChatGPT Work now ships a Data agent that connects to your warehouse, writes its own SQL, and renders a dashboard before your coffee cools. Ask it for churn, active customers, or overdue invoices and it hands back a chart — axis labels, legend, the works. No analyst required. What it just did, quietly, is automate the argument that usually happens in the Tuesday leadership meeting: what does “active customer” actually mean here?

That argument used to be a feature, not a bug. Someone would push back on a number, ask how it was calculated, and the room would either agree on a definition or escalate to finance. Slow, but it caught errors. An agent that skips straight to a polished chart skips that friction too — and the friction was doing real work.

The SQL Problem Is Solved. The Definition Problem Isn’t.

Text-to-SQL is no longer the hard part. Modern models handle joins, window functions, and date arithmetic without much coaching, and context windows are wide enough to hold a full schema plus sample rows. The syntax is fine. What hasn’t moved is the problem sitting underneath it: an agent with table access still has to guess what “revenue,” “churn,” and “overdue” mean in your specific business, and it will guess with exactly the same confidence it would use for a correct answer.

That guess is the actual risk surface. Give an agent raw database credentials and a natural-language prompt, and you’ve handed it discretion over which columns represent truth. Nobody signed off on that discretion — it exists only because some column in the schema happened to look plausible.

A Churn Chart That Is Wrong and Looks Right

Suppose a team gives an agent a straightforward brief: calculate monthly user churn from the company’s subscription data and chart it. The agent finds the user table, finds a legacy subscription log left over from an earlier billing migration, joins them, and produces a clean, descending line — exactly the shape a churn chart is supposed to have.

It is wrong. The join uses account_created_at as the reference timestamp instead of subscription_cancelled_at, because the legacy table’s cancellation column has a different name than the one in the current schema, so the agent falls back to the timestamp that is present and well-populated in both tables. The query runs clean. The chart renders clean. The number is fabricated in the sense that actually matters: it measures account age decay, not churn.

That’s the failure mode worth losing sleep over. A broken query throws a stack trace, and someone fixes it before it ever reaches a slide. A wrong-but-plausible chart clears every technical check and walks straight into a deck. Nobody in that meeting is going to eyeball a smooth downward-sloping line and suspect the join condition — it looks like every other churn chart they’ve ever seen. That’s what makes it dangerous. It’s not obviously broken. It’s quietly wrong. And the cost of catching it late is exactly the kind of correction nobody wants to make in front of a board.

What Commission Payouts Show About “Recognized Revenue”

The same problem can appear in an ERP integration built around commission payouts. Sales wants commission calculated the moment a deal closes. Finance wants it calculated against recognized revenue, which under many accounting treatments means revenue tied to delivery milestones, not signature dates. Both sides have a defensible definition. Both sides have a spreadsheet that proves they are right.

A better SQL view doesn’t fix that — a better view just encodes one side’s opinion more elegantly, with cleaner syntax. The fix is to get finance and sales into a room, write down the exact definition of recognized revenue for commission purposes on paper, and have the CFO sign it. Only then should that document be translated into code. The definition will change as new product lines with different delivery terms come online, and every one of those changes should require a fresh signature before it ships. That signature isn’t ceremony. It’s the thing that makes the number defensible when someone challenges it in a payout dispute.

An LLM agent staring at a schema has none of that context. It has column names and data types — nothing more. Expecting it to reconstruct “recognized revenue” or “active customer” from that alone, and then trusting the output enough to put it in front of a board, is asking a language model to do the CFO’s job without the CFO’s context. That’s not a model capability gap. It’s a governance gap we’re pretending is a technical one.

The Semantic Layer Is the Firewall

The fix is to stop letting the agent see raw tables at all. Instead of prompting against the schema, the agent should prompt against a semantic model: a curated layer that defines each metric once, locking in the exact tables, joins, and filters that produce it. When someone asks for Q3 revenue, the agent doesn’t write a SELECT from scratch. It calls the revenue node in the semantic model and asks for it grouped by quarter.

This shrinks the agent’s action space on purpose. It can aggregate, filter, and visualize a defined metric. What it can’t do is decide, mid-query, that account_created_at looks like a reasonable stand-in for a missing cancellation date. That decision got made once, by a human, when the metric was defined — and it stays frozen until someone deliberately changes it. Semantic layer tools like dbt’s metrics layer or Cube exist precisely to put this boundary in place; which specific tool you pick matters far less than the principle underneath it — the agent’s SQL generation happens inside a fence, not outside one.

The trade-off is real, though. Analysts lose the ability to ask a genuinely novel question on the fly — if a metric isn’t modeled yet, the agent can’t invent it, and someone has to define and ship it first. Building and maintaining that layer is ongoing engineering work, not a one-time setup you check off and forget. That’s a fair price for not shipping fabricated numbers. But it’s a cost, not a free lunch.

Metrics as Signed Commits, Not Tribal Knowledge

The practical move is to treat metric definitions the way we already treat infrastructure: as version-controlled code with an approval gate. Each metric — revenue, churn, overdue invoice, active customer — lives as a definition in a repository. Changing it means opening a commit. Merging it requires sign-off from whoever owns that number in the business: finance for revenue, product for active-user definitions, credit control for overdue status.

That gives you what the signed commission document gives you: a paper trail. If a number gets challenged six months later, you don’t debug a prompt — you look at the commit history and see who approved the current definition and when it last changed. The AI agent’s job shrinks down to executing against whatever definition is currently active. Which, frankly, is the right amount of authority for a model to have.

No Contract, No Chart

SQL generation looked like the bottleneck holding back AI-driven analytics. It wasn’t. Trust was — and trust doesn’t come from better syntax. It comes from provenance. An AI data agent without a versioned, human-signed semantic layer underneath it isn’t an analytics tool. It’s a fast way to produce a wrong number that looks exactly like a right one.

The rule going forward should be simple enough to enforce in a design review: if a metric doesn’t have a signed contract behind it, the agent doesn’t get to chart it. Everything else is a prompt engineering problem wearing a governance problem’s clothes.