Routing Classifications to Llama 3 to Save Frontier Tokens

The Homogeneous Architecture Trap

An engineering team discovers the OpenAI API, ships something in a week, and it works. Six months later the token bill is scaling with user growth and nobody can explain exactly why — because nobody stopped to ask whether every request actually needed a frontier model.

If you’re routing boolean classification tasks to GPT-4o, you’re paying senior-engineer rates for work a lightweight classifier could handle.

This is homogeneous LLM architecture: one endpoint, one model, every request hitting the same computational weight regardless of what you’re actually asking. The API equivalent of using a surgical robot to sort mail.

The problem runs deeper than cost. Sending a simple “is this a billing query or a technical query?” decision to a hosted frontier model introduces three penalties at once: API latency from the external network round-trip, per-token billing on a task that barely needs tokens to resolve, and contention on shared infrastructure you have no control over.

An intent router addresses this. The system evaluates each inbound request and routes the bulk of traffic to a locally hosted Llama 3 8B instance running on vLLM, escalating the rest to GPT-4o. Take a 70/30 split as an illustrative example. The right split isn’t arbitrary — it should reflect the actual complexity distribution of your production workloads, measured over a period of shadow traffic before you touch a single routing rule.

Unit Economics and the 70/30 Split

The cost argument for local models starts at the token level but needs to be evaluated at the decision level. Those are different things.

At time of writing, OpenAI’s published GPT-4o rates stand at $2.50 per million input tokens and$10.00 per million output tokens (openai.com/api/pricing — verify before modelling). An intent classification payload of, say, 200 input tokens and 10 output tokens runs about $0.0006 per call before network overhead. That sounds negligible. Then you run, say, 50,000 classifications per day.

Llama 3 8B on a mid-tier GPU changes the denominator entirely. The model weights are open; vLLM is open-source. Your recurring cost is GPU instance time; A10G on-demand rates vary by provider, region, and commitment tier, so pull the current figure from your cloud provider’s pricing page before modelling the economics. An A10G carries 24 GB of VRAM, enough to fit the 8B model in FP16 with room for KV cache across concurrent short-sequence classification requests. At high utilization, the amortized cost per classification call can be well below the GPT-4o path, and lower still if you’re filling the GPU efficiently.

Latency is the second variable, and it’s the one that shows up in user experience. Hosted frontier-model classification calls carry end-to-end latency dominated by provider-side queueing and generation, while a local vLLM path for the same short payloads avoids the external network round-trip and can respond considerably faster. Cold connections add DNS and TLS setup costs on top, but those don’t recur on every pooled request. When a human is waiting on a classification result before the next screen loads, they feel that difference. Every time.

A split like 70/30 comes from measuring the confidence distribution of the local model across a period of shadow traffic. If 70% of inbound requests produce logprob-confident classifications from Llama 3 8B, the other 30% are ambiguous, multi-intent, or edge cases requiring reasoning depth the 8B model simply doesn’t have.

Routing Support Tickets via vLLM

A clear example is a ticket triaging system. If an organization processes a large volume of citizen feedback submissions — across portal forms and email channels — and every single submission hits a commercial LLM API for classification across a set of predefined department codes, the cost profile is easy to predict.

In that setup, token costs scale linearly with portal usage, with no complexity filter in sight. A submission reading “I want to update my address” incurs the same computational overhead as a multi-paragraph complaint spanning three departments with a dispute history attached.

A two-stage pipeline fits this case. Stage one: a Llama 3 8B classifier on a vLLM inference server receives the submission text and returns a department code from the predefined set. Convert the token logprobs into a normalized top-class probability; the router routes locally when that score exceeds a threshold calibrated during a shadow period. Submissions scoring above it route locally. Those falling below get packaged and forwarded to GPT-4o for deeper analysis.

The likely result is a higher capture rate at the local tier than on general traffic. Structured form submissions don’t vary as wildly as open-domain requests, so the local model’s confidence tends to stay high. Every submission handled locally never touches the external API, so external LLM API spend for this classifier falls with the capture rate; total serving cost depends separately on the fixed GPU-hours required to keep the local tier warm. Routing latency for locally handled submissions drops too — a difference that matters when the confirmation screen needs the classification result to show the citizen their correct next steps.

Escalating IoT Telemetry to Frontier Models

Text classification is the easy case. The more instructive one is mixed telemetry analysis.

IoT sensors on water infrastructure stream continuous telemetry: water quality parameters, pump pressures, flow rates, equipment status flags. Anomaly detection runs against that stream in near-real-time.

First-pass analysis can run on Llama 3 8B. The model receives structured context covering normal operational envelopes for each equipment type, reads each telemetry window, and makes a binary pass/flag decision. If most events at normal operating volumes are routine, the local capture rate can be higher than the ticket routing case, because anomaly detection against known signatures is a narrower, more structured problem than open-domain intent classification. The per-inference cost is low enough to treat this as a continuous background process.

When the local model flags an event, a second check applies. Pattern matches a known failure signature? Routes to a deterministic alert handler with no LLM in the loop.

The escalation path to GPT-4o fires on one specific condition: the local model flags an event and the pattern doesn’t match anything in the reference set. That means the anomaly is novel, multi-variable, or involves cross-sensor correlation the 8B model can’t confidently resolve. The router packages the relevant time-series window, the sensor metadata, and the local model’s uncertainty signal, then forwards the bundle to GPT-4o for root cause synthesis.

The analogy that fits: Llama 3 8B is the monitoring engineer reading logs at 3am. GPT-4o is the senior engineer you call when something doesn’t match anything in the runbook. You don’t pay senior engineers to read healthy server logs.

Infrastructure Footprint for Local Models

Owning the local inference tier means owning the operational complexity that comes with it. This is the trade-off you’re accepting when you move off a pure API approach — and it’s worth being honest about upfront.

vLLM is a sensible choice because naive model loading in Python doesn’t handle concurrent requests without manual batching logic. vLLM is an inference server built for production serving. Its core mechanism is paged attention, which manages the key-value cache in GPU memory using non-contiguous memory blocks — borrowing the virtual memory paging concept from operating systems. Without it, a spike in concurrent requests exhausts contiguous KV cache memory and requests queue or fail. With paged attention, the GPU handles variable-length concurrent sequences with significantly higher utilization.

A reasonable deployment footprint for each 8B model instance is a single mid-tier GPU node. These instances run as microservices with standard health endpoints. The intent router sits upstream of both the vLLM server and the external LLM API — structurally it’s an API gateway with a routing policy keyed on confidence scores rather than URL path patterns.

Cold-start latency is the operational penalty worth naming explicitly. An 8B model takes a while to load onto a GPU. Keep a minimum of one warm instance per workload type to avoid cold-starts on the first request of the day. That adds a small fixed cost worth accounting for in your capacity plan.

Composite Systems as the Production Standard

Single-model LLM architectures work at prototype scale. At production scale, treating every request identically — regardless of complexity — is a cost structure that compounds quietly until it shows up on a finance report.

The teams who reduce AI operating costs over the next 12 months are likely those treating their LLM stack the way they already treat their compute stack: tiered by cost and capability, with explicit routing logic between tiers.

The first step isn’t building a router. It’s measuring your own traffic. Pull 30 days of LLM API logs, classify calls by output complexity, and count how many were binary decisions or entity extraction against a fixed schema. Route those locally.

Keep the frontier API for synthesis, multi-step reasoning, and the cases your local model can’t handle with confidence. The 70% figure here is illustrative. Your distribution will differ — classification-heavy workloads skew higher; open-ended generation workloads skew lower. The routing principle holds regardless.

There’s also a resilience argument that gets less attention than cost. When you own the routing layer and the local inference tier, you own your core classification loop. When an API provider revises pricing, enforces new rate limits, or goes down, the locally-served share of your traffic keeps moving without interruption. That’s not a contingency plan. It’s a structural property of the architecture.

Measure your traffic. Build the router. The cost and resilience gains follow from those two steps.