Runtime plane / Tier 1 / Phase 1
AI Gateway
How is model traffic controlled?
One control point for every model call: authentication, tenant quota, routing, caching, cost metering and the edge layer of guardrail enforcement.
- Authentication and authorisation for callers, human and agent
- Per-tenant quotas, rate limits and budget ceilings
- Semantic and cost-aware routing across model tiers and providers
- Response and prefix caching
- Token metering and chargeback per team, tenant and task
- Edge guardrail hooks: request and response screening before anything reaches a model
all channel traffic enters here
applies guardrail policy at the edge
fronts every inference endpoint
cost and latency metering
routing signals from classifiers
enforces tenant and budget policy
Platform Engineering, with Identity & Access Management
- Platform engineering lead — accountable owner
- API gateway engineer — routing, caching, rate limits
- IAM engineer — token issuance, scopes, workload identity
- FinOps analyst — budgets and chargeback
Architectural recommendation, not a research finding.
Pilot needs authentication, per-tenant quota and token metering. Semantic routing and caching are optimisations for phase 2.
Signals to watch
- Requests per second and gateway-added P99 overhead
- Quota and budget rejections by tenant
- Cache hit rate and cache-attributable savings
- Cost per tenant per day against budget
- A single point of failure for every agent in the estate
- P99 spikes above roughly 500 requests per second on a single gateway process — shard horizontally
- Policy drift between gateway guardrails and the in-loop guardrails, so a call blocked at one layer passes at the other
- Cache keys that ignore tenant, leaking responses across boundaries
Tool and standards snapshot · last reviewed approximately mid-2026
Open source
- LiteLLM
- Envoy AI Gateway
- Kong AI Gateway
- Apache APISIX
- Higress
Managed
- Cloud API gateways with AI plugins
- Provider-native routing layers
- Commercial LLM gateways with observability bundled
Standards
- OpenAI-compatible routing surface
- OAuth 2.1 / OIDC for caller identity
- FOCUS-aligned cost records
Splitting the gateway out of Infrastructure was one of two structural corrections to the original ten-harness model. It is the only place where cost, identity and routing can be governed for every call at once.