AI Gateway

• Authentication and authorisation for callers, human and agent
• Per-tenant quotas, rate limits and budget ceilings
• Semantic and cost-aware routing across model tiers and providers
• Response and prefix caching
• Token metering and chargeback per team, tenant and task
• Edge guardrail hooks: request and response screening before anything reaches a model

Accountable owner

Platform Engineering, with Identity & Access Management

• Platform engineering lead — accountable owner
• API gateway engineer — routing, caching, rate limits
• IAM engineer — token issuance, scopes, workload identity
• FinOps analyst — budgets and chargeback

Ownership and phase are architectural recommendations, not assessed results.

Pilot needs authentication, per-tenant quota and token metering. Semantic routing and caching are optimisations for phase 2.

Signals to watch

• Requests per second and gateway-added P99 overhead
• Quota and budget rejections by tenant
• Cache hit rate and cache-attributable savings
• Cost per tenant per day against budget

Routing and caching rules may adapt under versioned policy. Identity checks, spending limits and deny decisions cannot be weakened by an optimizer.

Failure modes

1. A single point of failure for every agent in the estate
2. P99 spikes above roughly 500 requests per second on a single gateway process — shard horizontally
3. Policy drift between gateway guardrails and the in-loop guardrails, so a call blocked at one layer passes at the other
4. Cache keys that ignore tenant, leaking responses across boundaries

Existing tool and standards snapshot · approximately mid-2026. Examples, not required dependencies; verify current versions before adoption.

Splitting the gateway out of Infrastructure was one of two structural corrections to the original ten-harness model. It is the only place where cost, identity and routing can be governed for every call at once.