Model & Inference

• Inference serving: batching, KV cache, tensor and pipeline parallelism
• Model registry and version pinning, including deprecation calendars
• Routing between model tiers, plus fallbacks when a provider degrades
• Embedding and reranking endpoints used by retrieval
• Capacity and quota per tenant on shared accelerators

ML Platform / AI Engineering

• Head of ML Platform — accountable owner
• Inference engineer — serving performance and memory
• GPU performance engineer — batching, quantisation, cache
• Model operations engineer — versions, rollouts, deprecations

Architectural recommendation, not a research finding.

Pilot needs one hosted model and one self-hosted fallback behind the same interface, both version-pinned, with a documented switch procedure.

Signals to watch

• Time to first token and tokens per second
• P99 end-to-end latency by model tier
• Prefix cache hit rate
• Cost per thousand tokens by tenant
• Share of traffic that could fail over to the open-weight path

1. Vendor lock-in through provider-specific features in prompts and tools
2. Silent provider upgrades shifting behaviour mid-quarter
3. KV cache reuse leaking context across tenants if isolation is careless
4. P99 latency collapsing under burst without admission control

Tool and standards snapshot · last reviewed approximately mid-2026