Runtime plane / Phase 1
Model & Inference
Which intelligence executes?
Serve, version and scale the models — proprietary and open-weight — with predictable latency and cost, and keep them swappable so no single provider becomes load-bearing.
Use this harness in my journey →- Inference serving: batching, KV cache, tensor and pipeline parallelism
- Model registry and version pinning, including deprecation calendars
- Routing between model tiers, plus fallbacks when a provider degrades
- Embedding and reranking endpoints used by retrieval
- Capacity and quota per tenant on shared accelerators
Accountable owner
ML Platform / AI Engineering
- Head of ML Platform — accountable owner
- Inference engineer — serving performance and memory
- GPU performance engineer — batching, quantisation, cache
- Model operations engineer — versions, rollouts, deprecations
Ownership and phase are architectural recommendations, not assessed results.
runs on scheduled GPU capacity
exposed only through the gateway
consumes assembled context under a token budget
emits token and latency telemetry
model changes gated by regression evaluation
Pilot needs one hosted model and one self-hosted fallback behind the same interface, both version-pinned, with a documented switch procedure.
Signals to watch
- Time to first token and tokens per second
- P99 end-to-end latency by model tier
- Prefix cache hit rate
- Cost per thousand tokens by tenant
- Share of traffic that could fail over to the open-weight path
Model selection and decoding settings require new quality/safety evidence. Weight changes are a distinct release surface, not an invisible harness tweak.
How governed change crosses the architecture ↗Failure modes
- Vendor lock-in through provider-specific features in prompts and tools
- Silent provider upgrades shifting behaviour mid-quarter
- KV cache reuse leaking context across tenants if isolation is careless
- P99 latency collapsing under burst without admission control
Existing tool and standards snapshot · approximately mid-2026. Examples, not required dependencies; verify current versions before adoption.
Open source
- vLLM
- llm-d
- SGLang
- TensorRT-LLM
- KServe or Ray Serve
- Hugging Face TGI
- Text Embeddings Inference
Managed
- Anthropic API
- Bedrock, Vertex AI, Azure AI Foundry
- Managed embedding and rerank endpoints
Standards
- OpenAI-compatible API surface
- JSON Schema structured outputs
- Model cards for each served version
Keeping this harness distinct from Infrastructure is what makes model choice reversible. If serving is buried in the platform, changing models becomes a platform migration.