Runtime plane / Tier 1 / Phase 0
Infrastructure & Runtime
Where does it run?
The compute, scheduling, networking and resilience substrate every other harness sits on — Kubernetes across public and private cloud, GPU-aware and multi-tenant from the start.
- Kubernetes clusters across public and private cloud, with a private path for regulated workloads
- GPU scheduling, quota and gang-scheduling so accelerators are not idle
- Storage, queues, service mesh, secrets distribution and network policy
- High availability, disaster recovery and capacity planning
- Tenancy primitives: namespaces, node pools, network segmentation
GPU pools and autoscaling for inference
ingress and egress paths
durable execution needs stateful backing services
sandbox isolation primitives
platform metrics and cost data
network policy, secrets, workload identity roots
Platform Engineering / Cloud Infrastructure
- Head of Platform Engineering — accountable owner
- Site reliability engineer — SLOs, capacity, DR drills
- Kubernetes platform engineer — cluster and GPU scheduling
- Network and infrastructure security engineer
- FinOps analyst — accelerator cost per team
Architectural recommendation, not a research finding.
Done when a team can deploy an agent workload to a private-cloud cluster with GPU quota, secrets, a network policy and a tested restore — without filing a ticket.
Signals to watch
- GPU utilisation and queue wait time
- Scale-from-zero cold start P95
- Cluster availability against SLO
- Cost per GPU hour by tenant
- Restore time in the last DR drill
- GPU utilisation stuck at 25–35% without quota and gang scheduling; disciplined scheduling moves it to roughly 60–85%
- Noisy-neighbour effects across tenants sharing accelerators
- Cold starts on scale-from-zero making interactive agents feel broken
- Disaster recovery documented but never rehearsed
Tool and standards snapshot · last reviewed approximately mid-2026
Open source
- Kubernetes
- Kueue (GPU quota and gang scheduling)
- Karpenter / Cluster Autoscaler
- NVIDIA GPU Operator
- Istio, Linkerd or Cilium
- Argo CD
- Terraform or Crossplane
- Kafka or NATS
- MinIO, PostgreSQL, Redis
- cert-manager, External Secrets
Managed
- EKS / GKE / AKS
- OpenShift
- Managed Kafka and Postgres services
Standards
- OCI images
- CNCF conformance
- SPIFFE workload identity roots
- FinOps FOCUS cost schema
Two things commonly hide inside Infrastructure and should not: model serving and the AI gateway. Both have their own lifecycle and trust boundary, which is why they are separate harnesses here.