Runtime plane / Phase 0
Infrastructure & Runtime
Where does it run?
The compute, scheduling, networking and resilience substrate every other harness sits on — Kubernetes across public and private cloud, GPU-aware and multi-tenant from the start.
Use this harness in my journey →- Kubernetes clusters across public and private cloud, with a private path for regulated workloads
- GPU scheduling, quota and gang-scheduling so accelerators are not idle
- Storage, queues, service mesh, secrets distribution and network policy
- High availability, disaster recovery and capacity planning
- Tenancy primitives: namespaces, node pools, network segmentation
Accountable owner
Platform Engineering / Cloud Infrastructure
- Head of Platform Engineering — accountable owner
- Site reliability engineer — SLOs, capacity, DR drills
- Kubernetes platform engineer — cluster and GPU scheduling
- Network and infrastructure security engineer
- FinOps analyst — accelerator cost per team
Ownership and phase are architectural recommendations, not assessed results.
GPU pools and autoscaling for inference
ingress and egress paths
durable execution needs stateful backing services
sandbox isolation primitives
platform metrics and cost data
network policy, secrets, workload identity roots
Done when a team can deploy an agent workload to a private-cloud cluster with GPU quota, secrets, a network policy and a tested restore — without filing a ticket.
Signals to watch
- GPU utilisation and queue wait time
- Scale-from-zero cold start P95
- Cluster availability against SLO
- Cost per GPU hour by tenant
- Restore time in the last DR drill
Resource policy and deployment configuration may change within reviewed limits. Isolation, budget ceilings and recovery compatibility remain protected.
How governed change crosses the architecture ↗Failure modes
- GPU utilisation stuck at 25–35% without quota and gang scheduling; disciplined scheduling moves it to roughly 60–85%
- Noisy-neighbour effects across tenants sharing accelerators
- Cold starts on scale-from-zero making interactive agents feel broken
- Disaster recovery documented but never rehearsed
Existing tool and standards snapshot · approximately mid-2026. Examples, not required dependencies; verify current versions before adoption.
Open source
- Kubernetes
- Kueue (GPU quota and gang scheduling)
- Karpenter / Cluster Autoscaler
- NVIDIA GPU Operator
- Istio, Linkerd or Cilium
- Argo CD
- Terraform or Crossplane
- Kafka or NATS
- MinIO, PostgreSQL, Redis
- cert-manager, External Secrets
Managed
- EKS / GKE / AKS
- OpenShift
- Managed Kafka and Postgres services
Standards
- OCI images
- CNCF conformance
- SPIFFE workload identity roots
- FinOps FOCUS cost schema
Two things commonly hide inside Infrastructure and should not: model serving and the AI gateway. Both have their own lifecycle and trust boundary, which is why they are separate harnesses here.