Skip to main content

Two layers — most RL users only care about the first.

Inside a forked cluster (your GRPO scenario)#

This is the k3s that lives inside the VMs you fork (cluster mode). The fork engine is N-node (tree_create_cluster(vm_count) / adopt any live set) — there is no hard-coded 3.

Two guests prove the rows below. The minimal CI guest fixture is a 3-node control plane with flannel + kube-proxy and a real in-cluster Deployment; its default profile keeps stock k3s add-ons off so the lean path stays lean, and each add-on is an opt-in profile with its own proving test. The Ubuntu node runs stock Ubuntu 24.04 with k3s’ own containerd and takes Cilium, Tetragon, Longhorn, and the ML operators. Every ✅ is an integration test that brings the feature up, warm-forks the cluster, and checks source and fork keep working, then diverge independently. The exceptions are rows marked “not fork-tested”: their tests prove the feature on a live cluster without forking it.

FeatureStatusProven onNotes
k3s control plane (server + agents)✅ Todayboth3 Ready nodes; TLS / node IPs survive fork
N-node clusters (5, 20, …)✅ Today—Same fork path for any vm_count; limited by host RAM, not the API. Ubuntu nodes cost ~1.1–1.4 GiB each; the CI fixture ~3.2 GiB. Shrink guest memory or use a bigger box and a 20-node cluster forks like the 3-node case
Flannel (host-gw)✅ TodaybothShared L2 between member VMs
kube-proxy (ClusterIP by IP)✅ Todayboth
Deployments / ReplicaSets / Pods✅ Todaybothe.g. inner-load Ready on source and fork
ConfigMaps / Secrets✅ TodaybothExercised under churn before fork
overlayfs snapshotter (guest containerd)✅ Todayboth
CoreDNS✅ TodayCI guestDNS answers on all members across a warm fork
Traefik / Ingress✅ TodayCI guestHTTP through an Ingress on source and fork
ServiceLB✅ TodayCI guestTraefik Service reports guest IPs as LB ingress
metrics-server✅ TodayCI guestkubectl top / metrics.k8s.io Available on all three nodes across a fork
local-path / in-cluster PVC provisioning✅ TodayCI guestSame bytes on source and fork, then independent divergence
NetworkPolicy✅ TodayCI guestDefault-deny + allow enforced on source and fork (guest kernel carries ipset + xt_set, NFLOG/limit, nft_log/nft_limit)
HPA✅ TodayCI guestCPU-metrics scale-up on source and fork; utilization diverges after a fork-only load change
Embedded etcd datastore✅ TodayCI guestSingle-member --cluster-init WAL and writes survive a fork; per-bridge writes stay isolated
ArgoCD core GitOps✅ TodayCI guestIn-cluster git://; Application Synced/Healthy on both bridges; a fork-only commit syncs only on the fork
Cilium (eBPF CNI, kube-proxy replacement, CiliumNetworkPolicy)✅ TodaybothNative routing on the CI guest; Helm Cilium 1.16.10 on Ubuntu (k7d.cni=none)
Tetragon 1.7.0✅ TodayUbuntuFork-and-diff of the event stream: inject on a child, the source’s stream stays clean
Longhorn 1.12.1 (RWO / RWX CSI)✅ TodayUbuntuRWO + RWX PVCs and their markers survive a fork as reflink-CoW named data disks. Needs 4 vCPU / 4 GiB nodes
Mock GPU + DCGM fixture✅ TodayUbuntunvidia.com/gpu from a real kubelet device plugin; a pod requesting one GPU becomes Running and free GPUs drop; a dcgm-exporter-shaped scrape survives a fork. Mock only — not CUDA
JobSet v0.12.0 on mock GPUs✅ TodayUbuntu2-worker JobSet rendezvous over cluster DNS and Completes; in-flight + Complete survive a fork. Not CUDA / not DeepSpeed
Kubeflow Trainer v2.2.1 (TrainJob) on mock GPUs✅ TodayUbuntu2-worker TrainJob stays Running across a fork; a child-only TrainJob stays invisible on the source. Not a PyTorch runtime
Kueue v0.19.2 on mock GPUs✅ TodayUbuntuA ClusterQueue with GPU quota 1 admits one Job and holds the second Pending; the split survives a fork. Not a batch-scheduler claim
Configurable fake GPU count (0.8.0)✅ TodayUbuntuFAKE_GPU_COUNT (default 2). At 8 per node an 8-GPU pod fits and a second stays Pending on fragmented nodes. Not fork-tested. Mock only
Fake GPU Allocate device IDs (0.8.0)✅ TodayUbuntuNVIDIA_VISIBLE_DEVICES carries the kubelet-assigned fake-gpu-N IDs, matching the pod-resources API. Not fork-tested. Not /dev/nvidia*
Per-pod fake DCGM (0.8.0)✅ TodayUbuntuOne DCGM_FI_DEV_* series per fake GPU, labelled with the owning pod; labels drop on pod delete; the scrape survives a fork. Not NVML
Fake GPU flavors + topology, Kueue TAS (0.8.0)✅ TodayUbuntuProduct and block/rack labels; Kueue flavors fake-h100 / fake-l40s route Jobs; a required-block TAS Job lands in one block. Not fork-tested. Mock only
KubeRay v1.7.0 on mock GPUs✅ TodayUbuntuCPU head + GPU worker schedule on the mock and survive a fork. Not CUDA
RayJob, CPU Ray 2.47.1 on mock GPUs (0.8.0)✅ TodayUbuntuA RayJob with a 1-GPU worker reaches SUCCEEDED; Ray reports 0.0/1.0 GPU from KubeRay’s injected --num-gpus. Not fork-tested. Not Ray GPU hardware placement
Soft-RoCE (rdma_rxe)✅ TodayUbunturxe0 on guest eth0; rdma link + ibv_devinfo on ≥2 nodes survive a fork; a child-only rxe1 stays invisible on the source. Not NCCL / not GPUDirect
Nested hostNetwork pods❌ Not today—Known failure mode in the guest

Need CoreDNS + Ingress on one CI-guest cluster? Flip the profile flags — the switchboard is configuration, not a redesign. Same for a larger fixture: wiring 20 nodes is configuration + RAM, not a new fork feature. Per-row fork latencies: Benchmarks.

Host RuntimeClass (pods as k7d VMs)#

This is the outer layer: kubectl on the host schedules pods into k7d microVMs via runtimeClassName: k7 (native engine) or k7-fc (stock Firecracker under the jailer — see Backends). Relevant if you also want single-VM sandboxes, not only whole-cluster forks. Rows apply to both RuntimeClasses unless the notes say otherwise.

FeatureStatusNotes
runtimeClassName: k7 (CRI / containerd shim)✅ Today
runtimeClassName: k7-fc (same shim, Firecracker + jailer)✅ TodayConfigPath runtime option; PodOverhead 32Mi / 50m; per-pod opt-in on k7 via k7d.katakate.org/backend: firecracker
kubectl logs / exec / exec -it (PTY)✅ TodayIncl. resize, Ctrl-C, detach
Pod IP, Services, DNS, egress✅ TodayHost CNI dataplane
ConfigMap / Secret / projected / downwardAPI / emptyDir✅ Today
hostPath, local-path PVC, k7d RWO disk volumes✅ Todayk7 only — k7-fc has no virtiofs; hostPath / virtiofs fallback fails Create with a named error. K7D_DENY_VIRTIOFS_FALLBACK=1 makes k7 behave the same
Writable scratch on a host disk; per-container scratch; named persistent data disks✅ Todayk7d.katakate.org/scratch-size, container-scratch-size, data-disks annotations
Memory / CPU limits; multi-container / sidecars✅ Today
Pod securityContext inside the VM (seccomp, masked/readonly paths, device cgroups, caps, noNewPrivileges)✅ TodayPassed through to guest runc unchanged; only apparmorProfile, host cgroupsPath and host namespace paths are stripped
In-guest Docker (k7d.katakate.org/docker=true)✅ TodayAgent-supervised pinned dockerd, overlay2 graph on a per-VM disk, forkable; both engines
Warm VM + whole-cluster fork / snapshot tree✅ TodayThe point of the project
Init containers; natural exit / restartPolicy✅ TodayMulti-container pods
Multi-vCPU guests (cpu: "2"+)✅ TodayvCPU count derived from the pod’s CPU limit; fork/snapshot parity
Pause / resume a pod’s VM✅ Todaypause_vm / resume_vm; CID and memory retained. On k7-fc, resume→exec needs Firecracker v1.16.2+ (v1.16.1 gated vsock RX after a bare resume)
Time warp / dilation / autowarp✅ Todayk7 only — unsupported on k7-fc (named error)
Warm fork keeps the pod network (0.9.0)✅ TodayA forked pod VM gets its own bridge with a TAP in the fork pod’s netns (NAT); fork and source have egress at the same time. Earlier releases forked without a NIC — see Network identity
NetworkPolicy on the pod IP✅ Todayk3s kube-router, same verdicts as runc for egress ipBlocks and ingress podSelector (test_agent_sandbox V6). Other CNIs not measured on the host RuntimeClass
kubectl port-forward✅ TodayPod-netns loopback TCP is forwarded into the guest (post-0.9.0 main, CHALLENGES #301); what agent-sandbox’s SDK tunnel mode uses
agent-sandbox Sandbox / warm pool / claim✅ Todayv1.0.5 on k7 and k7-fc, incl. the Python SDK through the router, port-forward and Docker-in-VM. Kubelet volume updates (projected token rotation) do not reach the guest
hostNetwork, IPv6, arbitrary CSI / RWX🔜 LaterHost RuntimeClass only — in-cluster Cilium NetworkPolicy and Longhorn RWX are the rows above
Cross-node fork🔮 LaterHost-local trees today