k7d ships two guests. The minimal CI guest — a busybox initramfs with the k7-agent as PID 1 — is what the release tarball installs, what runtimeClassName: k7 pods boot, and what the ~105 ms cluster-fork clock is measured on. The Ubuntu node is a stock Ubuntu 24.04 root disk with systemd as PID 1 and the distro -virtual kernel, booted by create_vm with no initramfs. It runs the real thing: k3s’ own containerd, Cilium, Tetragon, Longhorn, Kubeflow, KubeRay — and still warm-forks, at the ~1.1 s clock.
Both guests go through the same VMM. The delta is entirely userspace.
| Minimal CI guest | Ubuntu node | |
|---|---|---|
| Init | k7-agent (PID 1) | systemd; k7-agent as a unit |
| Kernel | k7d’s own minimal vmlinux | Ubuntu linux-image-virtual (+ extra modules for haltpoll, Cilium, Longhorn) |
| Root | initramfs | sparse ext4 image on the reflink XFS, root=/dev/vda |
| Default size | 256 MiB / 1 vCPU | 2 GiB / 2 vCPU (Longhorn: 4 GiB / 4 vCPU) |
| Boot → agent | ~163–205 ms | ~3.2 s |
| 3-node k3s fork | ~105 ms (1 s budget) | ~1.1 s agent path; 420–700 ms in isolated tests (1 s ceiling) |
| Host RAM per k3s node | ~3.2 GiB (fixture) | ~1.1–1.4 GiB |
Build the node image#
guest/ubuntu/build-node-image.sh builds one pristine image. Every input is pinned (Ubuntu cloud-image tarball digest, apt package list, k3s artifacts, the agent binary, the helper scripts) and the output is content-keyed, so an unchanged input set never rebuilds and any change produces a new image rather than a stale hit.
Requires root (loop mount + chroot), network for apt, and an OUT_DIR on a reflink-capable filesystem — the daemon’s /var/lib/k7d/disks XFS is the intended home. The image bakes in k3s (airgap images imported offline), open-iscsi + nfs-common for Longhorn, the k7-agent unit, and 25 VM-useless systemd units masked (each recorded in the manifest with a reason).
Never boot a VM from the pristine image. Reflink a per-VM copy first (cp --reflink=always); the copy is O(1) and shares every extent until written.
The boot contract#
Everything an Ubuntu node needs to know arrives on the kernel command line — one image, role decided at boot, nothing to discover:
| Token | Meaning |
|---|---|
k7d.ip=<addr>/<prefix>,<gateway>,<iface> | Static networking, applied by a oneshot unit before k3s |
k7d.k3s=server|agent,<server-ip>,<token> | Which k3s unit to start and how to join. Agents wait for the server’s /readyz before joining. Absent = k3s stays disabled |
k7d.cni=flannel|none | none turns off flannel, kube-proxy, and k3s’ network-policy controller so Cilium owns the datapath. Absent = flannel. Any other value fails the boot loudly |
This is a replacement command line, not an addition to the minimal guest’s: there is no rdinit=/init and no initramfs. Pass it as VmConfig.cmdline.
Boot a 3-node cluster and root a tree at it#
Members of an Ubuntu cluster need different command lines (IP, role) and their own root disk, so create them one by one on a shared bridge, then adopt the set as a cluster tree:
Wait for the agents over vsock, then for kubectl get nodes to show 3 Ready (kubeconfig comes out of the server over the agent’s read_file; ~39–42 s for k3s with flannel). Then:
Adoption writes the root keyframe (a full memory copy, the only one the tree ever needs; children take O(divergence) reflink keyframes). On 0.7.0 that froze each member in turn, ~8 s for 3 × 2 GiB. In 0.8.0 it is a coordinated, pre-copied cut and each member pauses for tens of milliseconds. After adoption, tear the cluster down with tree_drop / tree_prune, not stop_vm on the old ids, and delete your root-disk images yourself afterwards. From here it is an ordinary cluster tree: tree_fork_batch, tree_protect, tree_prune, time warp. The forked child’s root disk is a FICLONE reflink of the source’s (~2500 shared extents), so the fork is CoW on both memory and disk.
kernel must be the Ubuntu vmlinuz; leave initrd out. The daemon never substitutes the busybox pair for a disk-booted guest.
What runs inside, and forks#
All of these are proven by an integration test that brings the workload up on the source, warm-forks the cluster, and checks both sides keep working and then diverge independently. Versions are pinned.
| Workload | How | Fork (3 × 2 GiB unless noted) |
|---|---|---|
k3s v1.32.5+k3s1, flannel | baked in the image, k7d.k3s= | 488 ms |
| Cilium 1.16.10 (eBPF datapath, kube-proxy replacement, CiliumNetworkPolicy) | k7d.cni=none + host helm install | 416 ms |
| Tetragon 1.7.0 (fork-and-diff of the event stream) | host Helm on the Cilium cluster | 453–551 ms |
| Longhorn 1.12.1 (RWO + RWX PVCs) | host Helm; 4 GiB / 4 vCPU nodes; a named data_disks entry per node at /var/lib/longhorn | 538–696 ms (3 root + 3 data disks) |
Mock GPU + DCGM (nvidia.com/gpu from a real kubelet device plugin, dcgm-exporter-shaped scrape) | manifests | 458 ms |
| JobSet v0.12.0 on mock GPUs | digest-pinned | 298 ms |
| Kubeflow Trainer v2.2.1 (TrainJob) on mock GPUs | digest-pinned | under 1 s |
| Kueue v0.19.2 on mock GPUs (quota admits one, holds the second) | digest-pinned | under 1 s |
| KubeRay v1.7.0 (CPU head + GPU worker) | digest-pinned | 580 ms |
Per-pod fake DCGM (one DCGM_FI_DEV_* series per fake-gpu-N, labelled with the owning pod / namespace / container) (0.8.0) | fixture DaemonSet | scrape survives the fork |
Soft-RoCE (rdma_rxe on guest eth0, rdma link + ibv_devinfo on ≥2 nodes) | in-guest | 540 ms |
Redis 7 (AOF on) with an INCR loop mid-write on another node | manifests | 575 ms client / 568 ms daemon; the child continues from the in-flight write |
Held apiserver watch + ClusterIP TCP stream through the fork | in-guest | streams survive on source and child, then diverge |
The ML rows are mocks — a fake device plugin and a fake metrics endpoint. They prove the operators and the schedulers fork; they are not CUDA, not NCCL, not GPUDirect, not a PyTorch runtime.
Fake-GPU fixture knobs#
The fake device plugin now looks enough like a GPU node for scheduler and operator tests. All of it is still a mock with no /dev/nvidia*. Only the per-pod DCGM test also warm-forks the cluster; the others prove the feature on a live cluster without forking it:
| Knob | What it does | Test |
|---|---|---|
FAKE_GPU_COUNT (default 2) | GPUs advertised per node. At 8 per node, an 8-GPU pod fits and a second stays Pending once the leftover nodes are 1-GPU fragments | test_ubuntu_ml_gpu_count_eight_fits_and_fragments |
Allocate device IDs | Sets NVIDIA_VISIBLE_DEVICES to the kubelet-assigned fake-gpu-N IDs, matching the kubelet pod-resources API | test_ubuntu_ml_gpu_allocate_two_ids_match_pod_resources |
FAKE_DCGM_GPU_UTIL / FAKE_DCGM_FB_USED (defaults 42 / 1024), or pod annotations k7d.katakate.org/fake-dcgm-gpu-util / fake-dcgm-fb-used | Values of the per-pod DCGM_FI_DEV_* series. The labels drop when the pod is deleted | test_ubuntu_ml_dcgm_per_pod |
FAKE_GPU_PRODUCT, topology env, or a per-node fake-gpu-identity ConfigMap | Product and block/rack node labels. Kueue ResourceFlavors fake-h100 / fake-l40s route Jobs to matching nodes, and a required-block Topology-Aware Scheduling Job lands in one fake block | test_ubuntu_ml_gpu_flavors |
A RayJob on digest-pinned CPU rayproject/ray:2.47.1-py311 (8 GiB / 4 vCPU nodes) with a worker requesting nvidia.com/gpu: 1 reaches SUCCEEDED (test_ubuntu_ml_rayjob_gpu_succeeded). ray status shows 0.0/1.0 GPU even without rayStartParams.num-gpus, because KubeRay injects --num-gpus from the nvidia.com/gpu request. Ray sees a count, not a device.
Image pulls are the long pole, not the fork: Cilium to 3/3 Ready is ~190 s after Helm, Longhorn ~350 s. The fork of the finished cluster stays under the 1 s ceiling in every row. Full tables: Benchmarks.
Surviving a daemon crash#
The on-disk tree outlives the daemon that forked it. If k7d is killed with SIGKILL, live KVM state dies with the process, but {K7D_TREE_ROOT}/{tree_id}/tree.json and every keyframe stay on disk: after systemctl start k7d, tree_open re-opens the tree with every node Suspended, tree_resume root brings the cluster back Ready from its last point, and tree_fork_batch forks it again — no source re-boot, no image pull. This is operational rehydrate, not live-KVM resurrection; a member’s in-memory state since its last keyframe is lost.
Persistent data disks#
An Ubuntu node’s root disk is a throwaway per-VM clone. It is a caller-supplied block_devices image, so k7d never unlinks it, and the caller deletes it along with the VM or tree. Storage nodes need a disk the daemon keeps across VMs, so VmConfig.data_disks attaches named ext4 images:
The image data/<id>.img is created sparse and formatted on first use, reused as-is on every later attach, and only delete_data_disk removes it. A size_bytes that disagrees with the existing image is an error, not a resize. On a warm fork every data disk is reflinked with the memory, so the child’s Longhorn replicas diverge from the parent’s without a byte copy. Each disk costs one virtio-blk slot out of the guest’s device budget.
Guest hardening#
The node image ships two AppArmor profiles. k7-agent may create AF_VSOCK — it is the harness’s control plane. Everything else, including every pod (CRI AppArmor is disabled so pods inherit the node profile), runs under k7-deny-vsock: unconfined minus the vsock address family. A rogue workload inside a forked cluster cannot open a channel to the host. Combined with the tree network jail on the host side, a compromised guest is contained to its own bridge.
Related pages#
- Cluster mode — adopt, fork, suspend, resume as one unit
- Feature matrix — every in-cluster feature and which guest proves it
- Daemon API —
VmConfig.cmdline,block_devices,data_disks