K7
K7druntimeClassName: k7 pods boot, and what the ~105 ms cluster-fork clock is measured on. The Ubuntu node is a stock Ubuntu 24.04 root disk with systemd as PID 1 and the distro -virtual kernel, booted by create_vm with no initramfs. It runs the real thing: k3s’ own containerd, Cilium, Tetragon, Longhorn, Kubeflow, KubeRay — and still warm-forks, at the ~1.1 s clock.
Both guests go through the same VMM. The delta is entirely userspace.
Build the node image
guest/ubuntu/build-node-image.sh builds one pristine image. Every input is pinned (Ubuntu cloud-image tarball digest, apt package list, k3s artifacts, the agent binary, the helper scripts) and the output is content-keyed, so an unchanged input set never rebuilds and any change produces a new image rather than a stale hit.
OUT_DIR on a reflink-capable filesystem — the daemon’s /var/lib/k7d/disks XFS is the intended home. The image bakes in k3s (airgap images imported offline), open-iscsi + nfs-common for Longhorn, the k7-agent unit, and 25 VM-useless systemd units masked (each recorded in the manifest with a reason).
Never boot a VM from the pristine image. Reflink a per-VM copy first (cp --reflink=always); the copy is O(1) and shares every extent until written.
The boot contract
Everything an Ubuntu node needs to know arrives on the kernel command line — one image, role decided at boot, nothing to discover:
This is a replacement command line, not an addition to the minimal guest’s: there is no
rdinit=/init and no initramfs. Pass it as VmConfig.cmdline.
Boot a 3-node cluster and root a tree at it
Members of an Ubuntu cluster need different command lines (IP, role) and their own root disk, so create them one by one on a shared bridge, then adopt the set as a cluster tree:kubectl get nodes to show 3 Ready (kubeconfig comes out of the server over the agent’s read_file; ~39–42 s for k3s with flannel). Then:
tree_fork_batch, tree_protect, tree_prune, time warp. The forked child’s root disk is a FICLONE reflink of the source’s (~2500 shared extents), so the fork is CoW on both memory and disk.
kernel must be the Ubuntu vmlinuz; leave initrd out. The daemon never substitutes the busybox pair for a disk-booted guest.
What runs inside, and forks
All of these are proven by an integration test that brings the workload up on the source, warm-forks the cluster, and checks both sides keep working and then diverge independently. Versions are pinned.
Image pulls are the long pole, not the fork: Cilium to 3/3 Ready is ~190 s after Helm, Longhorn ~350 s. The fork of the finished cluster stays under the 1 s ceiling in every row. Full tables: Benchmarks.
Persistent data disks
An Ubuntu node’s root disk is ephemeral like any other VM disk —stop_vm deletes it. Storage nodes need a disk that outlives the VM, so VmConfig.data_disks attaches named ext4 images:
data/<id>.img is created sparse and formatted on first use, reused as-is on every later attach, and only delete_data_disk removes it. A size_bytes that disagrees with the existing image is an error, not a resize. On a warm fork every data disk is reflinked with the memory, so the child’s Longhorn replicas diverge from the parent’s without a byte copy. Each disk costs one virtio-blk slot out of the guest’s device budget.
Guest hardening
The node image ships two AppArmor profiles.k7-agent may create AF_VSOCK — it is the harness’s control plane. Everything else, including every pod (CRI AppArmor is disabled so pods inherit the node profile), runs under k7-deny-vsock: unconfined minus the vsock address family. A rogue workload inside a forked cluster cannot open a channel to the host. Combined with the tree network jail on the host side, a compromised guest is contained to its own bridge.
Related pages
- Cluster mode — adopt, fork, suspend, resume as one unit
- Feature matrix — every in-cluster feature and which guest proves it
- Daemon API —
VmConfig.cmdline,block_devices,data_disks