Ubuntu nodes

k7d ships two guests. The minimal CI guest — a busybox initramfs with the k7-agent as PID 1 — is what the release tarball installs, what runtimeClassName: k7 pods boot, and what the ~105 ms cluster-fork clock is measured on. The Ubuntu node is a stock Ubuntu 24.04 root disk with systemd as PID 1 and the distro -virtual kernel, booted by create_vm with no initramfs. It runs the real thing: k3s’ own containerd, Cilium, Tetragon, Longhorn, Kubeflow, KubeRay — and still warm-forks, at the ~1.1 s clock. Both guests go through the same VMM. The delta is entirely userspace.

	Minimal CI guest	Ubuntu node
Init	k7-agent (PID 1)	systemd; k7-agent as a unit
Kernel	k7d’s own minimal `vmlinux`	Ubuntu `linux-image-virtual` (+ `extra` modules for haltpoll, Cilium, Longhorn)
Root	initramfs	sparse ext4 image on the reflink XFS, `root=/dev/vda`
Default size	256 MiB / 1 vCPU	2 GiB / 2 vCPU (Longhorn: 4 GiB / 4 vCPU)
Boot → agent	~163–205 ms	~3.2 s
3-node k3s fork	~105 ms (1 s budget)	~1.1 s agent path; 420–700 ms in isolated tests (1 s ceiling)
Host RAM per k3s node	~3.2 GiB (fixture)	~1.1–1.4 GiB

Build the node image

guest/ubuntu/build-node-image.sh builds one pristine image. Every input is pinned (Ubuntu cloud-image tarball digest, apt package list, k3s artifacts, the agent binary, the helper scripts) and the output is content-keyed, so an unchanged input set never rebuilds and any change produces a new image rather than a stale hit.

sudo OUT_DIR=/var/lib/k7d/disks/base-images guest/ubuntu/build-node-image.sh
# → ubuntu-24.04-<key>.img      ext4 root disk, sparse, pristine
#   ubuntu-24.04-<key>.vmlinuz  the kernel copied out of that disk
#   ubuntu-24.04-<key>.json     manifest: digests, kernel, masked units and why

Requires root (loop mount + chroot), network for apt, and an OUT_DIR on a reflink-capable filesystem — the daemon’s /var/lib/k7d/disks XFS is the intended home. The image bakes in k3s (airgap images imported offline), open-iscsi + nfs-common for Longhorn, the k7-agent unit, and 25 VM-useless systemd units masked (each recorded in the manifest with a reason). Never boot a VM from the pristine image. Reflink a per-VM copy first (cp --reflink=always); the copy is O(1) and shares every extent until written.

The boot contract

Everything an Ubuntu node needs to know arrives on the kernel command line — one image, role decided at boot, nothing to discover:

console=ttyS0 noapic reboot=k panic=1 pci=off net.ifnames=0
root=/dev/vda rootfstype=ext4 rw mitigations=off
clocksource=kvm-clock cpuidle_haltpoll.force=1 haltpoll.guest_halt_poll_ns=200000 …
k7d.ip=10.200.0.2/24,10.200.0.1,eth0
k7d.k3s=server,10.200.0.2,<token>
k7d.cni=none

Token	Meaning
`k7d.ip=<addr>/<prefix>,<gateway>,<iface>`	Static networking, applied by a oneshot unit before k3s
`k7d.k3s=server\|agent,<server-ip>,<token>`	Which k3s unit to start and how to join. Agents wait for the server’s `/readyz` before joining. Absent = k3s stays disabled
`k7d.cni=flannel\|none`	`none` turns off flannel, kube-proxy, and k3s’ network-policy controller so Cilium owns the datapath. Absent = flannel. Any other value fails the boot loudly

This is a replacement command line, not an addition to the minimal guest’s: there is no rdinit=/init and no initramfs. Pass it as VmConfig.cmdline.

Boot a 3-node cluster and root a tree at it

Members of an Ubuntu cluster need different command lines (IP, role) and their own root disk, so create them one by one on a shared bridge, then adopt the set as a cluster tree:

{"op": "create_vm", "config": {
  "kernel": "/var/lib/k7d/disks/base-images/ubuntu-24.04-<key>.vmlinuz",
  "memory_mb": 2048, "vcpus": 2,
  "cmdline": "… root=/dev/vda … k7d.ip=10.200.0.2/24,10.200.0.1,eth0 k7d.k3s=server,10.200.0.2,<token>",
  "block_devices": [{"host_path": "/var/lib/k7d/disks/ubuntu-t1-0.img", "readonly": false}],
  "network": {"cluster_id": "t1", "vm_index": 0}
}}
{"op": "create_vm", "config": {"…": "vm_index 1, k7d.ip=10.200.0.3/24,…, k7d.k3s=agent,10.200.0.2,<token>"}}
{"op": "create_vm", "config": {"…": "vm_index 2, k7d.ip=10.200.0.4/24,…"}}

Wait for the agents over vsock, then for kubectl get nodes to show 3 Ready (kubeconfig comes out of the server over the agent’s read_file; ~39–42 s for k3s with flannel). Then:

{"op": "tree_adopt_cluster", "vm_ids": ["vm-…", "vm-…", "vm-…"], "tree_id": "t1", "budget": {"…": "…"}}

Adoption writes the root keyframe (a full memory copy, the only one the tree ever needs; children take O(divergence) reflink keyframes). On 0.7.0 that froze each member in turn, ~8 s for 3 × 2 GiB. In 0.8.0 it is a coordinated, pre-copied cut and each member pauses for tens of milliseconds. After adoption, tear the cluster down with tree_drop / tree_prune, not stop_vm on the old ids, and delete your root-disk images yourself afterwards. From here it is an ordinary cluster tree: tree_fork_batch, tree_protect, tree_prune, time warp. The forked child’s root disk is a FICLONE reflink of the source’s (~2500 shared extents), so the fork is CoW on both memory and disk. kernel must be the Ubuntu vmlinuz; leave initrd out. The daemon never substitutes the busybox pair for a disk-booted guest.

What runs inside, and forks

All of these are proven by an integration test that brings the workload up on the source, warm-forks the cluster, and checks both sides keep working and then diverge independently. Versions are pinned.

Workload	How	Fork (3 × 2 GiB unless noted)
k3s `v1.32.5+k3s1`, flannel	baked in the image, `k7d.k3s=`	488 ms
Cilium 1.16.10 (eBPF datapath, kube-proxy replacement, CiliumNetworkPolicy)	`k7d.cni=none` + host `helm install`	416 ms
Tetragon 1.7.0 (fork-and-diff of the event stream)	host Helm on the Cilium cluster	453–551 ms
Longhorn 1.12.1 (RWO + RWX PVCs)	host Helm; 4 GiB / 4 vCPU nodes; a named `data_disks` entry per node at `/var/lib/longhorn`	538–696 ms (3 root + 3 data disks)
Mock GPU + DCGM (`nvidia.com/gpu` from a real kubelet device plugin, dcgm-exporter-shaped scrape)	manifests	458 ms
JobSet v0.12.0 on mock GPUs	digest-pinned	298 ms
Kubeflow Trainer v2.2.1 (TrainJob) on mock GPUs	digest-pinned	under 1 s
Kueue v0.19.2 on mock GPUs (quota admits one, holds the second)	digest-pinned	under 1 s
KubeRay v1.7.0 (CPU head + GPU worker)	digest-pinned	580 ms
Per-pod fake DCGM (one `DCGM_FI_DEV_*` series per `fake-gpu-N`, labelled with the owning `pod` / `namespace` / `container`) (0.8.0)	fixture DaemonSet	scrape survives the fork
Soft-RoCE (`rdma_rxe` on guest `eth0`, `rdma link` + `ibv_devinfo` on ≥2 nodes)	in-guest	540 ms
Redis 7 (AOF on) with an `INCR` loop mid-write on another node	manifests	575 ms client / 568 ms daemon; the child continues from the in-flight write
Held apiserver `watch` + ClusterIP TCP stream through the fork	in-guest	streams survive on source and child, then diverge

The ML rows are mocks — a fake device plugin and a fake metrics endpoint. They prove the operators and the schedulers fork; they are not CUDA, not NCCL, not GPUDirect, not a PyTorch runtime.

Fake-GPU fixture knobs

The fake device plugin now looks enough like a GPU node for scheduler and operator tests. All of it is still a mock with no /dev/nvidia*. Only the per-pod DCGM test also warm-forks the cluster; the others prove the feature on a live cluster without forking it:

Knob	What it does	Test
`FAKE_GPU_COUNT` (default 2)	GPUs advertised per node. At 8 per node, an 8-GPU pod fits and a second stays Pending once the leftover nodes are 1-GPU fragments	`test_ubuntu_ml_gpu_count_eight_fits_and_fragments`
`Allocate` device IDs	Sets `NVIDIA_VISIBLE_DEVICES` to the kubelet-assigned `fake-gpu-N` IDs, matching the kubelet pod-resources API	`test_ubuntu_ml_gpu_allocate_two_ids_match_pod_resources`
`FAKE_DCGM_GPU_UTIL` / `FAKE_DCGM_FB_USED` (defaults 42 / 1024), or pod annotations `k7d.katakate.org/fake-dcgm-gpu-util` / `fake-dcgm-fb-used`	Values of the per-pod `DCGM_FI_DEV_*` series. The labels drop when the pod is deleted	`test_ubuntu_ml_dcgm_per_pod`
`FAKE_GPU_PRODUCT`, topology env, or a per-node `fake-gpu-identity` ConfigMap	Product and block/rack node labels. Kueue ResourceFlavors `fake-h100` / `fake-l40s` route Jobs to matching nodes, and a required-block Topology-Aware Scheduling Job lands in one fake block	`test_ubuntu_ml_gpu_flavors`

A RayJob on digest-pinned CPU rayproject/ray:2.47.1-py311 (8 GiB / 4 vCPU nodes) with a worker requesting nvidia.com/gpu: 1 reaches SUCCEEDED (test_ubuntu_ml_rayjob_gpu_succeeded). ray status shows 0.0/1.0 GPU even without rayStartParams.num-gpus, because KubeRay injects --num-gpus from the nvidia.com/gpu request. Ray sees a count, not a device. Image pulls are the long pole, not the fork: Cilium to 3/3 Ready is ~190 s after Helm, Longhorn ~350 s. The fork of the finished cluster stays under the 1 s ceiling in every row. Full tables: Benchmarks.

Surviving a daemon crash

The on-disk tree outlives the daemon that forked it. If k7d is killed with SIGKILL, live KVM state dies with the process, but {K7D_TREE_ROOT}/{tree_id}/tree.json and every keyframe stay on disk: after systemctl start k7d, tree_open re-opens the tree with every node Suspended, tree_resume root brings the cluster back Ready from its last point, and tree_fork_batch forks it again — no source re-boot, no image pull. This is operational rehydrate, not live-KVM resurrection; a member’s in-memory state since its last keyframe is lost.

Persistent data disks

An Ubuntu node’s root disk is a throwaway per-VM clone. It is a caller-supplied block_devices image, so k7d never unlinks it, and the caller deletes it along with the VM or tree. Storage nodes need a disk the daemon keeps across VMs, so VmConfig.data_disks attaches named ext4 images:

"data_disks": [{"id": "longhorn-0", "size_bytes": 17179869184, "guest_mount": "/var/lib/longhorn"}]

The image data/<id>.img is created sparse and formatted on first use, reused as-is on every later attach, and only delete_data_disk removes it. A size_bytes that disagrees with the existing image is an error, not a resize. On a warm fork every data disk is reflinked with the memory, so the child’s Longhorn replicas diverge from the parent’s without a byte copy. Each disk costs one virtio-blk slot out of the guest’s device budget.

Guest hardening

The node image ships two AppArmor profiles. k7-agent may create AF_VSOCK — it is the harness’s control plane. Everything else, including every pod (CRI AppArmor is disabled so pods inherit the node profile), runs under k7-deny-vsock: unconfined minus the vsock address family. A rogue workload inside a forked cluster cannot open a channel to the host. Combined with the tree network jail on the host side, a compromised guest is contained to its own bridge.

Cluster mode — adopt, fork, suspend, resume as one unit
Feature matrix — every in-cluster feature and which guest proves it
Daemon API — VmConfig.cmdline, block_devices, data_disks

Getting started

Concepts

Guides

API

Deep dives

Build the node image

The boot contract

Boot a 3-node cluster and root a tree at it

What runs inside, and forks

Fake-GPU fixture knobs

Surviving a daemon crash

Persistent data disks

Guest hardening

​Build the node image

​The boot contract

​Boot a 3-node cluster and root a tree at it

​What runs inside, and forks

​Fake-GPU fixture knobs

​Surviving a daemon crash

​Persistent data disks

​Guest hardening

​Related pages

Build the node image

The boot contract

Boot a 3-node cluster and root a tree at it

What runs inside, and forks

Fake-GPU fixture knobs

Surviving a daemon crash

Persistent data disks

Guest hardening

Related pages