Cluster mode

Cluster mode is what makes k7d different from every sandbox runtime: the unit of fork is not a VM but a set of VMs sharing a network — typically a k3s control plane plus agents. A cluster fork copies every member VM and replays their network identities onto a fresh bridge, so the copied cluster keeps running without a restart. See Network identity for why that works.

Two ways to get a cluster tree#

1. Create one from scratch — tree_create_cluster#

{"op": "tree_create_cluster", "vm_count": 3, "config": {"memory_mb": 1024}}

The daemon boots vm_count member VMs on a tree-owned bridge and roots a cluster snapshot tree at them. The fork engine is N-node — there is no hard-coded 3. The per-member config is a regular VmConfig (its network field must be omitted: the tree creates and owns its own cluster network). Every member is booted from the same config, which is what k7d quickstart and the busybox demo use.

Practical sizing: the minimal CI guest costs ~3.2 GiB of host RAM per k3s node, an Ubuntu node ~1.1–1.4 GiB. The limit is host RAM, not the API — a 20-node cluster forks the same way the 3-node one does.

2. Adopt live VMs — tree_adopt_cluster#

{"op": "tree_adopt_cluster", "vm_ids": ["vm-1", "vm-2", "vm-3"], "tree_id": "t1"}

Takes VMs the daemon already owns and converts them into a cluster tree root. This is the path when members need different configs — an Ubuntu k3s server and its agents with their own command lines and root disks, or pods deployed via RuntimeClass k7 with a shared cluster-id annotation. The VMs must already share one cluster bridge — same cluster_id, distinct vm_indexes covering 0..N-1.

Adoption fails loud on any half-wired state (missing VM, mixed bridges, non-contiguous indices, already tree-owned). Validation failures put the VMs back untouched; a failure after extraction tears everything down — no half-adopted state survives. Adoption also writes the root’s keyframe: a full memory copy, and the only full copy the tree ever needs. In 0.8.0 it is one coordinated cut, pre-copied while the members run, and tree_created.pause_ns reports what each member lost (tens of milliseconds, even for a 30 GiB member).

After adoption the tree owns the VMs. stop_vm on their old ids errors in 0.8.0 (was adopted into tree …; use tree_prune / tree_drop), and they disappear from list_vms. Tear down with tree_drop or by pruning the root, then delete any block_devices images you supplied yourself. Details: Who owns an adopted VM.

Egress posture#

Both create ops also take ksm: true (0.8.0) to let ksmd merge identical guest pages across the tree’s members and forks. It is for same-tenant hosts only; see Snapshot tree — KSM.

Both create ops take an optional egress policy — open (default), deny, or whitelist with CIDR:port pairs — enforced by the per-tree nftables jail. Forks inherit their source’s policy. Full shape: EgressPolicy; what the jail contains: Security model.

Forking, suspending, resuming a cluster#

Once rooted, a cluster tree uses the same verbs as a single-VM tree — every operation applies to all members as one unit:

{"op": "tree_fork", "tree_id": "t1", "source_id": "root", "label": "branch-a"}{"op": "tree_fork_batch", "tree_id": "t1", "source_id": "root", "labels": ["b0", "b1", "b2", "b3"]}{"op": "tree_suspend", "tree_id": "t1", "node_id": "branch-a"}{"op": "tree_resume", "tree_id": "t1", "node_id": "branch-a"}

Responses carry one guest_cids entry per member (ordered by vm_index), so your harness can dial each member’s agent. Keep them: exec is vsock straight to a CID, and tree_nodes.live_guest_cids is the only other place to find them. They also carry latency_ns — the daemon’s own wall time for the fork, which is what the budgets are asserted on. Batch cluster forks run sequential warm forks — each branch gets its own bridge.

On the Firecracker engine (backend: firecracker) the same verbs apply: members are paused in parallel (skew p99 ≈ 125 µs), each takes a Diff snapshot onto a reflinked base, and every child is a fresh jailed Firecracker restored from its own reflink — ~0.67 s for a 3-member busybox cluster in a debug build, versus ~105 ms native. Suspend exports the point out of the jail; resume re-stages a jail and restores with the member’s network identity. See Backends.

The fork_memory_full_copy flags in fork responses tell you whether each member forked CoW (false, the expected fast path) or had to fall back to a full memory copy (true — happens only for virtiofs-backed sources, e.g. pods with read-write hostPath mounts).

Forking from a child#

Any live node is a fork source, so trees go as deep as the search does. The first fork from a child writes that child’s keyframe — a reflink clone of the nearest ancestor’s memory image plus only the pages the child dirtied, so a mid-tree fork point costs O(divergence), not O(guest RAM): ~0.8 s for an idle 3 × 2 GiB Ubuntu cluster versus ~7.8 s for the root’s full dump on 0.7.0. tree_prepare_fork_source writes it explicitly if you want to show the checkpoint as its own step. In 0.8.0, refresh: true re-keyframes a long-lived root into a new generation, so later forks stop re-copying its accumulated dirty set. Children already mapping the old generation are unaffected. See Snapshot tree — keyframes.

Forking at a different clock#

tree_fork / tree_fork_batch accept dilation: the children run at that factor while the source keeps its own. One ×1 source tree can serve ×8 exploration branches and ×1 control branches at once. In 0.8.0 they also accept dilation_policy (fixed / down / track), so the daemon lowers a branch’s factor when its vCPUs saturate. See Time warp.

What the numbers look like#

Methodology and reproduction commands: Benchmarks. What works inside a forked cluster today (CoreDNS, Ingress, Cilium, Longhorn, …): Feature matrix.