Benchmarks

Reproducible numbers for the claim: warm-fork a live multi-VM Kubernetes cluster in well under a second on the minimal guest and in about a second on a stock Ubuntu distro cluster, with TLS sessions surviving and the source cluster still answering.

Every number below is an integration-test assertion, not a one-off benchmark paste. The budget constants live in crates/k7d/src/latency.rs (the single source of truth, imported by the tests) and are documented in LATENCY_BUDGETS.md. If a number here drifts from those files, trust the tests. Budgets are set at roughly 2× the typical observed value.

Cluster-tree fork budgets are asserted on the daemon’s own latency_ns, not on the client’s round trip around the Unix socket. Tests print both; a client RTT over budget while the daemon stays under is a finding, not a failure.

Hardware#

One pinned bare-metal node (~€40/month Hetzner) used for all figures:

Two clocks#

Headline results — minimal CI guest#

Inner-k3s add-on profiles#

Same 3-VM fixture with one stock add-on switched on, plus load churn. Deltas are against the ~104 ms lean baseline. All share the 1 s budget except ArgoCD, which gets its own 2 s.

Headline results — Ubuntu nodes#

Stock Ubuntu 24.04 root disk, systemd, distro kernel; 3 VMs at 2 GiB / 2 vCPU unless noted. Boot ceilings are loose hang detectors (600 s), not 2× typical. Every fork row reuses one 1 s ceiling, which is also the sub-second target; missing it is a recorded finding, not a silent pass.

Six CoW disks and twice the guest RAM (Longhorn) still fork inside the ceiling the 2 GiB clusters were measured against: fork cost tracks the number of devices and page tables, not the volume of storage behind them. The pull is the long pole every time, not the fork.

Firecracker engine (k7-fc)#

backend: firecracker trees have no MAP_PRIVATE mapping to fork from: a point is a Firecracker Diff snapshot FICLONERANGEd onto a reflinked base, and every child is a fresh jailed Firecracker LoadSnapshoted from its own reflink. Ceilings (500 ms per child, children answer ping within 500 ms) are asserted in test_fc_tree.rs / test_fc_cluster.rs. Exclusive Hetzner node, 2026-09-08, Firecracker v1.16.1, 256 MiB / 1 vCPU busybox guests:

The point is O(dirty) and the base is O(1); the 50–75 ms per child is staging a chroot, leasing a uid, spawning jailer and waiting for its API socket — what a warm pool of pre-staged jails recovers. Do not measure a child with du: its load.mem reports 256 MiB of st_blocks while owning no extent; the tests check FIEMAP_EXTENT_SHARED instead. PodOverhead for the RuntimeClass (32 MiB / 50m) is derived in OVERHEAD.md. See Backends.

Keyframes: O(divergence)#

A fork point below the root clones the nearest ancestor’s memory.bin with FICLONE and writes only that node’s dirty pages. 3 × 2 GiB Ubuntu k3s, 30 s of divergence each; ceiling 4.5 s (~2× the worst level):

Test: test_keyframe_cost_ubuntu_divergence_levels. The bytes are the daemon’s base_disk_bytes; du and stat cannot measure a reflinked file. Neither the keyframe nor the full dump it replaces is fsynced — syncing a patched reflinked file costs 2–5 s per member because every written page breaks a shared extent.

The table above is 0.7.0, where members were keyframed one after another. In 0.8.0, root and child keyframes are one coordinated cut with parallel member writes. The same test on the same AMD node (2026-09-27): root 2.13 s, children idle 0.50 s, k8s 0.48 s, Kafka 0.78 s (49.8%).

Keyframe pause#

In 0.8.0, members are also pre-copied while running, so what the guest loses is a short residual cut, not the dump. tree_created.pause_ns reports the per-member figure. These are measurements, not budgets:

Turning on dirty logging for a member that does not have it yet is one short separate stop before the cut (the “arm stop”), because it write-protects the memory slots.

Time warp#

An 8.7-minute guest-time episode on a fresh 3-node Ubuntu k3s cluster, three runs per arm (make warp-bench, a Criterion bench kept out of make check for its runtime):

The down and track policies that the adaptive loop in 0.8.0 runs, against two fixed factors, over one scripted 240 s guest-time episode: fixed ×8 32.0 s (8.00×), fixed ×4 60.9 s (4.00×), down 128.4 s (1.91×), track 73.9 s (3.37×), with zero hard failures in every arm. A live factor change takes 140 µs on one VM and 450 µs on 3 nodes.

The dilation N sweep (compression exactly N through 16; extra probe failures 0 / 0 / 0 / 3 / 19 / 32 for N ∈ 16) and the fork-of-a-dilated-VM tests need the carried KVM patch loaded (make dilation-install; utils/kvm-dilation/) and are kept out of make check. See Time warp.

Full budget table (minimal guest)#

Reproduce#

On a node with the repo and toolchain installed (see HACKING.md for the remote loop):

# Full enforced suite (fmt + deny + clippy + all tests, one VM-booting binary at a time):make check                       # on the nodemake remote-check                # from anywhere, K7_NODE_IP set# Minimal-guest headline path:cargo test -p k7d --test test_inner_k3s \  test_inner_k3s_cluster_warm_fork_no_restart -- --test-threads=1 --nocapture# Ubuntu headline path (builds the node image on first run):cargo test -p k7d --test test_ubuntu_tetragon \  test_ubuntu_tetragon_fork_event_diff -- --test-threads=1 --nocapture# Scale check — shared-pause batch of 50× 3-VM clusters:cargo test -p k7d --test test_cluster_tree \  test_cluster_tree_fork_batch_50 -- --test-threads=1 --nocapture# Keyframe cost at three divergence levels:cargo test -p k7d --test test_keyframe_cost -- --test-threads=1 --nocapture# Two-arm time-warp benchmark (~10 min guest time per run + bring-up):make warp-bench# Five-minute try path, no toolchain:sudo ./scripts/quickstart.sh

The inner-k3s test prints a line like:

TreeFork node=warm-0000 cids=[…] elapsed=105.062575ms budget=1sdaemon: fork_cluster_warm '…': 3 VMs in 69.703248ms

How the fast path was won#

Before the block-rootfs work, cluster pods carried a virtiofs share for the k3s hostPath — so every fork forced a full memory copy of 3 × ~3.2 GiB, landing at ~2 s typical. Moving read-only hostPath mounts to cached erofs images on virtio-blk (zero virtiofs shares) let every member fork MAP_PRIVATE + dirty-page CoW: ~105 ms under a 1 s budget. The share-less fork prep also preserves the source’s vsock CID, so kubectl exec against the source still works after adopt/fork.

The Ubuntu clock is a different guest — systemd and a full distro rootfs instead of a busybox initramfs — through the same VMM. Its root disk forks as a reflink on the daemon’s XFS (~2500 shared extents per child), so the fork is CoW on disk as well as memory.

Comparison (honest)#

What we are not claiming#

• Firecracker-beating cold boot (we boot once, then fork).
• “<5 MB per instance” density (our guests run kubelet).
• That the virtiofs era (~2 s full-copy) was the intended steady state — it is now the explicit fallback for read-write hostPath only.
• That dilation is free — it is a measured, bounded fidelity cost, and the jump policy exists because it has none.