Skip to main content

You do not need to be a VMM engineer to use k7d. You do need to know why a millisecond-scale fork of a running VM is even possible, because that is the product.

Memory is shared until someone writes#

Guest RAM lives in one file. A fork pauses the source for a moment, notes which pages changed since the last checkpoint, maps the child’s memory as a copy-on-write view of the parent’s, and copies only those dirty pages. Everything else is shared. That is why 50 forks of a cluster fit in 64 GB: you pay for divergence, not for the base.

The two kernel mechanisms underneath:

  • mmap(MAP_PRIVATE) restore. A naïve restore reads the whole memory snapshot into a buffer — a 256 MB memcpy, ~50–100 ms. Instead, k7d maps the snapshot file MAP_PRIVATE: the kernel maps file pages into the new VM’s address space without copying. Reads come straight from the page cache; the first guest write to a page triggers a copy-on-write fault. Restore time becomes O(1) — ~3 ms regardless of memory size.
  • KVM dirty-page tracking. KVM_MEM_LOG_DIRTY_PAGES makes KVM write-protect every guest page and record first writes in a bitmap (one bit per 4 KiB). This is how a warm fork knows which pages to copy from the source.

Cold fork vs warm fork#

Cold fork (~250 ms): snapshot the source to disk (pause, capture CPU + device state, dump memory), then restore into a new VM with MAP_PRIVATE memory. Bottlenecked by writing guest RAM to disk.

Warm fork (~5 ms) avoids the disk write:

  1. One-time prep: cold-snapshot once → restore once → enable KVM dirty tracking. Now the base memory image lives on disk and the source VM is “fork-ready”.
  2. Per fork: pause the source for ~1 ms → read the dirty bitmap (only pages dirtied since the last fork) → memcpy those pages into the new fork’s MAP_PRIVATE mapping of the base image → capture CPU/device state in memory → resume the source.

The only work proportional to dirty pages is the memcpy (~0.5 µs per page — ≤15 ms even at 100% dirty). Everything else is O(1). Batch forks (tree_fork_batch) share a single pause + dirty-bitmap capture across N forks.

Disk state is copied with filesystem reflinks (FICLONE on the daemon’s XFS): a “copy” that shares all data extents copy-on-write, O(1) regardless of size. Root disks, scratch disks, and named data disks all fork this way — a forked Ubuntu node’s root shares ~2500 extents with its parent until it writes.

The base image a child’s MAP_PRIVATE mapping points at is its source’s keyframe. A root’s keyframe is a full dump. A child’s is a reflink clone of the nearest ancestor’s keyframe with only that child’s dirty pages written over it: O(divergence) rather than O(guest RAM), ~0.8 s instead of ~7.8 s for an idle 3 × 2 GiB cluster on 0.7.0. That is what makes deep trees affordable: every fork point below the root costs what the node changed, not what the guest is. Details and the measured divergence levels: Snapshot tree — keyframes.

In 0.8.0, a native cluster’s keyframes are also pre-copied while the members keep running. Dirty-log rounds copy what changed, and the members are then paused together for a short cut that writes only the residual. The guest’s pause no longer grows with its RAM: 44 ms for a 30 GiB member, against 48.9 s for a paused full dump. This uses the same dirty tracking as the warm fork, applied to a guest that is still running. See Snapshot tree — how a cluster keyframe is cut.

What had to be right#

These are the kinds of bugs that silently break “byte-identical”. The project’s CHALLENGES.md catalogs nearly 200 of them; four headline ones:

  1. Device writes are invisible to the hypervisor’s dirty log. Block and network devices in k7d write guest memory from userspace. The hypervisor only sees CPU writes — so a fork taken after disk I/O would resurrect pre-I/O bytes on those pages (silent corruption). Fix: track device dirty pages separately, merge them into the fork bitmap, and drain in-flight I/O before the bitmap is read. The same applies to the kernel’s vhost-vsock DMA: its ring pages are marked by hand, or a child boots from a stale used.idx and its agent never answers.
  2. A restored guest with a blank timer chip freezes time. After fork, the interval timer was left unprogrammed — no timer interrupts, CLOCK_REALTIME stuck, and Kubernetes quietly parks. Fix: re-arm the timer and re-base the paravirtual clock (KVM_SET_CLOCK) on every restore — so a guest resumed hours later reports the right wall-clock time, not 1999, and TLS validation keeps working.
  3. An interrupt captured mid-injection must be re-queued. Pausing a vCPU can land between the interrupt controller ACKing a vector and the VM entry delivering it. Drop that event and the restored guest is alive but permanently deaf on that IRQ line — about 1% of warm forks, before the fix.
  4. Identical IPs only work if the L2 domains are separate. Replay the source’s addresses onto a fresh bridge per fork. Same view from inside; no conflicts across forks. See Network identity.

After the fork: reconnecting to the guest#

The fork inherits the source’s vsock device state, including any open agent connection — but the fork gets a fresh vsock CID. k7d injects a virtio-vsock TRANSPORT_RESET event into the fork’s device state only: the guest sees its connection drop, the in-guest agent reconnects on the new CID, and the host can talk to the fresh fork. The source keeps its CID across the prepare that made it fork-ready, so kubectl exec against a source pod keeps working.

Connections opened to the source before that prepare do not error. They go silent, because a native vsock read does not treat the swapped backend’s EOF as fatal. A client holding one has to bound its reads and redial the same CID. The shim does this itself in 0.8.0: its log bridge caps each read at 1 s and closes the container’s stdio FIFOs when it is torn down. Without that, a fork source pod could take the full 30 s agent deadline to delete.

Guest clocks are re-based so time does not jump backwards — and because k7d owns the guest clock end to end, the same pause can also move it forward on purpose. That is Time warp.

The Firecracker variant#

Everything above describes the native engine, where parent and child memory are mappings in one process. A backend: firecracker VM lives in a separate jailed process, so there is nothing to MAP_PRIVATE: the daemon asks Firecracker for a Diff snapshot (only pages dirtied since the last point), FICLONERANGEs those extents onto a shared base file, reflinks the base once per child, and LoadSnapshots a fresh jailed Firecracker from that reflink with mem_backend: File. The child’s memory is still CoW — every extent reports FIEMAP_EXTENT_SHARED — but the fork costs ~190 ms instead of ~5 ms, mostly in staging the jail. Backends has the mechanics and the latency table.