Skip to main content
You do not need to be a VMM engineer to use k7d. You do need to know why a millisecond-scale fork of a running VM is even possible, because that is the product.

​
Memory is shared until someone writes

Guest RAM lives in one file. A fork pauses the source for a moment, notes which pages changed since the last checkpoint, maps the child’s memory as a copy-on-write view of the parent’s, and copies only those dirty pages. Everything else is shared. That is why 50 forks of a cluster fit in 64 GB: you pay for divergence, not for the base. The two kernel mechanisms underneath:
  • mmap(MAP_PRIVATE) restore. A naïve restore reads the whole memory snapshot into a buffer — a 256 MB memcpy, ~50–100 ms. Instead, k7d maps the snapshot file MAP_PRIVATE: the kernel maps file pages into the new VM’s address space without copying. Reads come straight from the page cache; the first guest write to a page triggers a copy-on-write fault. Restore time becomes O(1) — ~3 ms regardless of memory size.
  • KVM dirty-page tracking. KVM_MEM_LOG_DIRTY_PAGES makes KVM write-protect every guest page and record first writes in a bitmap (one bit per 4 KiB). This is how a warm fork knows which pages to copy from the source.

​
Cold fork vs warm fork

Cold fork (~250 ms): snapshot the source to disk (pause, capture CPU + device state, dump memory), then restore into a new VM with MAP_PRIVATE memory. Bottlenecked by writing guest RAM to disk. Warm fork (~5 ms) avoids the disk write:
  1. One-time prep: cold-snapshot once → restore once → enable KVM dirty tracking. Now the base memory image lives on disk and the source VM is “fork-ready”.
  2. Per fork: pause the source for ~1 ms → read the dirty bitmap (only pages dirtied since the last fork) → memcpy those pages into the new fork’s MAP_PRIVATE mapping of the base image → capture CPU/device state in memory → resume the source.
The only work proportional to dirty pages is the memcpy (~0.5 µs per page — ≤15 ms even at 100% dirty). Everything else is O(1). Batch forks (tree_fork_batch) share a single pause + dirty-bitmap capture across N forks. Disk state is copied with filesystem reflinks (FICLONE on the daemon’s XFS): a “copy” that shares all data extents copy-on-write, O(1) regardless of size. Root disks, scratch disks, and named data disks all fork this way — a forked Ubuntu node’s root shares ~2500 extents with its parent until it writes. The base image a child’s MAP_PRIVATE mapping points at is its source’s keyframe. A root’s keyframe is a full dump. A child’s is a reflink clone of the nearest ancestor’s keyframe with only that child’s dirty pages written over it — O(divergence) rather than O(guest RAM), ~0.8 s instead of ~7.8 s for an idle 3 × 2 GiB cluster. That is what makes deep trees affordable: every fork point below the root costs what the node changed, not what the guest is. Details and the measured divergence levels: Snapshot tree — keyframes.

​
What had to be right

These are the kinds of bugs that silently break “byte-identical”. The project’s CHALLENGES.md catalogs nearly 200 of them; four headline ones:
  1. Device writes are invisible to the hypervisor’s dirty log. Block and network devices in k7d write guest memory from userspace. The hypervisor only sees CPU writes — so a fork taken after disk I/O would resurrect pre-I/O bytes on those pages (silent corruption). Fix: track device dirty pages separately, merge them into the fork bitmap, and drain in-flight I/O before the bitmap is read. The same applies to the kernel’s vhost-vsock DMA: its ring pages are marked by hand, or a child boots from a stale used.idx and its agent never answers.
  2. A restored guest with a blank timer chip freezes time. After fork, the interval timer was left unprogrammed — no timer interrupts, CLOCK_REALTIME stuck, and Kubernetes quietly parks. Fix: re-arm the timer and re-base the paravirtual clock (KVM_SET_CLOCK) on every restore — so a guest resumed hours later reports the right wall-clock time, not 1999, and TLS validation keeps working.
  3. An interrupt captured mid-injection must be re-queued. Pausing a vCPU can land between the interrupt controller ACKing a vector and the VM entry delivering it. Drop that event and the restored guest is alive but permanently deaf on that IRQ line — about 1% of warm forks, before the fix.
  4. Identical IPs only work if the L2 domains are separate. Replay the source’s addresses onto a fresh bridge per fork. Same view from inside; no conflicts across forks. See Network identity.

​
After the fork: reconnecting to the guest

The fork inherits the source’s vsock device state, including any open agent connection — but the fork gets a fresh vsock CID. k7d injects a virtio-vsock TRANSPORT_RESET event into the fork’s device state only: the guest sees its connection drop, the in-guest agent reconnects on the new CID, and the host can talk to the fresh fork. The source keeps its CID across the prepare that made it fork-ready, so kubectl exec against a source pod keeps working. Guest clocks are re-based so time does not jump backwards — and because k7d owns the guest clock end to end, the same pause can also move it forward on purpose. That is Time warp.

​
The Firecracker variant

Everything above describes the native engine, where parent and child memory are mappings in one process. A backend: firecracker VM lives in a separate jailed process, so there is nothing to MAP_PRIVATE: the daemon asks Firecracker for a Diff snapshot (only pages dirtied since the last point), FICLONERANGEs those extents onto a shared base file, reflinks the base once per child, and LoadSnapshots a fresh jailed Firecracker from that reflink with mem_backend: File. The child’s memory is still CoW — every extent reports FIEMAP_EXTENT_SHARED — but the fork costs ~190 ms instead of ~5 ms, mostly in staging the jail. Backends has the mechanics and the latency table.