Skip to main content

The non-obvious half of the sub-second cluster fork claim is networking. Forking the VMs is memory mechanics (CoW fork); keeping Kubernetes running inside them is a network-identity problem.

The cluster does not reboot because the network lies consistently#

Each cluster lives on its own private Linux bridge, in its own network namespace. A fork gets a new bridge (and namespace) with the same guest IPs and MAC addresses as the source. From inside the guest, nothing moved — same addresses, same ARP cache, same TLS certs, same established TCP — so kubelet, the CNI, and the control plane keep running.

Separate bridges mean forks cannot see each other, so the identical addresses never conflict: identical IPs only work because the L2 domains are separate.

Without this trick, every fork would force a kubelet restart and a ~1–2 s agent restart per node, and the ~105 ms / ~1.1 s claims would be impossible. The benchmark suite proves the point by holding a ?watch=true streaming connection to the kube-apiserver across the fork — the TLS session survives on both the source and the copy.

The plumbing#

  • Each VM gets a TAP device (a virtual Ethernet interface: writes to its fd become frames on the host network stack, and vice versa).
  • TAPs attach to a per-network Linux bridge (k7-br-<id>). Since 0.9.0 each bridge lives in its own network namespace, /run/netns/k7-br-<id>, connected to the host through a /30 uplink veth pair addressed from 198.18.0.0/15; NAT happens on that uplink. Before 0.9.0 every bridge sat in the host namespace behind one iptables MASQUERADE rule, so two networks carrying the same guest IP could not both have egress — only one of a fork and its live source could reach outside.
  • Each VM gets a stable IP allocated from the bridge’s subnet, determined by its vm_index (10.200.0.{2 + vm_index}). Guests keep 10.200.0.0/24 whatever the namespace.
  • A fork replays the source’s vm_index → IP/MAC assignments onto its own fresh bridge and namespace, so a fork and its source have egress at the same time.
  • When a pod is CNI-managed (RuntimeClass mode), the TAP is moved into the CNI pod network namespace instead, so the VM answers on the pod IP like any other pod; pod-backed sandboxes never build a k7-br-* namespace. A warm fork of such a pod (fork_vm.pod_netns, 0.9.0) gets its own bridge with a TAP attached into the fork pod’s netns with NAT, so the fork pod’s NetworkPolicy governs it — earlier releases left the fork without a NIC (native) or outside the pod netns (Firecracker).
  • The daemon reaps stale k7-br-* namespaces at startup and in the orphan sweep.

Separate is not enough: the jail#

Separate bridges stop broadcast domains from overlapping, but the host still routes between bridges — before the jail, two trees on one node could reach each other’s guests at L3, because both bridges were explicitly accepted in FORWARD. The per-tree nftables jail closes that: inter-bridge forwarding is dropped, each tree carries an egress posture (open / deny / whitelist) that its forks inherit, connection counts and rates are capped per tree, and a per-TAP anti-spoof chain refuses any frame whose MAC, ARP sender, or IP is not the one the daemon assigned — necessary precisely because forks reuse IPs, so a global source-IP set cannot express “valid”. Details: Security model — the tree jailer.

The jail costs ~2 ms per 3-VM cluster fork, because every member TAP is pinned in a single nft transaction.

What survives a fork — and what doesn’t#

Survives: everything inside the forked set — in-cluster TLS, established TCP between member VMs, disk state, process state. Guest clocks are re-based so time does not jump backwards.

Doesn’t survive: TCP connections to the outside world — the far end never forked, and the fork’s host-facing identity (its uplink /30 or pod IP) is new. Forks keep their in-cluster addresses; only the host-facing side changes. New outbound connections from the fork work, since 0.9.0, without disturbing the source’s.

  • Cluster mode — creating, adopting, and forking multi-VM clusters
  • Security model — isolation boundaries between forks
  • Time warp — the other thing a whole-cluster pause buys