Backends: native and Firecracker

k7d boots every VM through the same daemon and the same containerd shim. The backend is which VMM process owns the KVM fd:

native (default) — k7d’s in-process rust-vmm VMM. The VM lives in the daemon’s address space; warm fork is a MAP_PRIVATE CoW of guest memory (~5 ms). This is what every other k7d page describes.
firecracker — a stock, sha-pinned upstream Firecracker v1.16.2 process, launched under the stock jailer. Same guest kernel, initramfs, agent, protocol, fork/tree/pool verbs and Docker service; the VMM is the audited upstream binary and sits in its own uid, chroot, pid namespace, cgroup and seccomp filter. v1.16.2 over v1.16.1 is upstream #6100: earlier 1.16.x gated vsock RX after a bare pause → resume, so CRI exec never answered.

Note

The Firecracker engine ships in k7d 0.6.0 (install.sh --with-k3s registers RuntimeClass k7-fc and the ConfigPath runtime option). The 0.5.0 tarball does not.

Selecting a backend#

Three knobs, one result (VmConfig.backend). The first one that yields a backend wins; a contradiction is a loud Create error.

Knob	Where	Native	Firecracker
`VmConfig.backend`	control socket `create_vm`	`native` (serde default)	`firecracker`
Pod annotation `k7d.katakate.org/backend`	forwarded into the OCI spec by `pod_annotations`	`native`	`firecracker`
RuntimeClass `k7` / `k7-fc`	containerd handler; `runtimes.k7-fc` sets `[options] ConfigPath = /etc/k7d/shim-k7-fc.toml` (`backend = "firecracker"`) because the k3s CRI does not stamp `io.kubernetes.cri.runtime-handler`	handler `k7`	handler `k7-fc`

Rules:

runtimeClassName: k7-fc ⇒ Firecracker, even with no pod annotation.
runtimeClassName: k7 ⇒ native unless the pod carries k7d.katakate.org/backend: firecracker (per-pod opt-in).
k7d.katakate.org/backend: native on a k7-fc pod is a Create error containing contradicts. The RuntimeClass is never silently overridden.
Any other annotation value fails Create — a typo must not boot a pod on the wrong engine.

Gate admission on runtimeClassName (k7 vs k7-fc); k7d does not ship a Kyverno/OPA policy. In K7 the two RuntimeClasses surface as the k7d and k7d-fc backends.

apiVersion: v1kind: Podmetadata:  name: demo-fcspec:  runtimeClassName: k7-fc  containers:    - name: main      image: alpine:3.20      command: ["sleep", "infinity"]

k7d vs k7d-fc#

That is not “bare VM vs RuntimeClass.” Both surfaces run on either VMM. Privilege separation is the daemon, not a third backend: one node broker (root, no listening socket) plus one VM broker (uid k7d, empty caps) for the whole node. See Privilege separation.

	native (`k7d`)	firecracker (`k7d-fc`)
Pick on the socket	`backend: native` (default)	`backend: firecracker`
Kubernetes RuntimeClass	`runtimeClassName: k7`	`runtimeClassName: k7-fc`
VMM	k7d’s in-process rust-vmm	stock Firecracker v1.16.2 + jailer
Per-VM jail (own uid, chroot, seccomp)	❌	✅
Independently audited VMM	❌	✅
Privilege split (root node broker / uid-`k7d` VM broker)	✅ same daemon	✅ same daemon
virtiofs	✅	❌
Built for	RL, gym, fleets you control	multi-tenant / hostile guests

Bare VM / tree / gym (daemon API — not a Kubernetes RuntimeClass):

	native	firecracker
Warm VM fork	✅ ~5–50 ms	✅ ~76–140 ms / child
Whole-cluster fork	✅	✅
Snapshot tree / warm pool	✅	✅
Guest time warp / ×N dilation	✅	❌
Lockstep cluster pause	✅	⚠️ parallel pause (µs skew)

Kubernetes RuntimeClass pods (host runtimeClassName — not the gym):

	`k7` (native)	`k7-fc` (firecracker)
`kubectl logs` / `exec`	✅	✅
hostPath (virtiofs)	✅	❌
docker-in-guest	✅	✅
Pause / resume → exec	✅	✅ on Firecracker v1.16.2+ (v1.16.0/v1.16.1 gated vsock RX after resume)

k7d-fc vs forkd#

Both wrap Firecracker and CoW-fork guest RAM. forkd is a sandbox fan-out. k7d-fc is k7d’s tree / cluster / RuntimeClass product with Firecracker swapped in as the VMM.

	k7d-fc	forkd
Stock Firecracker (no patch)	✅	❌ vendored FC for live BRANCH (`MAP_SHARED` + UFFD)
Stock jailer (uid, chroot, seccomp)	✅ always	❌ drives the API socket; no jailer
Whole-cluster fork	✅	❌ one sandbox VM
Snapshot tree / RAM+disk budget	✅ daemon API	⚠️ snapshot tag + branch
Kubernetes RuntimeClass	✅ `k7-fc`	❌
Privilege-separated daemon	✅	❌
Warm fork of a running VM	✅ ~76–140 ms / child	✅ ~56–150 ms BRANCH
100 children from one snapshot	❌ jailer spawn per child	✅ ~100 ms (`mmap MAP_PRIVATE`)
Live BRANCH (source barely pauses)	❌ Diff snapshot + `LoadSnapshot`	✅ v0.4 UFFD-WP (needs the FC patch)
Guest time warp / ×N dilation	❌	❌
Built for	k7d tree + hostile guests	agent sandbox fan-out

Native k7d (~5 ms in-process fork) is a different mechanism — do not compare that number to forkd. k7d-fc pays the jailer on every child so a guest→VMM bug dies in the jail, not in a patched VMM.

Native is the default and the 250 ms latency suite. Use Firecracker when the guest might try to escape — especially multi-tenant --docker. Asking for virtiofs or time warp on the Firecracker backend fails Create loudly; nothing is silently dropped.

Feature matrix#

Feature	Native (`k7`)	Firecracker (`k7-fc`)
VMM	in-process rust-vmm	stock Firecracker v1.16.2 under the stock jailer (sha-pinned)
Guest → host boundary	KVM	KVM + jailer (uid, chroot, pid ns, cgroup) + Firecracker seccomp
Agent	vsock	vsock (Unix socket inside the jail, same protocol)
Rootfs	erofs on virtio-blk	erofs on virtio-blk (Firecracker drives)
Scratch / data disks / image slots	virtio-blk	virtio-blk, filled after boot with `PATCH /drives`
Network	daemon TAP + bridge	same TAPs, moved into the jail by name
Warm fork (CoW memory)	`MAP_PRIVATE`	Diff snapshot → `FICLONERANGE` onto a reflinked base → `LoadSnapshot`
Cluster fork / snapshot tree / warm pool	yes	yes (`K7D_POOL_BACKEND=firecracker` for the pool)
`k7d.katakate.org/docker` (in-guest dockerd)	yes	yes — same agent service
virtiofs / hostPath fallback	yes (or `K7D_DENY_VIRTIOFS_FALLBACK=1`)	unsupported — Create fails naming the backend
Time warp / dilation / autowarp	yes	unsupported
Fork latency (256 MiB busybox, warm)	~5–50 ms	~190 ms for one child, ~76 ms/child at n=8
PodOverhead	none	`32Mi` / `50m` on the RuntimeClass

Unsupported is a named error, never a silent downgrade: a pod that asked for virtiofs or time warp on k7-fc does not reach Running.

How a Firecracker fork works#

A native tree forks by mapping the source’s memory MAP_PRIVATE. A Firecracker tree has no such mapping, so a point is:

PUT /snapshot/create on the paused source — a Diff snapshot by default (only pages dirtied since the previous point; K7D_FC_FORK_SNAPSHOT=full writes all of RAM and exists for cross-checking, not as a fallback).
The Diff extents are FICLONERANGEd onto a root-owned base.mem; restore-N.mem is a FICLONE of the result — the point costs its dirty set, the base costs O(1).
Each child is a fresh jailed Firecracker LoadSnapshoted from its own reflink of restore-N.mem (mem_backend: File, clock_realtime), with writable disks reflinked from clones taken at the same paused instant. Every child extent reports FIEMAP_EXTENT_SHARED — CoW is proven, not assumed.

Cluster fork runs that primitive on every member after a parallel PATCH /vm Paused (pause skew p99 ≈ 125 µs across 20 runs). Suspend exports a Full point out of the jail and drops the process; resume re-stages a jail and LoadSnapshots with network_overrides so the member keeps its TAP identity.

Measured on an exclusive Hetzner node (2026-09-08, 256 MiB / 1 vCPU busybox guests, 500 ms per-child ceiling asserted in tests):

Operation	Typical
Root point at `tree_create` (first Diff, 43 MiB)	169 ms
`prepare_fork_source` (Diff, ~1–2 MiB dirtied)	140–157 ms
`fork_batch` n=1	190 ms, child answers `ping` 5 ms later
`fork_batch` n=8	607 ms = 76 ms per child
`rollback` (load the existing point)	75–90 ms
Cluster `fork_batch`, 3 members	~0.67 s (debug build)
Cluster suspend / resume (2 members)	~2.1 s / ~512 ms (debug build)
Warm pool acquire (`K7D_POOL_BACKEND=firecracker`)	~100 ms

Most of the per-child cost is staging a chroot, leasing a uid, spawning jailer (new namespaces, pivot_root, cgroup) and waiting for its API socket — the warm pool of pre-staged jails is what recovers it. See LATENCY_BUDGETS.md for the native rows.

What the jail buys, and what it does not#


Gained	The VMM is out of the daemon’s address space. A guest → VMM escape lands in a per-VM uid, chroot, pid namespace, cgroup and Firecracker’s seccomp filter — not in `k7d`.
Unchanged	The daemon is still the one host component both backends share (control socket, TAP/bridge, vsock proxy, tree bookkeeping). Privilege separation splits which of its processes holds root, not which host owns the work. The shim still rewrites the OCI spec, and the guest-controlled path rules are the floor for both backends. The guest agent is the same binary. Sibling forks of one tree still share the daemon.
Lost	Time warp, dilation, autowarp. virtiofs / hostPath (`backend firecracker has no virtiofs`). Native `MAP_PRIVATE` fork latency.

Multi-tenant --docker should use k7-fc: the guest side is identical, so it is a RuntimeClass switch, not a rewrite. The VMM jail is the extra boundary; the in-guest docker.sock is still guest root. See Security.

Installing#

packaging/install.sh --with-k3s registers both k7 and k7-fc (containerd blocks without BinaryName, /etc/k7d/shim-k7-fc.toml, RuntimeClass k7-fc with overhead.podFixed 32Mi / 50m).
utils/install-firecracker.sh puts the pinned firecracker and jailer in /usr/local/bin, verifying every sha256 from guest/fc/pins.env. A digest mismatch is fatal. k7d doctor reports the version and sha it finds.
Daemon knobs: K7D_FC_CHROOT_BASE (jail root; default /var/lib/k7d/fc, a link onto the reflink XFS that holds the writable disks so they can be hard-linked into a jail), K7D_FC_UID_BASE (first uid of the per-VM lease range; default 60000, 1024 live VMs per daemon, exhaustion is an error rather than a wrap-around), K7D_FC_FORK_SNAPSHOT (diff default, full), K7D_POOL_BACKEND (native | firecracker for the warm pool).

k7 0.4.0+ (0.5.0 included) and k7d 0.7.0+ (0.9.0 included) all pin Firecracker v1.16.2. k7 0.3.1 still vendored v1.16.1, which gated vsock RX after a bare PATCH /vm Resumed (upstream #6100). Current k7d fails k7 resume loudly on that old pin (guest unreachable over vsock after Resumed (Firecracker < v1.16.2)). Native k7d is unaffected. Any k7 ≥ 0.4.0 with k7d ≥ 0.7.0 gives resume→exec on k7d-fc; the k7 playbook still defaults to k7d 0.7.0 (--k7d-artifact for a newer tarball).

Reference#

docs/backends.md — selection matrix, source of truth
OVERHEAD.md — how the k7-fc PodOverhead was measured
RuntimeClass k7 — installing the runtime, annotation reference
agent-sandbox on k7d — both backends behind kubernetes-sigs/agent-sandbox, with the per-feature matrix