Backends

A backend in Katakate combines a VM runtime with a storage driver. There are four: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d, and k7d-fc. A node can install any combination (k7 install --backend kfd,kql,k7d,k7d-fc) and each sandbox picks one (k7 create --backend …).

At a glance#

Numbers are medians of 3 on one Hetzner AX41 node; ranges and methodology are in PERFORMANCE.md.

Privilege separation is the k7d daemon, not a fifth backend: a root node broker (no listening socket) plus a uid-k7d VM broker for the whole node. Both k7d and k7d-fc sandboxes go through that pair. Tables: k7d vs k7d-fc and privilege separation.

Choosing a backend#

Pick kata-firecracker-devmapper when:

• You run many short-lived sandboxes per node and want the smallest possible footprint.
• You don’t need to pause, resume, fork, or persist state between restarts.
• You’re on a single node and don’t need replication.
• Security is paramount: Firecracker has the smallest attack surface, and the jailer is active (verified by tests/integration/test_firecracker.py::test_jailer_active).

Pick kata-qemu-longhorn when:

• You need a real persistent disk that survives pod restarts (/mnt/state mount inside the sandbox).
• You want to pause / resume / fork sandboxes at the disk level, or k7 snapshot / k7 restore named snapshots.
• You run multi-node and want data to survive node failure (Longhorn replicates).
• You need bigger root disks (sized with --root-disk-size), or a Docker graph that survives pause / fork / restore.

Pick k7d when:

• Fork latency matters: k7d warm-forks a running VM in ~5 ms with faithful memory, disk, process, and network state — versus ~45 s for a disk-only Longhorn fork.
• You want snapshot trees: fork many branches from a checkpoint, protect winners, prune losers, under RAM/disk budgets.
• Your workload is RL training or agent tree search over many resettable environments.
• You want a forkable Docker engine: k7 create --docker --backend k7d runs dockerd inside the guest and k7 fork clones it, running inner containers included.

Pick k7d-fc when:

• You want k7d’s fork / pause lifecycle and Firecracker’s jailer around every VM — a separate VMM process per sandbox with its own uid, chroot, pid namespace, cgroup, and Firecracker’s default seccomp. A guest-to-VMM escape lands in that jail instead of the k7d daemon.
• You run --docker for mutually distrusting tenants. The guest side is identical to k7d, so this is a RuntimeClass switch, not a rewrite.
• You can live without virtiofs / hostPath volumes (refused up front on k7d-fc) and without k7d’s time warp.

See the K7d documentation for the runtime’s own concepts, API, and benchmarks, and Backends: native vs Firecracker for how the k7d daemon selects the engine.

Specifying the backend#

At install time#

k7 install --backend kfd --api-mode development   # --backend and --api-mode are both required; kfd auto-detects a raw diskk7 install --backend kfd --disk /dev/nvme2n1   # pin the thin-pool disk explicitlyk7 install --backend kql                       # kata-qemu-longhorn only, no raw disk neededk7 install --backend kql --longhorn-extra-disk /mnt/longhorn-extra   # optional extra storagek7 install --backend k7d                       # k7d runtime (daemon + shim + RuntimeClass k7)k7 install --backend k7d,k7d-fc                # add pinned Firecracker + jailer and RuntimeClass k7-fck7 install --backend k7d --k7d-version 0.7.0   # pin the k7d release (playbook default: 0.7.0; other versions need k7d_sha256=<hex>)k7 install --backend k7d --k7d-artifact /root/k7d-v0.9.0-x86_64-linux.tar.gz   # install a newer k7d from a local tarballk7 install --backend kfd,kql --longhorn-extra-disk /mnt/longhorn-extra   # omit --disk on dual-NVMe (auto-detect)k7 install -i inventory.ini                    # multi-node, per-host backendsk7 install -i inventory.ini --ha               # multi-master HA (3+ servers, embedded etcd)

At sandbox creation time#

# Auto-detect (uses /etc/k7/backend on the API node, else kfd)k7 create demo alpine:3.20 --cpu 1 --memory 1Gi# Explicitk7 create demo alpine:3.20 --backend kql --cpu 1 --memory 1Gik7 create demo alpine:3.20 --backend k7d --cpu 1 --memory 1Gik7 create builder ubuntu:24.04 --backend k7d-fc --docker --egress-open --cpu 2 --memory 3Gi

In YAML:

backend: kata-qemu-longhorn   # or kata-firecracker-devmapper, k7d, k7d-fc — or alias kql / kfd / k7-fc

If the requested backend isn’t available on any node, you’ll see a clear FailedScheduling message listing which backends are.

Backend-specific behavior#

kata-qemu-longhorn#

• Each sandbox gets a Longhorn PVC named <sandbox>-root-lh, defaulting to 10Gi (override with --root-disk-size 20Gi or in YAML).
• The container runs privileged to allow the persistence wrapper to bind-mount /mnt/state from the PVC.
• Image base must include /bin/sh, tar, and mount (Debian/Ubuntu/Alpine all work).
• Image ENTRYPOINT+CMD are preserved by default — override with --entrypoint / --cmd.
• pause takes a crash-consistent VolumeSnapshot before scaling to 0 (optional; pass --snapshot <name>).
• fork snapshots the source PVC, clones it via Longhorn’s CSI VolumeSnapshot+Restore, then creates a new Deployment pointing at the cloned PVC.
• --docker adds a second Longhorn Block PVC (<name>-docker-lh, default 20Gi via --docker-disk) for the overlay2 graph; it is included in pause / fork / restore. The two VolumeSnapshots are crash-consistent per volume, not cross-volume atomic.

kata-firecracker-devmapper#

• Sandbox storage is a per-pod logical volume on the LVM thin-pool — fast to create, lost on delete.
• The Firecracker process runs inside the jailer: a chroot, with dropped capabilities and a restrictive seccomp filter, as an unprivileged UID.
• Kata’s guest seccomp is enforced (disable_guest_seccomp = false): the container’s RuntimeDefault seccomp profile is applied by the in-guest runtime, not silently dropped.
• pause / resume only scale the Deployment; fork, k7 snapshot create and k7 pause --snapshot are rejected up front — there’s no persistent disk to snapshot.
• --root-disk-size has no effect (warned by the CLI).
• --docker puts the overlay2 graph on a generic ephemeral volume from StorageClass k7-docker-lvm (OpenEBS LVM LocalPV over kata-vg/thin-pool); it is deleted with the pod and k7 fork of a kfd --docker sandbox is rejected. Docker-heavy kfd nodes share the thin-pool with these LVs — size kata_thinpool_pv_size accordingly.

k7d#

• Pods run as k7d microVMs via runtimeClassName: k7 — a small containerd shim talks to the k7d daemon over /run/k7d/k7d.sock.
• Fork copies the whole VM: memory, disk, processes, and network identity — not just the disk. Warm fork is ~5 ms. With k7d 0.9.0+ the fork keeps a working NIC (own bridge, NAT), so forked sandboxes have egress like their source; older k7d forks came up without network.
• Multi-vCPU guests are supported: the vCPU count is derived from the pod’s CPU limit (cpu: "2" boots a 2-vCPU guest), with fork/snapshot parity.
• VM ops work for sandboxes on any node. The k7d daemon and its socket are node-local, but k7 pause / resume / fork through the API are forwarded to the k7-agent DaemonSet pod on the sandbox’s node when needed. A fork still lands on the source’s node (cross-node fork is on the k7d roadmap).
• --docker does not add a second container: the pod is stamped k7d.katakate.org/docker=true and the guest agent (PID 1) supervises a pinned dockerd whose graph lives on a per-sandbox virtio-blk scratch disk (overlay2, default 20Gi). Because it is not a CRI sidecar, k7 fork of a --docker sandbox works — the child keeps dockerd and its running inner containers. Requires k7d 0.6.0+ (payload at /usr/local/share/k7d/docker); an older k7d fails loudly with this k7d has no docker service; upgrade.
• Warm fork works for single-workload sandboxes (including --docker); forking a sandbox with a real CRI sidecar is rejected (use kql for that).
• Per-node k7d disk-pool utilization is visible via k7 nodes storage (see CLI reference).
• See RuntimeClass k7 for how the runtime is registered, and the feature matrix for current Kubernetes feature support.

k7d-fc#

• Same daemon, shim, guest agent, and --docker service as k7d; pods use runtimeClassName: k7-fc. The daemon boots the guest on upstream Firecracker under the upstream jailer (sha-pinned binaries installed to /usr/local/bin by the playbook — a different binary from Kata’s /opt/kata/bin Firecracker used by kfd). k7 0.4.0+ (including 0.5.0) and k7d both pin v1.16.2; k7 0.3.1 still vendored v1.16.1.
• k7 install --backend k7d-fc writes /etc/k7d/shim-k7-fc.toml (backend = "firecracker"), registers containerd runtimes.k7-fc with a ConfigPath option (and no BinaryName), creates RuntimeClass k7-fc with a PodOverhead of 32Mi / 50m, and labels the node k7.katakate.org/backend-k7d-fc=true (plus backend-k7d, so native k7d pods still schedule there).
• Fork and pause go through the same /run/k7d/k7d.sock verbs and fork-source-* annotations as k7d. Under the hood a fork is a Firecracker Diff snapshot materialised onto a reflinked base image and LoadSnapshoted into a fresh jail (~2.5 s end-to-end via k7; ~2.45 s fork→exec measured).
• hostPath / PVC volumes are refused at create time — Firecracker has no virtiofs. Time warp / dilation is unavailable on this backend.
• Pause / resume → exec needs Firecracker v1.16.2+, which k7 0.4.0+ installs. k7 0.3.1 still copied v1.16.1, which gated vsock RX after a bare resume (upstream #6100 / k7d CHALLENGES); current k7d then fails k7 resume loudly (Firecracker < v1.16.2). Native k7d is unaffected.
• Requires k7d 0.6.0+ (Firecracker engine + ConfigPath). The stock 0.5.0 tarball does not know ConfigPath and the RuntimeClass will not come up.