Backends

​

At a glance

​

Choosing a backend

• You run many short-lived sandboxes per node and want the smallest possible footprint.
• You don’t need to pause, resume, fork, or persist state between restarts.
• You’re on a single node and don’t need replication.
• Security is paramount: Firecracker has the smallest attack surface, and the jailer is active (verified by tests/integration/test_firecracker.py::test_jailer_active).

• You need a real persistent disk that survives pod restarts (/mnt/state mount inside the sandbox).
• You want to pause / resume / fork sandboxes at the disk level, or k7 snapshot / k7 restore named snapshots.
• You run multi-node and want data to survive node failure (Longhorn replicates).
• You need bigger root disks (sized with --root-disk-size), or a Docker graph that survives pause / fork / restore.

• Fork latency matters: k7d warm-forks a running VM in ~5 ms with faithful memory, disk, process, and network state — versus ~45 s for a disk-only Longhorn fork.
• You want snapshot trees: fork many branches from a checkpoint, protect winners, prune losers, under RAM/disk budgets.
• Your workload is RL training or agent tree search over many resettable environments.
• You want a forkable Docker engine: k7 create --docker --backend k7d runs dockerd inside the guest and k7 fork clones it, running inner containers included.

• You want k7d’s fork / pause lifecycle and Firecracker’s jailer around every VM — a separate VMM process per sandbox with its own uid, chroot, pid namespace, cgroup, and Firecracker’s default seccomp. A guest-to-VMM escape lands in that jail instead of the k7d daemon.
• You run --docker for mutually distrusting tenants. The guest side is identical to k7d, so this is a RuntimeClass switch, not a rewrite.
• You can live without virtiofs / hostPath volumes (refused up front on k7d-fc) and without k7d’s time warp.

​

Specifying the backend

​

At install time

k7 install --backend kfd                       # required --backend; kata-firecracker-devmapper only (auto-detects raw disk)
k7 install --backend kfd --disk /dev/nvme2n1   # pin the thin-pool disk explicitly
k7 install --backend kql                       # kata-qemu-longhorn only, no raw disk needed
k7 install --backend kql --longhorn-extra-disk /mnt/longhorn-extra   # optional extra storage
k7 install --backend k7d                       # k7d runtime (daemon + shim + RuntimeClass k7)
k7 install --backend k7d,k7d-fc                # add pinned Firecracker + jailer and RuntimeClass k7-fc
k7 install --backend k7d --k7d-version 0.7.0   # pin the k7d release (playbook default: 0.7.0)
k7 install --backend k7d --k7d-artifact /root/k7d-v0.7.0-x86_64-linux.tar.gz   # install from a local tarball
k7 install --backend kfd,kql --longhorn-extra-disk /mnt/longhorn-extra   # omit --disk on dual-NVMe (auto-detect)
k7 install -i inventory.ini                    # multi-node, per-host backends
k7 install -i inventory.ini --ha               # multi-master HA (3+ servers, embedded etcd)

​

At sandbox creation time

# Auto-detect (uses /etc/k7/backend on the API node, else kfd)
k7 create demo alpine:3.20 --cpu 1 --memory 1Gi

# Explicit
k7 create demo alpine:3.20 --backend kql --cpu 1 --memory 1Gi
k7 create demo alpine:3.20 --backend k7d --cpu 1 --memory 1Gi
k7 create builder ubuntu:24.04 --backend k7d-fc --docker --egress-open --cpu 2 --memory 3Gi

backend: kata-qemu-longhorn   # or kata-firecracker-devmapper, k7d, k7d-fc — or alias kql / kfd / k7-fc

​

Backend-specific behavior

​

kata-qemu-longhorn

• Each sandbox gets a Longhorn PVC named <sandbox>-root-lh, defaulting to 10Gi (override with --root-disk-size 20Gi or in YAML).
• The container runs privileged to allow the persistence wrapper to bind-mount /mnt/state from the PVC.
• Image base must include /bin/sh, tar, and mount (Debian/Ubuntu/Alpine all work).
• Image ENTRYPOINT+CMD are preserved by default — override with --entrypoint / --cmd.
• pause takes a crash-consistent VolumeSnapshot before scaling to 0 (optional; pass --snapshot <name>).
• fork snapshots the source PVC, clones it via Longhorn’s CSI VolumeSnapshot+Restore, then creates a new Deployment pointing at the cloned PVC.
• --docker adds a second Longhorn Block PVC (<name>-docker-lh, default 20Gi via --docker-disk) for the overlay2 graph; it is included in pause / fork / restore. The two VolumeSnapshots are crash-consistent per volume, not cross-volume atomic.

​

kata-firecracker-devmapper

• Sandbox storage is a per-pod logical volume on the LVM thin-pool — fast to create, lost on delete.
• The Firecracker process runs inside the jailer: a chroot, with dropped capabilities and a restrictive seccomp filter, as an unprivileged UID.
• Kata’s guest seccomp is enforced (disable_guest_seccomp = false): the container’s RuntimeDefault seccomp profile is applied by the in-guest runtime, not silently dropped.
• No pause/resume/fork — there’s no persistent disk to snapshot.
• --root-disk-size has no effect (warned by the CLI).
• --docker puts the overlay2 graph on a generic ephemeral volume from StorageClass k7-docker-lvm (OpenEBS LVM LocalPV over kata-vg/thin-pool); it is deleted with the pod and k7 fork of a kfd --docker sandbox is rejected. Docker-heavy kfd nodes share the thin-pool with these LVs — size kata_thinpool_pv_size accordingly.

​

k7d

• Pods run as k7d microVMs via runtimeClassName: k7 — a small containerd shim talks to the k7d daemon over /run/k7d/k7d.sock.
• Fork copies the whole VM: memory, disk, processes, and network identity — not just the disk. Warm fork is ~5 ms.
• Multi-vCPU guests are supported: the vCPU count is derived from the pod’s CPU limit (cpu: "2" boots a 2-vCPU guest), with fork/snapshot parity.
• VM ops work for sandboxes on any node. The k7d daemon and its socket are node-local, but k7 pause / resume / fork through the API are forwarded to the k7-agent DaemonSet pod on the sandbox’s node when needed. A fork still lands on the source’s node (cross-node fork is on the k7d roadmap).
• --docker does not add a second container: the pod is stamped k7d.katakate.org/docker=true and the guest agent (PID 1) supervises a pinned dockerd whose graph lives on a per-sandbox virtio-blk scratch disk (overlay2, default 20Gi). Because it is not a CRI sidecar, k7 fork of a --docker sandbox works — the child keeps dockerd and its running inner containers. Requires k7d 0.6.0+ (payload at /usr/local/share/k7d/docker); an older k7d fails loudly with this k7d has no docker service; upgrade.
• Warm fork works for single-workload sandboxes (including --docker); forking a sandbox with a real CRI sidecar is rejected (use kql for that).
• Per-node k7d disk-pool utilization is visible via k7 nodes storage (see CLI reference).
• See RuntimeClass k7 for how the runtime is registered, and the feature matrix for current Kubernetes feature support.

​

k7d-fc

• Same daemon, shim, guest agent, and --docker service as k7d; pods use runtimeClassName: k7-fc. The daemon boots the guest on upstream Firecracker under the upstream jailer (sha-pinned binaries installed to /usr/local/bin by the playbook — a different binary from Kata’s /opt/kata/bin Firecracker used by kfd). k7 0.4.0 and k7d both pin v1.16.2. k7 0.3.1 still vendored v1.16.1.
• k7 install --backend k7d-fc writes /etc/k7d/shim-k7-fc.toml (backend = "firecracker"), registers containerd runtimes.k7-fc with a ConfigPath option (and no BinaryName), creates RuntimeClass k7-fc with a PodOverhead of 32Mi / 50m, and labels the node k7.katakate.org/backend-k7d-fc=true (plus backend-k7d, so native k7d pods still schedule there).
• Fork and pause go through the same /run/k7d/k7d.sock verbs and fork-source-* annotations as k7d. Under the hood a fork is a Firecracker Diff snapshot materialised onto a reflinked base image and LoadSnapshoted into a fresh jail (~2.5 s end-to-end via k7; ~2.45 s fork→exec measured).
• hostPath / PVC volumes are refused at create time — Firecracker has no virtiofs. Time warp / dilation is unavailable on this backend.
• Pause / resume → exec needs Firecracker v1.16.2+. k7 0.4.0 copies v1.16.2. k7 0.3.1 still copied v1.16.1, which gated vsock RX after a bare resume (upstream #6100 / k7d CHALLENGES); current k7d then fails k7 resume loudly (Firecracker < v1.16.2). Native k7d is unaffected.
• Requires k7d 0.6.0+ (Firecracker engine + ConfigPath). The stock 0.5.0 tarball does not know ConfigPath and the RuntimeClass will not come up.