Skip to main content
k7d makes a deliberate trade-off: it optimizes for fleets of your own environments (RL rollouts, agent tree search, CI sandboxes), not for hostile multi-tenant isolation between forks. Read this page before deciding where to run it.

​
The trade-off: one daemon, many VMs, one address space

Live copy-on-write fork requires parent and child memory to be mappings in the same process — that is what makes a ~5 ms fork possible. So the k7d daemon hosts all VMs in one address space. If your threat model is “mutually hostile customers on shared hardware”, run those pods on RuntimeClass k7-fc — the same daemon and shim, but each VM is a stock Firecracker process in its own jail — put tenants on separate k7d hosts, or use K7’s kata-firecracker-devmapper backend. If it is “thousands of copies of my own environment, some of which an agent will break on purpose”, this design buys you the fork — and the jailer below keeps a rogue guest from reaching anything but its own bridge.

​
Privilege separation

k7d.service starts one binary that forks itself in two before it allocates a thread or binds a socket. The shipped unit sets K7D_PRIVSEP=on (off is the escape hatch: one root process). k7d doctor’s privsep: section reports which it is. This is the daemon, not a third backend — both native and Firecracker VMs go through the same pair. A guest that gets out of KVM and out of the device models lands in a process that cannot mount, cannot create a network device, cannot write another uid’s cgroup, and cannot regain a capability through a setuid binary. What it can do is send the node broker one of a closed set of verbs, each naming an id the node broker already registered — never a path. Paths cross exactly once, at vm.register, before any disk is provisioned. Each is checked against a per-class root allowlist and resolved with openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS), then pinned as an O_PATH descriptor. /tmp is never a root on its own — only the /tmp/k7d-shim-<sandbox> family. Crash coupling is total: a VM-broker death makes the node broker release every registered id and exit non-zero so systemd replaces the pair. Native vCPU threads run in the VM broker (a crash kills native VMs). Standalone Firecracker pods can survive a VM-broker restart (K7D_VM_BROKER_RESTART, ships off). packaging/install.sh creates the k7d account (system user, nologin, member of kvm) and installs 60-k7d.rules so /dev/vhost-vsock is group kvm — that membership, not a capability, is what lets the unprivileged VM broker open /dev/kvm and /dev/vhost-vsock. /var/lib/k7d/fc stays root 0700. Full table and bounding-set rationale: k7d SECURITY.md.

​
The tree jailer

Every live cluster payload — a root, a fork, an adopted cluster — owns one bridge, and every guest on every bridge sits in the same 10.200.0.0/24. The jailer is the set of host-side controls that scope a tree to itself. All of them are on by default and every escape hatch is an explicit env var that logs a warning; an unrecognised value is a hard error, never a silent fallback.

​
Network jail (nftables, per bridge)

An nftables table per bridge, evaluated before the iptables rules:
  • Inter-tree traffic is dropped. Two trees on one node can no longer route to each other’s guests, even though they share a subnet and forks reuse IPs.
  • Egress policy per tree, inherited by forks: open (WAN + CNI egress, the default and the RuntimeClass product’s posture), deny (no WAN egress; intra-node CNI still forwarded), or whitelist (only named CIDR:port pairs, tcp+udp, plus an optional resolver on 53). See EgressPolicy.
  • Host lockdown: an input chain so a deny/whitelist tree cannot reach services on the host itself, except whitelist entries and established flows.
  • Cloud metadata (169.254.0.0/16) is dropped in every mode unless a whitelist names it.
  • Connection caps: per-tree concurrent-connection limit (16384) and new-connection rate (2000/s) — an inner cluster’s image pulls sit far below, a flood far above. K7D_NET_JAIL_CT_MAX / K7D_NET_JAIL_NEW_RATE.
  • Per-TAP anti-spoof: a netdev ingress chain on every TAP drops frames whose source MAC, ARP sender, or IP differs from the TAP’s assigned identity or claims another guest’s address in the shared /24, and drops every EtherType except IPv4 and ARP (which also disables guest IPv6 and VLAN-tag evasion).
Fork cost of the whole jail: ~2 ms per 3-VM cluster, because all member TAPs are pinned in one nft transaction. Teardown is RAII with the bridge and the TAP; stale tables from a daemon that died without running its destructors are reaped at startup. K7D_NET_JAIL=off skips installation for debugging.

​
Per-tree cgroup

Each tree gets a threaded cgroup for its vCPU and device threads (pids.max) and a domain cgroup for its helper processes (memory.max + pids.max), both sized from the tree’s TreeBudget. A tree that forks itself into a fork bomb, or a helper that leaks memory, hits its own wall. K7D_TREE_CGROUP=off disables.

​
Daemon seccomp

k7d is two processes when K7D_PRIVSEP=on (the shipped default): the node broker and the VM broker each install their own seccomp allowlist (NODE_SYSCALLS / VM_SYSCALLS) before any VM exists. The ioctl surface is matched by driver type byte (KVM, vhost, TUN, FICLONE, block, loop, XFS) rather than by enumerating request numbers. The filter survives execve, so it also covers helpers (virtiofsd, nft, ip, iptables). Default enforce; K7D_SECCOMP=log records instead of killing. k7d doctor run from a login shell reports privsep: K7D_PRIVSEP off because that process is not the daemon. Check the unit and the process table instead:

​
virtiofsd sandbox

virtiofsd — used only for the read-write hostPath fallback — is spawned in chroot sandbox mode so it cannot name a path outside its share. K7D_VIRTIOFSD_SANDBOX=namespace|none overrides.

​
Guest side (Ubuntu nodes)

The Ubuntu node image ships AppArmor profiles: the k7-agent may open AF_VSOCK (it is the harness’s control plane); everything else, pods included, runs under a profile that denies the vsock family. The agent vsock channel is deliberately not jailed on the host — it is point-to-point to the host CID, and no inter-tree vsock path exists.

​
Guest-controlled paths

The guest is attacker-controlled from its first instruction. The shim still runs as host root; with K7D_PRIVSEP=on the process that parses guest and shim input is the uid-k7d VM broker. Every host filesystem operation whose path the guest can influence follows four rules:
  1. No create, mkdir, open(O_CREAT), rename, unlink, chown, copy, or mount under a guest-writable directory unless the final component is opened relative to a share-root fd with O_NOFOLLOW / openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS), and an existing entry is an error, never reused.
  2. Anything attached to more than one VM is read-only and host-enforced (O_RDONLY + write requests rejected) — never “read-only because the guest mounts it ro”.
  3. Strings from the guest agent (paths, names, sizes) never reach a host path, command line, or cgroup path. Agent responses are bounded in length.
  4. Confinement fields in the OCI spec pass through to the guest; only what the guest provably lacks is stripped.
The virtiofs hostPath fallback (/var/lib/k7d/pods/<id>/volumes-rw) is the one guest-writable directory the host still operates under after boot: placeholders are created with mkdirat / openat(O_CREAT|O_EXCL|O_NOFOLLOW) relative to an O_PATH share-root fd, bind-mounts go through /proc/self/fd/N, and read-only shares are MS_RDONLY bind-remounted before virtiofsd chroots. K7D_DENY_VIRTIOFS_FALLBACK=1 fails Create for any pod that would take the fallback (hardened / Firecracker profiles). Named data disks are exclusive: a second sandbox asking for a disk id that is already attached is a Create error, not a shared writer.

​
Pod securityContext reaches guest runc

runtimeClassName: k7 does not weaken the confinement kubelet and containerd already encoded in the OCI spec. Guest runc sees linux.seccomp (including seccompProfile: RuntimeDefault), maskedPaths / readonlyPaths, linux.resources.devices exactly as CRI emitted them (deny-all for unprivileged pods, allow-all for privileged — which is what lets an in-guest kubelet open /dev/kmsg), capabilities, noNewPrivileges, and memory/CPU limits. The shim strips only what the guest kernel cannot honour: process.apparmorProfile (no LSM in the minimal guest), the host cgroupsPath, and host namespace paths. If guest runc rejects a seccomp profile, Create fails with runc’s error — the shim never strips the profile and retries.

​
Control socket

/run/k7d/k7d.sock is held by the VM broker (uid k7d) when privilege separation is on. serve creates the parent directory at 0750 and the socket at 0660; it never tightens a world-writable sticky parent, so keep it in /run/k7d, not /tmp. A process that can connect can create, stop, and fork every VM on the node — the same power as a compromised shim. K7D_ALLOWED_UIDS=0,1000 refuses connections from any other uid (SO_PEERCRED), and mutating ops (create_vm, stop_vm, fork_vm, tree writes, delete_data_disk, …) log the peer’s uid/gid/pid at info. This is an audit trail and a floor, not an auth system.

​
Firecracker profile (k7-fc)

runtimeClassName: k7-fc boots the same guest with the same shim, daemon and agent, but the VMM is a stock, sha-pinned Firecracker under the stock jailer instead of the in-process engine — see Backends. runtimeClassName: k7 stays the native in-process VMM; k7d.katakate.org/backend: firecracker on a k7 pod is the per-pod opt-in, and k7-fc is the admission-gateable default. An explicit backend: native on k7-fc is a loud Create error.

​
Docker as a guest service

k7d.katakate.org/docker=true runs a pinned dockerd as a child of the guest PID 1 agent (K7 exposes it as k7 create --docker). Its docker.sock is guest root: the sandbox securityContext no longer bounds what runs in the guest, and the only remaining boundaries are the VMM and the host-side rules above.
  • The payload slot (/run/k7d/docker-tools) is a read-only virtio-blk image, host-enforced (O_RDONLY, writes rejected). A guest that could rewrite dockerd would trojan every later sandbox.
  • The graph disk is a per-VM scratch image (k7d.katakate.org/docker-disk, default 20Gi), reflinked on fork and deleted with the VM — never a host directory shared across pods.
  • The CRI container is not privileged. Inner containers get dockerd’s default seccomp and capabilities. Because the guest root is an initramfs, dockerd runs with DOCKER_RAMDISK=1 so runc uses --no-pivot instead of pivot_root(2).
  • Multi-tenant Docker should use k7-fc: the guest side is identical, so it is a RuntimeClass switch, not a rewrite.

​
What survives a fork

Everything inside the forked set: in-cluster TLS sessions, established TCP between member VMs, disk state, process state. Guest clocks are re-based so time does not jump backwards. What doesn’t: TCP to the outside world (the far end never forked). Forks keep their in-cluster addresses; only the host-facing identity is new.

​
Hardening that exists today

  • ≤30k lines of Rust for the VMM + shim — a small, auditable surface, deliberately not a kitchen-sink VMM.
  • Memory safety by construction: Rust, with unsafe blocks contained, annotated with safety comments, and enforced by lint (deny(unsafe_op_in_unsafe_fn)).
  • Formal methods on selected paths: Kani bounded model checking over unsafe / address-arithmetic harnesses; Aeneas→Lean proofs of the tree budget/eviction model. Not a claim that everything is proven — see Formal verification.
  • Loud failures: no silent fallbacks; every error is surfaced. Fuzzing runs over the agent protocol and virtio-blk request parsing.
  • Guest agent is minimal: a static binary speaking JSON-lines over vsock; runc runs inside the guest, never on the host.
  • The daemon never displaces itself: k7d --version / --help answer without touching a running daemon, and serve refuses to start if a live listener already owns the socket.

​
Current limitations

  • Single host, x86_64 Linux + KVM only. Cross-node fork is on the roadmap. Does not build on macOS/Windows.
  • The node broker runs as root (KVM, TAP, bridge, nftables, and cgroup setup require it). With K7D_PRIVSEP=on (the shipped default) the process that parses guest and shim input is uid k7d with an empty capability set; the root half has no listening socket. The control socket is still a local Unix socket (0660, optional K7D_ALLOWED_UIDS) — protect host access to it.
  • Young project. One primary test machine, no external security audit yet. Small surface by design, but treat it accordingly.

​
Reporting a vulnerability

Email security@katakate.org, or open a private GitHub Security Advisory on Katakate/k7d. Include impact, reproduction steps or a PoC, and the affected version. Please do not open a public issue for security-sensitive reports. Fixes land on the latest release until 1.0.