The k7d daemon exposes its control plane on a Unix socket, by default /run/k7d/k7d.sock (override with --socket or the K7D_SOCKET env var).
Protocol: one JSON request object per line, one JSON response object per line. A connection may carry multiple request/response pairs. Requests are tagged with "op", responses with "status". Errors are always an explicit {"status": "error", "message": "..."} response — never a silent success.
Clients reach the guest agent themselves over vsock, using the guest_cid returned by create_vm — the daemon does not proxy the agent protocol (vsock CIDs are a kernel-global namespace, so any host process can connect to a running VM’s agent).
Fields and behaviour marked (0.8.0) shipped in 0.8.0 and were not in the 0.7.0 tarball: dilation_policy / dilation_config, ksm, refresh, pause_ns, the adopted-VM teardown rules, and the new tree_nodes / doctor fields. Those marked (0.9.0) are new in 0.9.0: fork_vm.pod_netns, the set_dilation_policy op, tree_nodes.live_cpu / dilation_ladder, and the per-network namespace behind every host bridge.
Reference clients: a thin Python client covering the tree verbs in examples/cluster-tree-search/k7d_client.py, an MCP server wrapping the tree for a frontier agent in examples/k7d-mcp/, and a read-only SSE dashboard in examples/k7view/.
Host operations#
op | Fields | What it does |
|---|---|---|
ping | — | Liveness check. → pong with the daemon version |
doctor | — | Host readiness: /dev/kvm, /dev/vhost-vsock, /dev/net/tun, cgroup v2. Same checks as k7d doctor; creates no VM. → doctor with kvm, vhost_vsock, tun, cgroup_v2, ok, fc_fork_mode (diff | full, from K7D_FC_FORK_SNAPSHOT), ksm_run (0.8.0), details[]. ksm_run is /sys/kernel/mm/ksm/run read as-is (0 stopped, 1 scanning, 2 unmerging, absent without KSM). It is advisory and does not affect ok |
pool_status | — | Warm-pool sizing and hit/miss counters. Always answers, even with the pool disabled (size: 0) |
VM lifecycle operations#
op | Fields | What it does |
|---|---|---|
create_vm | config (VmConfig) | Create and boot a VM. → vm_created |
vm_status | vm_id | Query a VM’s state. → vm_state with state, guest_cid, virtiofs_shares, block_devices, fork_memory_full_copy, and the kernel it actually booted |
list_vms | — | List the daemon’s sandbox registry: create_vm and fork_vm VMs. Tree members (adopted, tree_create*, tree_fork*) are not listed; find those with tree_list / tree_nodes. → vm_list |
stop_vm | vm_id | Shut down and forget a VM. Ephemeral disks go with it; named data disks stay. → ok. (0.8.0) On an id that tree_adopt_cluster took, it errors with VM … was adopted into tree …; use tree_prune / tree_drop and frees nothing. Unknown ids still return no such VM |
pause_vm / resume_vm | vm_id | Stop / restart the vCPU threads and park device workers; guest memory, devices, and the vsock CID are retained. Idempotent. → ok |
dump_serial | vm_id or guest_cid | Read a VM’s accumulated serial console output (guest kernel messages, agent stderr). The guest_cid form also finds live VMs inside snapshot trees; vm_id: "{tree_id}/{node_id}" works too. A fork’s console buffer starts empty, so dump the source before forking if you need its pre-fork output. → serial_dump |
fork_vm | vm_id, sandbox_id?, pod_netns?, dilation? | Warm-fork a daemon-owned VM into a new daemon-owned VM. The first fork of a VM prepares it as a fork source on the same vsock CID (echoed as source_guest_cid). sandbox_id registers the fork like a create_vm-born sandbox VM. (0.9.0) A networked source’s fork gets its own bridge with a TAP at the source’s vm_index (the guest kept that IP/MAC in memory); pod_netns attaches that TAP into the fork pod’s CNI netns with NAT, so the fork pod’s NetworkPolicy governs it. A pod-backed fork (sandbox_id set) of a pod-attached source must name pod_netns — it fails loudly otherwise. → vm_forked with vm_id, guest_cid, source_guest_cid, dilation |
delete_data_disk | id | Delete a named persistent data disk’s image — the only way its data goes away. Loud error if the image is missing or a live VM still has it attached |
update_drive | vm_id, device_index, host_path | Firecracker only: re-point block device i (/dev/vd{a+i}) at a new host file via PATCH /drives — how image slots are filled after boot. Sending it to a native VM is an error, not a no-op |
lookup_sandbox | sandbox_id | Resolve a CRI sandbox to its live VM. → sandbox_found |
reattach_sandbox | sandbox_id | Like lookup_sandbox, but verifies the guest agent still answers before handing the VM back to a respawned shim. Dead VMs are torn down; stale records garbage-collected (both return error) |
VmConfig#
The config object accepted by create_vm (and the tree-create operations):
| Field | Type | Default | Meaning |
|---|---|---|---|
kernel | string | env K7D_KERNEL | Guest kernel path |
initrd | string | env K7D_INITRD | Initramfs path. Omitted on a disk-booted guest (cmdline with root=) means no initrd — never the busybox default |
cmdline | string | busybox default | Replacement kernel command line. A guest with a different init contract (an Ubuntu node with root=/dev/vda) must set it; there is no merge |
memory_mb | u64 | 256 | Guest RAM in MiB |
vcpus | u32 | 1 | Guest vCPU count. Multi-vCPU (SMP) guests are supported; the shim derives the count from the pod’s CPU limit (ceil(quota/period), floor 1). Fork/snapshot have full parity across vCPU counts |
cpu_quota / cpu_period | u64 | none | Host CFS quota in µs per period, applied to the VM’s vCPU threads. Both set or neither |
vsock | bool | true | Attach a vhost-vsock device (needed for the agent) |
virtiofs | array | [] | virtiofs shares: {host_dir, guest_mount, readonly} |
block_devices | array | [] | virtio-blk images in device order (/dev/vda, …): {host_path, readonly, guest_mount?, fstype?}. guest_mount + fstype (e.g. erofs, ext4) makes the agent mount the device before containers start — both set or both absent, validated loudly |
guest_dirs | array | [] | Absolute guest tmpfs directories the agent creates at boot (emptyDir backing) |
scratch_size_bytes | u64 | none | Capacity of the sandbox’s writable scratch disk — a sparse ext4 image the daemon provisions and attaches as the last read-write virtio-blk device; every container’s overlay upper lives on it. None = a small, bounded tmpfs, said loudly. The shim always sets it (default 8 GiB via k7d.katakate.org/scratch-size) |
container_scratch | array | [] | {name, size_bytes} — Kubernetes container names that get their own writable image instead of a directory on the shared scratch. Each entry costs one virtio-blk device slot |
data_disks | array | [] | {id, size_bytes, guest_mount} — named persistent ext4 disks that outlive the VM. Created and formatted on first use, reused as-is on later attaches; a size_bytes that disagrees with the existing image is an error, not a resize. Only delete_data_disk removes one |
sandbox_id | string | none | Caller-supplied sandbox identity, echoed in list_vms and used by lookup_sandbox / reattach_sandbox |
network | object | none | {cluster_id, vm_index?, pod_netns?} — join (or create) the shared bridge k7-br-{cluster_id}; vm_index pins the IP/MAC (fork pods reuse the source’s), pod_netns moves the TAP into a CNI pod netns. (0.9.0) Without pod_netns the bridge lives in its own network namespace /run/netns/k7-br-{cluster_id} behind a /30 uplink veth from 198.18.0.0/15; guests keep 10.200.0.0/24, so two networks with the same guest IPs (a fork and its source) both have egress. Pod-backed sandboxes never build the netns |
backend | native | firecracker | native | Which VMM owns the KVM fd: k7d’s in-process engine, or a stock Firecracker under the stock jailer (Backends). Firecracker rejects virtiofs and dilation > 1 with a named error rather than ignoring them |
docker | object | none | {graph_size_bytes} — run a pinned dockerd as an agent-supervised guest service with an overlay2 graph on a dedicated per-VM scratch disk (reflinked on fork, deleted with the VM). The shim sets it from k7d.katakate.org/docker / docker-disk |
dilation | u32 | 1 | Continuous time dilation: guest time runs this many times faster than wall time for the life of the VM. Every member of a cluster is created from one VmConfig, so a cluster is dilated in lockstep by construction. A host without the KVM dilation patch fails VM creation for dilation > 1 instead of silently running at 1× |
vm_created responses carry vm_id, guest_cid, vm_index (determines the guest IP 10.200.0.{2 + vm_index}), pod_netns_attached, and pooled (whether the VM was served from the warm pool instead of cold-booted). vm_list entries carry backend and, for Firecracker VMs, jail — {pid, uid, chroot} of the jailed VMM process, the only handles an operator has on it (k7d doctor prints them too).
Tree operations#
Snapshot-tree semantics (budgets, LRU eviction, protect/prune, keyframes) are explained in Snapshot tree; cluster specifics in Cluster mode.
op | Fields | What it does |
|---|---|---|
tree_create | config, budget?, tree_id?, dilation_policy?, dilation_config? | Root a single-VM tree at a freshly booted VM (config.network must be omitted). → tree_created |
tree_create_cluster | vm_count, config, budget?, tree_id?, egress?, dilation_policy?, dilation_config?, ksm? | Root a cluster tree: vm_count members on a tree-owned bridge. egress is the tree’s egress policy; forks inherit it. → tree_created |
tree_adopt_cluster | vm_ids, budget?, tree_id?, egress?, dilation_policy?, dilation_config?, ksm? | Adopt live daemon-owned VMs sharing one cluster bridge as a cluster-tree root. Fails loud on any half-wired state. The ids leave the sandbox registry (see stop_vm). → tree_created |
tree_fork | tree_id, source_id, label, dilation?, dilation_policy?, dilation_config? | Fork a new branch from a live or suspended node. dilation omitted inherits the source’s factor; 0 is a loud error. → tree_forked |
tree_fork_batch | tree_id, source_id, labels, dilation?, dilation_policy?, dilation_config? | Fork N branches in one call. Single-VM trees share one dirty-bitmap capture; cluster trees run sequential warm forks. Empty labels is a loud error. → tree_fork_batch |
tree_prepare_fork_source | tree_id, node_id, refresh? | Write a node’s keyframe now so it is a warm-fork source. tree_fork* calls this on demand; the explicit op lets a controller show the checkpoint as its own step. Idempotent unless refresh: true (0.8.0), which re-keyframes a node that already owns one into a new generation. See Refreshing a keyframe. → tree_prepared |
tree_rollback | tree_id, node_id, label | Non-destructive rollback: new branch from node_id’s state. → tree_forked |
tree_protect / tree_unprotect | tree_id, node_id | Mark / unmark a node as protected from auto-eviction. → ok |
tree_suspend | tree_id, node_id | Delta written to disk, RAM freed. → ok. (0.9.0) A second tree_suspend on an already-parked node is a BadState error, not “node not found” |
tree_resume | tree_id, node_id | Resume a suspended node. Members get fresh vsock CIDs. → tree_resumed |
tree_prune | tree_id, node_id | Delete a node and its whole subtree, disk included. (0.8.0) Pruning the root of an adopted tree also releases the adopted VMs’ disk pins (under K7D_PRIVSEP=on); a child prune does not. → ok |
tree_auto_evict | tree_id | Run budget-driven LRU eviction now. → tree_evicted with the evicted node ids |
tree_nodes | tree_id | Every node’s metadata plus usage, budget, live_guest_cids (node id → CIDs), live_agent_endpoints, and live_dilation (node id → factor). (0.8.0) Also dilation_policy (node id → mode), dilation_trajectory (node id → every factor the loop has run it at, oldest first), and live_ksm (node id → one {ksm_merging_pages, ksm_stat} per native member). (0.9.0) Also live_cpu (node id → one {member, vcpus, cpu_percent?} per native member, in member order; cpu_percent is 0..=100 of that member’s vCPUs and absent until two samples ≥ 200 ms apart exist; Firecracker members are omitted) and dilation_ladder (node id → the ladder stored for it, highest rung first; empty when this daemon never armed the node). → tree_node_list |
tree_list | — | All trees: {tree_id, kind ("vm"/"cluster"), nodes, usage, budget}. → tree_list |
tree_watch | tree_id? | Subscribe this connection to tree mutations. First reply is tree_watching (a snapshot per matched tree); then one tree_event line per fork / protect / unprotect / suspend / resume / prune / evict / drop until the client hangs up. The connection is event-only afterwards |
tree_drop | tree_id | Forget a tree: live guests stopped, on-disk tree state kept. (0.8.0) Releases an adopted tree’s disk pins. Caller-supplied block_devices images are never unlinked — deleting them is the caller’s job |
tree_open | tree_id | Re-open an on-disk tree left by tree_drop or a daemon crash. Every node comes back Suspended (live KVM is not resurrected) — tree_resume to go live. Loud error if tree.json is missing, unparseable, or the id is already in memory. → tree_opened |
TreeBudget#
| Field | Type | Default | Meaning |
|---|---|---|---|
max_live_vms | usize | 8 | Max concurrent live VMs |
max_live_ram_bytes | u64 | 8 GiB | Max non-reclaimable RAM across live VMs |
max_disk_bytes | u64 | 32 GiB | Max disk for keyframes + deltas across all nodes (physical bytes — an incremental keyframe counts only what it added) |
max_chain_depth | usize | 1 | Max restore-chain depth: non-keyframe hops from a node up to the nearest keyframe. Enforced on resume |
TreeUsage#
Carried on tree_nodes, tree_list, and tree_watching: live_vm_count, live_ram_bytes, disk_bytes, total_nodes, live_node_count.
EgressPolicy#
Per-tree posture for traffic leaving the tree’s bridge, enforced by the tree network jail. Forks inherit the source’s policy.
| Field | Meaning |
|---|---|
mode | open (default — WAN + CNI egress), deny (no WAN egress; intra-node CNI still forwarded), whitelist (only allow CIDR:port pairs, tcp+udp, plus allow_dns on port 53) |
allow | Whitelist entries {cidr, port}. Only meaningful with mode: whitelist — anything else is a config error |
allow_dns | IPv4 resolver allowed on udp/tcp 53 (whitelist only) |
169.254.0.0/16 (cloud metadata) is dropped in every mode unless a whitelist entry names it. RoCEv2 leaves as UDP/4791 — a whitelist that needs RDMA out of the tree must name that port.
Dilation policy fields#
These fields opt a tree node into the daemon’s adaptive dilation loop.
dilation_policy | Behaviour |
|---|---|
fixed (default) | No loop. The node keeps its create-time or fork-time factor |
down | Drop one rung when the cluster is saturated. Never raises |
track | Drop when saturated, raise one rung when quiet |
dilation_config sets the loop’s knobs. Omitted fields take the defaults below:
| Field | Default | Meaning |
|---|---|---|
window | 3 | Samples a decision rests on. Must be ≥ 1 |
saturated | 0.8 | Drop at this fraction of the cluster’s vCPUs awake, or more |
quiet | 0.45 | Raise at this fraction or less (track only). Must be below saturated |
cadence_ms | 1000 | Wall time between samples. Only the tree’s root sets it; a fork that sets it is refused |
ladder | [8, 4, 2, 1] | Factors the loop moves between, strictly decreasing, each in 1..=64 |
A fork that omits either field inherits the source node’s mode and knobs. down / track need the native backend and a factor that sits on the ladder (config.dilation at create, or the child’s factor at fork). Anything else is refused before a VM boots. fixed still validates dilation_config, so a child can inherit it later. The loop is not persisted, so a tree reopened with tree_open has no modes recorded.
KSM flag#
ksm: true on tree_create_cluster or tree_adopt_cluster marks every native member’s MAP_PRIVATE guest RAM MADV_MERGEABLE once the guest is running. Forks inherit it. The default is off. Firecracker members and members with virtiofs shares, whose RAM is MAP_SHARED, are refused with a named error. k7d never writes /sys/kernel/mm/ksm: an operator has to start ksmd, and doctor reports its state in ksm_run. Merged pages show up per member in tree_nodes.live_ksm. A tree that did not opt in reports zeros.
Same-tenant hosts only. KSM merges identical pages across every mergeable mapping on the host, including another tree that also opted in. That is a cross-VM timing side channel. The flag lasts as long as the mapping does, because MADV_UNMERGEABLE unmerges eagerly and can OOM the host.
Fork responses#
tree_created, tree_forked, and tree_resumed responses carry guest_cids — one vsock CID per payload member, ordered by vm_index (a single entry for single-VM trees). Fork responses additionally carry:
fork_memory_full_copy(per member):falsemeans the CoWMAP_PRIVATEfast path;truemeans a full memory copy (only virtiofs-backed sources — e.g. read-write hostPath — take that fallback).latency_ns: the daemon’s own wall time for the fork (or the whole batch). Budgets are asserted on this figure, not on the client’s round trip.dilation: the child’s clock factor after the fork.
tree_created for a native cluster tree also carries pause_ns (0.8.0), one entry per member: the host time that member’s guest did not run while the root keyframe was cut (pause, residual write, restore-swap). It also carries max_pause_ns. Both are measured values, not budgets, and are empty or zero for single-VM and Firecracker trees.
Keep the CIDs from these replies. Guest exec is vsock straight to a CID, and the vm_id from create_vm does not reach a tree fork. To get them back later, read tree_nodes.live_guest_cids[node_id] (or live_agent_endpoints on Firecracker). list_vms will not show them.
tree_prepared carries wrote_base (a new keyframe was written), wrote_base_incremental (it was a reflink clone of the nearest ancestor plus this node’s dirty pages, O(divergence) instead of O(guest RAM); a refresh is incremental against the node’s own previous keyframe), base_disk_bytes (the physical bytes it actually added — stat cannot tell you this on a reflinked file), latency_ns, and guest_cids.
Clock operations#
All clock ops act on every live VM of one cluster at once: the whole cluster is paused for the write and resumed after it. Concepts, numbers, and caveats: Time warp.
op | Fields | What it does |
|---|---|---|
warp_clock | cluster_id, delta_ms | Advance every member’s guest-visible clock by the same delta. Forward only; 0 is a loud error. All-or-nothing across members. → warped with vm_ids, delta_ms, latency_ns |
set_dilation | cluster_id, factor | Change the continuous dilation factor of every member, 1..=64, live. Lowering is not a rewind — guests keep the nanoseconds they already have. Needs the KVM dilation patch above 1. (0.9.0) cluster_id is matched against the longest tree id first, then against the single guest of a one-VM tree. → dilation_set |
set_dilation_policy (0.9.0) | tree_id, node_id?, mode, config? | Arm (or re-arm) the adaptive dilation loop on a tree after creation: mode is fixed / down / track, config the same knobs as dilation_config (module defaults when omitted). node_id omitted means every live node, or every node when mode is fixed. fixed stores the ladder and starts no thread — the caller still sets the rate with set_dilation. This is what k7view’s ladder presets and “fork N children” buttons call. → dilation_policy_set with tree_id, mode, nodes (ids armed, in list order), ladder |
auto_warp | cluster_id, mode | Start, switch, or stop the auto-warp policy loop: off, jump (when every vCPU is provably idle, jump to just before the earliest armed timer), or micro (fixed quantum, regardless of idleness). Setting a mode replaces the running loop; off stops it and is also how you collect a finished run’s counters. → auto_warp |
auto_warp_status | cluster_id | Read the counters without changing the mode. → auto_warp |
auto_warp_hold | cluster_id, holder, hold | Take (true) or release (false) a named hold that blocks the loop. Needed by any client driving a member from outside the daemon (e.g. an agent exec straight over vsock) — otherwise that guest sees its clock jump mid-syscall. Releasing a hold nobody took is an error |
cluster_id is the NetConfig.cluster_id given to create_vm, or for tree-owned clusters {tree_id}-{node_id} (tree_id alone for the root).
The auto_warp reply carries mode, holders, and stats: cycles, warps, guest_ns_added, wall_ns_paused, plus a breakdown of where the pause went (pause_ns, freeze_ns, stop_vcpus_ns, quiesce_ns, decide_ns, …).
Daemon environment#
| Variable | Default | Meaning |
|---|---|---|
K7D_SOCKET | /run/k7d/k7d.sock | Control socket (also --socket) |
K7D_KERNEL / K7D_INITRD | required | Default guest kernel and initramfs. The daemon refuses to create a VM without them |
K7D_TREE_ROOT | /var/lib/k7d/trees | Snapshot trees ({root}/{tree_id}/) |
K7D_SANDBOX_ROOT | /var/lib/k7d/sandboxes | Sandbox reattach records ({sandbox_id}.json) |
K7D_POOL_SIZE / K7D_POOL_MEMORY_MB / K7D_POOL_VCPUS | 0 / 256 / 1 | Warm pool: target idle VMs and the template they are booted from (0 disables the pool) |
K7D_POOL_BACKEND | native | Engine of the pooled VMs: native, firecracker, or a split native:N,firecracker:M (N+M must equal K7D_POOL_SIZE) |
K7D_FC_BINARY / K7D_FC_JAILER | /usr/local/bin/firecracker / /usr/local/bin/jailer | Firecracker and jailer executables (what utils/install-firecracker.sh installs) |
K7D_FC_CHROOT_BASE | /var/lib/k7d/fc | Root of every Firecracker jailer chroot; must sit on the reflink XFS that holds the writable disks (hard-linked into the jail) |
K7D_FC_UID_BASE | 60000 | First uid/gid leased to a jailed Firecracker VM; 1024 live VMs per daemon, exhaustion is an error |
K7D_FC_FORK_SNAPSHOT | diff | Firecracker fork point: diff (O(dirty) pages onto a reflinked base) or full (all of guest RAM; for cross-checking, not a fallback) |
K7D_ALLOWED_UIDS | — | Comma-separated uids allowed to connect to the control socket (SO_PEERCRED); unset means any uid that can open the 0660 socket |
K7D_DENY_VIRTIOFS_FALLBACK | — | Set to 1 to fail Create for any pod that would take the read-write virtiofs hostPath fallback (hardened / Firecracker profiles) |
K7D_NET_JAIL | enforce | Per-tree nftables jail. off skips installation — the escape hatch for debugging; any other value is a loud error |
K7D_NET_JAIL_CT_MAX / K7D_NET_JAIL_NEW_RATE | 16384 / 2000 | Per-tree concurrent-connection cap and new-connection rate (pkt/s). 0 is refused |
K7D_NET_JAIL_ALLOW_IFACES | — | Comma-separated extra host interfaces treated as intra-node (allowed in every egress mode), on top of detected CNI interfaces |
K7D_TREE_CGROUP | enforce | Per-tree cgroup jail (pids + memory, sized from the tree budget). off disables |
K7D_SECCOMP | enforce | Daemon-wide seccomp allowlist. log records instead of killing; off disables |
K7D_VIRTIOFSD_SANDBOX | chroot | virtiofsd sandbox mode (chroot / namespace / none). Anything else is a loud error |
Every unknown value above is a hard error rather than a silent fallback.
On-disk locations#
| Path | Contents |
|---|---|
/run/k7d/k7d.sock | Control socket |
/var/lib/k7d/trees | Snapshot trees ({tree_id}/tree.json, memory.bin keyframes, delta.pack deltas) |
/var/lib/k7d/sandboxes | Sandbox reattach records |
/var/lib/k7d/disks | A loop-mounted reflink-capable XFS image (/var/lib/k7d/disks.img, 96 GiB sparse) holding every writable image: scratch disks, per-container scratch, named data disks (data/<id>.img), Ubuntu node root disks. Reflinks require it — a host without reflink support fails loudly rather than falling back to byte copies |
