Skip to main content

The k7d daemon exposes its control plane on a Unix socket, by default /run/k7d/k7d.sock (override with --socket or the K7D_SOCKET env var).

Protocol: one JSON request object per line, one JSON response object per line. A connection may carry multiple request/response pairs. Requests are tagged with "op", responses with "status". Errors are always an explicit {"status": "error", "message": "..."} response — never a silent success.

Note

Clients reach the guest agent themselves over vsock, using the guest_cid returned by create_vm — the daemon does not proxy the agent protocol (vsock CIDs are a kernel-global namespace, so any host process can connect to a running VM’s agent).

Note

Fields and behaviour marked (0.8.0) shipped in 0.8.0 and were not in the 0.7.0 tarball: dilation_policy / dilation_config, ksm, refresh, pause_ns, the adopted-VM teardown rules, and the new tree_nodes / doctor fields. Those marked (0.9.0) are new in 0.9.0: fork_vm.pod_netns, the set_dilation_policy op, tree_nodes.live_cpu / dilation_ladder, and the per-network namespace behind every host bridge.

Reference clients: a thin Python client covering the tree verbs in examples/cluster-tree-search/k7d_client.py, an MCP server wrapping the tree for a frontier agent in examples/k7d-mcp/, and a read-only SSE dashboard in examples/k7view/.

Host operations#

opFieldsWhat it does
ping—Liveness check. → pong with the daemon version
doctor—Host readiness: /dev/kvm, /dev/vhost-vsock, /dev/net/tun, cgroup v2. Same checks as k7d doctor; creates no VM. → doctor with kvm, vhost_vsock, tun, cgroup_v2, ok, fc_fork_mode (diff | full, from K7D_FC_FORK_SNAPSHOT), ksm_run (0.8.0), details[]. ksm_run is /sys/kernel/mm/ksm/run read as-is (0 stopped, 1 scanning, 2 unmerging, absent without KSM). It is advisory and does not affect ok
pool_status—Warm-pool sizing and hit/miss counters. Always answers, even with the pool disabled (size: 0)

VM lifecycle operations#

opFieldsWhat it does
create_vmconfig (VmConfig)Create and boot a VM. → vm_created
vm_statusvm_idQuery a VM’s state. → vm_state with state, guest_cid, virtiofs_shares, block_devices, fork_memory_full_copy, and the kernel it actually booted
list_vms—List the daemon’s sandbox registry: create_vm and fork_vm VMs. Tree members (adopted, tree_create*, tree_fork*) are not listed; find those with tree_list / tree_nodes. → vm_list
stop_vmvm_idShut down and forget a VM. Ephemeral disks go with it; named data disks stay. → ok. (0.8.0) On an id that tree_adopt_cluster took, it errors with VM … was adopted into tree …; use tree_prune / tree_drop and frees nothing. Unknown ids still return no such VM
pause_vm / resume_vmvm_idStop / restart the vCPU threads and park device workers; guest memory, devices, and the vsock CID are retained. Idempotent. → ok
dump_serialvm_id or guest_cidRead a VM’s accumulated serial console output (guest kernel messages, agent stderr). The guest_cid form also finds live VMs inside snapshot trees; vm_id: "{tree_id}/{node_id}" works too. A fork’s console buffer starts empty, so dump the source before forking if you need its pre-fork output. → serial_dump
fork_vmvm_id, sandbox_id?, pod_netns?, dilation?Warm-fork a daemon-owned VM into a new daemon-owned VM. The first fork of a VM prepares it as a fork source on the same vsock CID (echoed as source_guest_cid). sandbox_id registers the fork like a create_vm-born sandbox VM. (0.9.0) A networked source’s fork gets its own bridge with a TAP at the source’s vm_index (the guest kept that IP/MAC in memory); pod_netns attaches that TAP into the fork pod’s CNI netns with NAT, so the fork pod’s NetworkPolicy governs it. A pod-backed fork (sandbox_id set) of a pod-attached source must name pod_netns — it fails loudly otherwise. → vm_forked with vm_id, guest_cid, source_guest_cid, dilation
delete_data_diskidDelete a named persistent data disk’s image — the only way its data goes away. Loud error if the image is missing or a live VM still has it attached
update_drivevm_id, device_index, host_pathFirecracker only: re-point block device i (/dev/vd{a+i}) at a new host file via PATCH /drives — how image slots are filled after boot. Sending it to a native VM is an error, not a no-op
lookup_sandboxsandbox_idResolve a CRI sandbox to its live VM. → sandbox_found
reattach_sandboxsandbox_idLike lookup_sandbox, but verifies the guest agent still answers before handing the VM back to a respawned shim. Dead VMs are torn down; stale records garbage-collected (both return error)

VmConfig#

The config object accepted by create_vm (and the tree-create operations):

FieldTypeDefaultMeaning
kernelstringenv K7D_KERNELGuest kernel path
initrdstringenv K7D_INITRDInitramfs path. Omitted on a disk-booted guest (cmdline with root=) means no initrd — never the busybox default
cmdlinestringbusybox defaultReplacement kernel command line. A guest with a different init contract (an Ubuntu node with root=/dev/vda) must set it; there is no merge
memory_mbu64256Guest RAM in MiB
vcpusu321Guest vCPU count. Multi-vCPU (SMP) guests are supported; the shim derives the count from the pod’s CPU limit (ceil(quota/period), floor 1). Fork/snapshot have full parity across vCPU counts
cpu_quota / cpu_periodu64noneHost CFS quota in µs per period, applied to the VM’s vCPU threads. Both set or neither
vsockbooltrueAttach a vhost-vsock device (needed for the agent)
virtiofsarray[]virtiofs shares: {host_dir, guest_mount, readonly}
block_devicesarray[]virtio-blk images in device order (/dev/vda, …): {host_path, readonly, guest_mount?, fstype?}. guest_mount + fstype (e.g. erofs, ext4) makes the agent mount the device before containers start — both set or both absent, validated loudly
guest_dirsarray[]Absolute guest tmpfs directories the agent creates at boot (emptyDir backing)
scratch_size_bytesu64noneCapacity of the sandbox’s writable scratch disk — a sparse ext4 image the daemon provisions and attaches as the last read-write virtio-blk device; every container’s overlay upper lives on it. None = a small, bounded tmpfs, said loudly. The shim always sets it (default 8 GiB via k7d.katakate.org/scratch-size)
container_scratcharray[]{name, size_bytes} — Kubernetes container names that get their own writable image instead of a directory on the shared scratch. Each entry costs one virtio-blk device slot
data_disksarray[]{id, size_bytes, guest_mount} — named persistent ext4 disks that outlive the VM. Created and formatted on first use, reused as-is on later attaches; a size_bytes that disagrees with the existing image is an error, not a resize. Only delete_data_disk removes one
sandbox_idstringnoneCaller-supplied sandbox identity, echoed in list_vms and used by lookup_sandbox / reattach_sandbox
networkobjectnone{cluster_id, vm_index?, pod_netns?} — join (or create) the shared bridge k7-br-{cluster_id}; vm_index pins the IP/MAC (fork pods reuse the source’s), pod_netns moves the TAP into a CNI pod netns. (0.9.0) Without pod_netns the bridge lives in its own network namespace /run/netns/k7-br-{cluster_id} behind a /30 uplink veth from 198.18.0.0/15; guests keep 10.200.0.0/24, so two networks with the same guest IPs (a fork and its source) both have egress. Pod-backed sandboxes never build the netns
backendnative | firecrackernativeWhich VMM owns the KVM fd: k7d’s in-process engine, or a stock Firecracker under the stock jailer (Backends). Firecracker rejects virtiofs and dilation > 1 with a named error rather than ignoring them
dockerobjectnone{graph_size_bytes} — run a pinned dockerd as an agent-supervised guest service with an overlay2 graph on a dedicated per-VM scratch disk (reflinked on fork, deleted with the VM). The shim sets it from k7d.katakate.org/docker / docker-disk
dilationu321Continuous time dilation: guest time runs this many times faster than wall time for the life of the VM. Every member of a cluster is created from one VmConfig, so a cluster is dilated in lockstep by construction. A host without the KVM dilation patch fails VM creation for dilation > 1 instead of silently running at 1×

vm_created responses carry vm_id, guest_cid, vm_index (determines the guest IP 10.200.0.{2 + vm_index}), pod_netns_attached, and pooled (whether the VM was served from the warm pool instead of cold-booted). vm_list entries carry backend and, for Firecracker VMs, jail — {pid, uid, chroot} of the jailed VMM process, the only handles an operator has on it (k7d doctor prints them too).

Tree operations#

Snapshot-tree semantics (budgets, LRU eviction, protect/prune, keyframes) are explained in Snapshot tree; cluster specifics in Cluster mode.

opFieldsWhat it does
tree_createconfig, budget?, tree_id?, dilation_policy?, dilation_config?Root a single-VM tree at a freshly booted VM (config.network must be omitted). → tree_created
tree_create_clustervm_count, config, budget?, tree_id?, egress?, dilation_policy?, dilation_config?, ksm?Root a cluster tree: vm_count members on a tree-owned bridge. egress is the tree’s egress policy; forks inherit it. → tree_created
tree_adopt_clustervm_ids, budget?, tree_id?, egress?, dilation_policy?, dilation_config?, ksm?Adopt live daemon-owned VMs sharing one cluster bridge as a cluster-tree root. Fails loud on any half-wired state. The ids leave the sandbox registry (see stop_vm). → tree_created
tree_forktree_id, source_id, label, dilation?, dilation_policy?, dilation_config?Fork a new branch from a live or suspended node. dilation omitted inherits the source’s factor; 0 is a loud error. → tree_forked
tree_fork_batchtree_id, source_id, labels, dilation?, dilation_policy?, dilation_config?Fork N branches in one call. Single-VM trees share one dirty-bitmap capture; cluster trees run sequential warm forks. Empty labels is a loud error. → tree_fork_batch
tree_prepare_fork_sourcetree_id, node_id, refresh?Write a node’s keyframe now so it is a warm-fork source. tree_fork* calls this on demand; the explicit op lets a controller show the checkpoint as its own step. Idempotent unless refresh: true (0.8.0), which re-keyframes a node that already owns one into a new generation. See Refreshing a keyframe. → tree_prepared
tree_rollbacktree_id, node_id, labelNon-destructive rollback: new branch from node_id’s state. → tree_forked
tree_protect / tree_unprotecttree_id, node_idMark / unmark a node as protected from auto-eviction. → ok
tree_suspendtree_id, node_idDelta written to disk, RAM freed. → ok. (0.9.0) A second tree_suspend on an already-parked node is a BadState error, not “node not found”
tree_resumetree_id, node_idResume a suspended node. Members get fresh vsock CIDs. → tree_resumed
tree_prunetree_id, node_idDelete a node and its whole subtree, disk included. (0.8.0) Pruning the root of an adopted tree also releases the adopted VMs’ disk pins (under K7D_PRIVSEP=on); a child prune does not. → ok
tree_auto_evicttree_idRun budget-driven LRU eviction now. → tree_evicted with the evicted node ids
tree_nodestree_idEvery node’s metadata plus usage, budget, live_guest_cids (node id → CIDs), live_agent_endpoints, and live_dilation (node id → factor). (0.8.0) Also dilation_policy (node id → mode), dilation_trajectory (node id → every factor the loop has run it at, oldest first), and live_ksm (node id → one {ksm_merging_pages, ksm_stat} per native member). (0.9.0) Also live_cpu (node id → one {member, vcpus, cpu_percent?} per native member, in member order; cpu_percent is 0..=100 of that member’s vCPUs and absent until two samples ≥ 200 ms apart exist; Firecracker members are omitted) and dilation_ladder (node id → the ladder stored for it, highest rung first; empty when this daemon never armed the node). → tree_node_list
tree_list—All trees: {tree_id, kind ("vm"/"cluster"), nodes, usage, budget}. → tree_list
tree_watchtree_id?Subscribe this connection to tree mutations. First reply is tree_watching (a snapshot per matched tree); then one tree_event line per fork / protect / unprotect / suspend / resume / prune / evict / drop until the client hangs up. The connection is event-only afterwards
tree_droptree_idForget a tree: live guests stopped, on-disk tree state kept. (0.8.0) Releases an adopted tree’s disk pins. Caller-supplied block_devices images are never unlinked — deleting them is the caller’s job
tree_opentree_idRe-open an on-disk tree left by tree_drop or a daemon crash. Every node comes back Suspended (live KVM is not resurrected) — tree_resume to go live. Loud error if tree.json is missing, unparseable, or the id is already in memory. → tree_opened

TreeBudget#

FieldTypeDefaultMeaning
max_live_vmsusize8Max concurrent live VMs
max_live_ram_bytesu648 GiBMax non-reclaimable RAM across live VMs
max_disk_bytesu6432 GiBMax disk for keyframes + deltas across all nodes (physical bytes — an incremental keyframe counts only what it added)
max_chain_depthusize1Max restore-chain depth: non-keyframe hops from a node up to the nearest keyframe. Enforced on resume

TreeUsage#

Carried on tree_nodes, tree_list, and tree_watching: live_vm_count, live_ram_bytes, disk_bytes, total_nodes, live_node_count.

EgressPolicy#

Per-tree posture for traffic leaving the tree’s bridge, enforced by the tree network jail. Forks inherit the source’s policy.

FieldMeaning
modeopen (default — WAN + CNI egress), deny (no WAN egress; intra-node CNI still forwarded), whitelist (only allow CIDR:port pairs, tcp+udp, plus allow_dns on port 53)
allowWhitelist entries {cidr, port}. Only meaningful with mode: whitelist — anything else is a config error
allow_dnsIPv4 resolver allowed on udp/tcp 53 (whitelist only)

169.254.0.0/16 (cloud metadata) is dropped in every mode unless a whitelist entry names it. RoCEv2 leaves as UDP/4791 — a whitelist that needs RDMA out of the tree must name that port.

Dilation policy fields#

These fields opt a tree node into the daemon’s adaptive dilation loop.

dilation_policyBehaviour
fixed (default)No loop. The node keeps its create-time or fork-time factor
downDrop one rung when the cluster is saturated. Never raises
trackDrop when saturated, raise one rung when quiet

dilation_config sets the loop’s knobs. Omitted fields take the defaults below:

FieldDefaultMeaning
window3Samples a decision rests on. Must be ≥ 1
saturated0.8Drop at this fraction of the cluster’s vCPUs awake, or more
quiet0.45Raise at this fraction or less (track only). Must be below saturated
cadence_ms1000Wall time between samples. Only the tree’s root sets it; a fork that sets it is refused
ladder[8, 4, 2, 1]Factors the loop moves between, strictly decreasing, each in 1..=64

A fork that omits either field inherits the source node’s mode and knobs. down / track need the native backend and a factor that sits on the ladder (config.dilation at create, or the child’s factor at fork). Anything else is refused before a VM boots. fixed still validates dilation_config, so a child can inherit it later. The loop is not persisted, so a tree reopened with tree_open has no modes recorded.

KSM flag#

ksm: true on tree_create_cluster or tree_adopt_cluster marks every native member’s MAP_PRIVATE guest RAM MADV_MERGEABLE once the guest is running. Forks inherit it. The default is off. Firecracker members and members with virtiofs shares, whose RAM is MAP_SHARED, are refused with a named error. k7d never writes /sys/kernel/mm/ksm: an operator has to start ksmd, and doctor reports its state in ksm_run. Merged pages show up per member in tree_nodes.live_ksm. A tree that did not opt in reports zeros.

Warning

Same-tenant hosts only. KSM merges identical pages across every mergeable mapping on the host, including another tree that also opted in. That is a cross-VM timing side channel. The flag lasts as long as the mapping does, because MADV_UNMERGEABLE unmerges eagerly and can OOM the host.

Fork responses#

tree_created, tree_forked, and tree_resumed responses carry guest_cids — one vsock CID per payload member, ordered by vm_index (a single entry for single-VM trees). Fork responses additionally carry:

  • fork_memory_full_copy (per member): false means the CoW MAP_PRIVATE fast path; true means a full memory copy (only virtiofs-backed sources — e.g. read-write hostPath — take that fallback).
  • latency_ns: the daemon’s own wall time for the fork (or the whole batch). Budgets are asserted on this figure, not on the client’s round trip.
  • dilation: the child’s clock factor after the fork.

tree_created for a native cluster tree also carries pause_ns (0.8.0), one entry per member: the host time that member’s guest did not run while the root keyframe was cut (pause, residual write, restore-swap). It also carries max_pause_ns. Both are measured values, not budgets, and are empty or zero for single-VM and Firecracker trees.

Keep the CIDs from these replies. Guest exec is vsock straight to a CID, and the vm_id from create_vm does not reach a tree fork. To get them back later, read tree_nodes.live_guest_cids[node_id] (or live_agent_endpoints on Firecracker). list_vms will not show them.

tree_prepared carries wrote_base (a new keyframe was written), wrote_base_incremental (it was a reflink clone of the nearest ancestor plus this node’s dirty pages, O(divergence) instead of O(guest RAM); a refresh is incremental against the node’s own previous keyframe), base_disk_bytes (the physical bytes it actually added — stat cannot tell you this on a reflinked file), latency_ns, and guest_cids.

Clock operations#

All clock ops act on every live VM of one cluster at once: the whole cluster is paused for the write and resumed after it. Concepts, numbers, and caveats: Time warp.

opFieldsWhat it does
warp_clockcluster_id, delta_msAdvance every member’s guest-visible clock by the same delta. Forward only; 0 is a loud error. All-or-nothing across members. → warped with vm_ids, delta_ms, latency_ns
set_dilationcluster_id, factorChange the continuous dilation factor of every member, 1..=64, live. Lowering is not a rewind — guests keep the nanoseconds they already have. Needs the KVM dilation patch above 1. (0.9.0) cluster_id is matched against the longest tree id first, then against the single guest of a one-VM tree. → dilation_set
set_dilation_policy (0.9.0)tree_id, node_id?, mode, config?Arm (or re-arm) the adaptive dilation loop on a tree after creation: mode is fixed / down / track, config the same knobs as dilation_config (module defaults when omitted). node_id omitted means every live node, or every node when mode is fixed. fixed stores the ladder and starts no thread — the caller still sets the rate with set_dilation. This is what k7view’s ladder presets and “fork N children” buttons call. → dilation_policy_set with tree_id, mode, nodes (ids armed, in list order), ladder
auto_warpcluster_id, modeStart, switch, or stop the auto-warp policy loop: off, jump (when every vCPU is provably idle, jump to just before the earliest armed timer), or micro (fixed quantum, regardless of idleness). Setting a mode replaces the running loop; off stops it and is also how you collect a finished run’s counters. → auto_warp
auto_warp_statuscluster_idRead the counters without changing the mode. → auto_warp
auto_warp_holdcluster_id, holder, holdTake (true) or release (false) a named hold that blocks the loop. Needed by any client driving a member from outside the daemon (e.g. an agent exec straight over vsock) — otherwise that guest sees its clock jump mid-syscall. Releasing a hold nobody took is an error

cluster_id is the NetConfig.cluster_id given to create_vm, or for tree-owned clusters {tree_id}-{node_id} (tree_id alone for the root).

The auto_warp reply carries mode, holders, and stats: cycles, warps, guest_ns_added, wall_ns_paused, plus a breakdown of where the pause went (pause_ns, freeze_ns, stop_vcpus_ns, quiesce_ns, decide_ns, …).

Daemon environment#

VariableDefaultMeaning
K7D_SOCKET/run/k7d/k7d.sockControl socket (also --socket)
K7D_KERNEL / K7D_INITRDrequiredDefault guest kernel and initramfs. The daemon refuses to create a VM without them
K7D_TREE_ROOT/var/lib/k7d/treesSnapshot trees ({root}/{tree_id}/)
K7D_SANDBOX_ROOT/var/lib/k7d/sandboxesSandbox reattach records ({sandbox_id}.json)
K7D_POOL_SIZE / K7D_POOL_MEMORY_MB / K7D_POOL_VCPUS0 / 256 / 1Warm pool: target idle VMs and the template they are booted from (0 disables the pool)
K7D_POOL_BACKENDnativeEngine of the pooled VMs: native, firecracker, or a split native:N,firecracker:M (N+M must equal K7D_POOL_SIZE)
K7D_FC_BINARY / K7D_FC_JAILER/usr/local/bin/firecracker / /usr/local/bin/jailerFirecracker and jailer executables (what utils/install-firecracker.sh installs)
K7D_FC_CHROOT_BASE/var/lib/k7d/fcRoot of every Firecracker jailer chroot; must sit on the reflink XFS that holds the writable disks (hard-linked into the jail)
K7D_FC_UID_BASE60000First uid/gid leased to a jailed Firecracker VM; 1024 live VMs per daemon, exhaustion is an error
K7D_FC_FORK_SNAPSHOTdiffFirecracker fork point: diff (O(dirty) pages onto a reflinked base) or full (all of guest RAM; for cross-checking, not a fallback)
K7D_ALLOWED_UIDS—Comma-separated uids allowed to connect to the control socket (SO_PEERCRED); unset means any uid that can open the 0660 socket
K7D_DENY_VIRTIOFS_FALLBACK—Set to 1 to fail Create for any pod that would take the read-write virtiofs hostPath fallback (hardened / Firecracker profiles)
K7D_NET_JAILenforcePer-tree nftables jail. off skips installation — the escape hatch for debugging; any other value is a loud error
K7D_NET_JAIL_CT_MAX / K7D_NET_JAIL_NEW_RATE16384 / 2000Per-tree concurrent-connection cap and new-connection rate (pkt/s). 0 is refused
K7D_NET_JAIL_ALLOW_IFACES—Comma-separated extra host interfaces treated as intra-node (allowed in every egress mode), on top of detected CNI interfaces
K7D_TREE_CGROUPenforcePer-tree cgroup jail (pids + memory, sized from the tree budget). off disables
K7D_SECCOMPenforceDaemon-wide seccomp allowlist. log records instead of killing; off disables
K7D_VIRTIOFSD_SANDBOXchrootvirtiofsd sandbox mode (chroot / namespace / none). Anything else is a loud error

Every unknown value above is a hard error rather than a silent fallback.

On-disk locations#

PathContents
/run/k7d/k7d.sockControl socket
/var/lib/k7d/treesSnapshot trees ({tree_id}/tree.json, memory.bin keyframes, delta.pack deltas)
/var/lib/k7d/sandboxesSandbox reattach records
/var/lib/k7d/disksA loop-mounted reflink-capable XFS image (/var/lib/k7d/disks.img, 96 GiB sparse) holding every writable image: scratch disks, per-container scratch, named data disks (data/<id>.img), Ubuntu node root disks. Reflinks require it — a host without reflink support fails loudly rather than falling back to byte copies