Skip to main content
If you already have Kubernetes tasks or scenarios (a Helm chart, a set of YAML manifests, an eval harness that talks to a kube-apiserver), the shape is:
  1. Boot the scenario once — bring up your cluster (or adopt a running one that k7d already hosts) and wait until it is in the state you want every rollout to start from.
  2. Root a tree at that checkpoint.
  3. For each GRPO group (or tree-search step): tree_fork_batch(N) → run your N policies against the N copies → score → tree_protect the winners, tree_prune the losers → let the daemon tree_auto_evict under your RAM/disk budget.
  4. Roll forward from a protected winner when you want the next generation to start from a better state, or tree_rollback to an earlier node when you don’t.
Your training loop owns rewards and policy. k7d owns environments and budgets.

​
Why byte-identical starts matter for GRPO

GRPO (and most group-relative methods) compare rewards within a group. If member A starts from a colder cache, a different etcd revision, or a half-ready Deployment than member B, the reward gap is noise, not signal. A k7d fork is a copy of the live machine — same memory, same disk, same in-cluster TLS sessions, same kube-apiserver state. Every member of the group begins from a byte-identical world, then diverges only because of what your policy did. That is the difference between “we reset the env” and “we cloned the universe.”

​
How your agent talks to the tree

The agent talks JSON-lines over a Unix socket (/run/k7d/k7d.sock). The verbs you actually need: Full request/response shapes: Daemon API. Budget semantics and LRU eviction: Snapshot tree.

​
Three reference clients

All three live in the repo’s examples/ and talk to the daemon over the socket. None is a finished SDK; each is the template for one way of driving the tree.

​
cluster-tree-search/ — the trainer loop

A thin Python client that covers exactly the loop above:
The demo forks N branches from a live checkpoint, scores a trivial reward, protects the winner, prunes the losers, and prints the fork wall-clock. Useful flags: --keep (leave the tree around for tree_nodes inspection), --tree-id NAME (stable tree id), K7D_SOCKET (override the control socket).

​
k7d-mcp/ — a frontier agent with a fork button

A stdio MCP server that wraps the tree for Cursor CLI / Claude Code / Codex-style agents. The agent gets tree_status, fork_batch(node_id, n), branch_exec(node_id, member, cmd), tetragon_diff(node_id), and protect / prune / rollback — every call appends one JSONL line (tool, args, latency, tree usage). The shipped scenario plants a self-healing miner in a live Ubuntu Cilium + Tetragon cluster. The agent does not touch production first: it forks the cluster three ways, tries delete / scale-to-zero / watch on the copies, learns the miner respawns, forks again from the copy that already knows that, and replays only the winning pair of moves on the source. One lab is left broken on purpose; no rollback needed. Video · how it works · JSONL.

​
k7view/ — the observer

A read-only dashboard: one SSE stream over tree_list / tree_nodes / tree_watch, a vsock probe thread that pings live agents for real RTT and /proc/loadavg, click-to-protect / evict. Fork latency chips are the daemon’s latency_ns; the UI never invents a number or a parent edge. The incident seen from the tree · a ×8 dilated tree recorded in real time.

​
Sizing the loop

  • A single 3-VM cluster fork is ~105 ms on the minimal CI guest under API churn and ~1.1 s on Ubuntu k3s + Cilium + Tetragon; a shared-pause batch of 50 CI-guest clusters is ~3.85 s (~77 ms/cluster). See Benchmarks.
  • Forks share memory until they diverge — RAM cost per group member is the dirty pages of its own rollout, not full guest RAM. Three idle k3s nodes dirty ~20% of guest RAM in 30 s (page-cache churn), so plan for that floor.
  • Forking from a child costs a keyframe first: O(divergence), ~0.8 s idle / ~1.1 s under Kafka-style churn for 3 × 2 GiB. Fork from the root or from protected checkpoints when you can; tree_prepare_fork_source lets you pay it at a moment of your choosing.
  • Set the tree budget to your box: max_live_vms, max_live_ram_bytes, max_disk_bytes. The daemon proactively suspends least-recently-active unprotected nodes at 85% of the RAM budget, so a long training run doesn’t OOM the host. tree_nodes and tree_list report live usage against the budget.
  • If an episode is mostly waiting, run the tree with auto_warp jump (zero fidelity cost, 1.83× on a probe-heavy episode) or fork children at dilation: 8 (8× on the same episode, bounded fidelity cost). Time warp has the trade-off table.