Snapshots & fork

The kata-qemu-longhorn backend supports disk-level snapshots of running sandboxes via Longhorn’s CSI VolumeSnapshot API. This unlocks three workflows:

• Pause — scale to 0 and (optionally) snapshot the disk for safekeeping.
• Resume — scale back to 1; the same disk re-attaches.
• Fork — clone a sandbox’s disk into a brand-new sandbox with its own pod, IP, and lifecycle.

Anatomy of a kata-qemu-longhorn sandbox#

When you create a kata-qemu-longhorn sandbox, k7 provisions:

• A Longhorn PVC named <sandbox>-root-lh, sized via --root-disk-size (default 10Gi).
• A Deployment with one replica, mounting the PVC at /mnt/state (with persistence wrapper bind-mounts).
• An ingress NetworkPolicy (deny-all unless ingress_ports is set) and an egress policy (CIDR or FQDN).
• An optional Secret from env_file.
• With --docker, a second Longhorn PVC <sandbox>-docker-lh holding Docker’s overlay2 graph (default 20Gi, --docker-disk). It is snapshotted, cloned and re-attached alongside the root PVC on pause/fork/restore.

State written to /mnt/state (and to the rootfs via the bind-mount overlay) survives pod restarts because it lives on the PVC.

Pause#

k7 pause demo                           # scale to 0, no snapshotk7 pause demo --snapshot                # auto-named <sandbox>-paused-<unix-ts>k7 pause demo --snapshot=demo-snap-1  # take a VolumeSnapshot, then scale to 0

--snapshot is the only pause flag besides -n. Bare --snapshot auto-names; --snapshot=NAME uses your name. There is no --pvc or --snapshot-class on the CLI (those were removed). Restore is k7 restore SNAPSHOT_NAME NEW_SANDBOX_NAME — there is no --latest.

The snapshot is crash-consistent: it captures the disk state at the moment of the snapshot, not the in-memory process state. The sandbox process is then terminated as the Deployment scales to 0.

Earlier releases could hand a kql fork a half-cloned disk on a replicated cluster (the child failed fsck and never went Ready): the source VolumeSnapshot was deleted while Longhorn was still copying it. Since 0.5.0 the snapshot is only removed once the Longhorn clone reports completed, so a fork of a busy volume comes up clean. The clone is still crash-consistent — pause the sandbox first when you need a quiesced copy, and prefer k7d warm fork when latency matters.

The snapshot is stored as a VolumeSnapshot object in the same namespace and persists until you delete it (kubectl delete volumesnapshot <name>).

Resume#

k7 resume demo

Scales the Deployment back to 1. The pod re-attaches to the same PVC; data on /mnt/state is exactly as it was at pause time. If the pause took a snapshot, the snapshot remains available — useful for rollback if the resumed workload corrupts something.

Wait until Ready before k7 exec. On a 3-node HA cluster Longhorn attach can leave the pod Pending for tens of seconds after k7 resume returns. Poll k7 list until Ready is True (the same wait you need after Client.resume() / k7 restore).

k7 snapshot list may show READY False for a few seconds after k7 snapshot create; wait for readyToUse: true before k7 restore.

Fork#

k7 fork demo demo-clonek7 fork demo demo-clone --snapshot custom-name

Fork takes a snapshot of the source sandbox’s PVC, clones it into a new PVC, and creates a new Deployment pointing at the cloned PVC. Implementation steps:

1. Create a VolumeSnapshot of <source>-root-lh (auto-named or --snapshot).
2. Wait for readyToUse: true on the snapshot.
3. Create a new PVC <new>-root-lh from the snapshot via dataSource.
4. Create a Deployment for the new sandbox using the cloned PVC.
5. Wait for the new pod to be Ready.
6. Delete the temporary snapshot (the cloned PVC is independent of it).

Scope: disk only. Fork does not clone:

• Memory state (the new sandbox boots fresh)
• CPU registers / process state
• Network identity (the fork gets a new pod IP)

What a fork inherits on every backend: the source’s egress whitelist and its ingress_ports / ingress_from policy, so a child is never more reachable than its parent (an earlier release left forks fully ingress-closed). It does not inherit an expose_ports NodePort Service — expose the child explicitly if you need it. A --docker source forks with its graph disk (kql: cloned PVC; k7d / k7d-fc: reflinked scratch disk, dockerd keeps running with overlay2); kfd rejects fork up front with a direct “kfd cannot fork” error (kfd has no Longhorn PVC). With k7d 0.9.0+ a k7d / k7d-fc fork also keeps a working NIC (own bridge, NAT), so the child has egress like its parent; older k7d forks had none. In production mode a fork counts against the tenant’s sandboxes / vcpu / memory quota like a create (403 QuotaExceeded).

Observed performance#

Single-node medians (Hetzner AX41, Longhorn replicas=1, 2026-08-10) — see PERFORMANCE.md:

On a two-node r=2 cluster (2026-08-18 Show HN cut) kql fork is ~83 s; k7d fork stays ~4 s and is the only backend whose fork inherits live memory (/tmp markers, running processes). k7d-fc (k7d driving stock Firecracker) sits close to k7d: create → Ready 2.1 s and fork → exec ~2.45 s on the same node (2026-09-10). Forking a warm --docker engine takes ~6.8 s (k7d) / ~9.1 s (k7d-fc) to Ready with overlay2 still active in the child — see Docker in a sandbox.

Example: a parallel exploration with fork#

A common AI workflow is to set up a heavy environment once, then explore many branches from it.

from k7_sdk import Clientk7 = Client(endpoint="https://node:31007", api_key="...", verify_ssl="./k7-ca.crt")# Step 1: prepare a base sandbox with a big environmentbase = k7.create({    "name": "base",    "image": "python:3.12-bookworm",    "backend": "kata-qemu-longhorn",    "root_disk_size": "20Gi",    "before_script": (        "apt-get update && "        "apt-get install -y build-essential && "        "pip install torch transformers datasets"    ),    "egress_whitelist": ["pypi.org", "files.pythonhosted.org", "huggingface.co"],})# (Wait until base is Ready; do whatever setup you want)# Step 2: fork the base for each parallel experimentforks = [base.fork(f"exp-{i}") for i in range(8)]for f in forks:    f.exec("python -c 'print(\"branch\", __import__(\"os\").getpid())'")

Or from the CLI on a node with API access:

for i in $(seq 0 7); do  k7 fork base exp-$i &donewait

Each exp-N boots from a clone of base’s disk — the heavy pip install doesn’t run again.

Cleanup#

k7 delete demo removes the Deployment, the PVCs (root and, with --docker, the graph disk), and any pause/fork snapshots associated with the sandbox. k7 snapshot gc reaps unreferenced snapshots and now also deletes orphan VolumeSnapshotContent objects and clears the PVC finalizer that used to leave Longhorn volumes stuck. To inspect what’s still around:

sudo k3s kubectl get pvc,volumesnapshot -A | grep demosudo k3s kubectl get volumes.longhorn.io,replicas.longhorn.io -n longhorn-system

Integration tests under tests/integration/ exercise pause/resume/fork end-to-end and detect orphaned Longhorn replicas or stuck PVCs.

Reference#

• Performance baseline: PERFORMANCE.md
• Integration tests: tests/integration/