Snapshots & fork

The kata-qemu-longhorn backend supports disk-level snapshots of running sandboxes via Longhorn’s CSI VolumeSnapshot API. This unlocks three workflows:

Pause — scale to 0 and (optionally) snapshot the disk for safekeeping.
Resume — scale back to 1; the same disk re-attaches.
Fork — clone a sandbox’s disk into a brand-new sandbox with its own pod, IP, and lifecycle.

Disk-level k7 snapshot requires the kata-qemu-longhorn backend. The kata-firecracker-devmapper backend uses ephemeral devmapper LVs and doesn’t support persistent snapshots or fork. On k7d and k7d-fc, k7 pause / k7 resume / k7 fork are VM-level (memory + disk, no VolumeSnapshot) and k7 snapshot is rejected loudly.

Looking for memory-faithful or whole-cluster forking? That’s the k7d runtime, not k7’s Longhorn-based fork. k7d warm-forks a running VM in ~5 ms (memory, disk, processes, and network identity all survive) and can fork an entire live Kubernetes cluster in ~100 ms — see CoW fork and Cluster mode. The fork described on this page is disk-only: the clone boots fresh from a copy of the source’s disk.

Anatomy of a `kata-qemu-longhorn` sandbox

When you create a kata-qemu-longhorn sandbox, k7 provisions:

A Longhorn PVC named <sandbox>-root-lh, sized via --root-disk-size (default 10Gi).
A Deployment with one replica, mounting the PVC at /mnt/state (with persistence wrapper bind-mounts).
An ingress NetworkPolicy (deny-all unless ingress_ports is set) and an egress policy (CIDR or FQDN).
An optional Secret from env_file.
With --docker, a second Longhorn PVC <sandbox>-docker-lh holding Docker’s overlay2 graph (default 20Gi, --docker-disk). It is snapshotted, cloned and re-attached alongside the root PVC on pause/fork/restore.

State written to /mnt/state (and to the rootfs via the bind-mount overlay) survives pod restarts because it lives on the PVC.

Pause

k7 pause demo                           # scale to 0, no snapshot
k7 pause demo --snapshot                # auto-named <sandbox>-paused-<unix-ts>
k7 pause demo --snapshot=demo-snap-1  # take a VolumeSnapshot, then scale to 0

--snapshot is the only pause flag besides -n. Bare --snapshot auto-names; --snapshot=NAME uses your name. There is no --pvc or --snapshot-class on the CLI (those were removed). Restore is k7 restore SNAPSHOT_NAME NEW_SANDBOX_NAME — there is no --latest. The snapshot is crash-consistent: it captures the disk state at the moment of the snapshot, not the in-memory process state. The sandbox process is then terminated as the Deployment scales to 0. On a 3-node HA cluster with longhorn_replicas=3, a live crash-consistent clone can fail fsck when the child PVC is mounted (UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY). k7 fork then never goes Ready. Prefer k7d warm fork when you need a reliable clone; for kql, pause the sandbox (quiesce) before snapshotting, or keep a named snapshot taken after a pause rather than forking a busy volume. The snapshot is stored as a VolumeSnapshot object in the same namespace and persists until you delete it (kubectl delete volumesnapshot <name>).

Resume

k7 resume demo

Scales the Deployment back to 1. The pod re-attaches to the same PVC; data on /mnt/state is exactly as it was at pause time. If the pause took a snapshot, the snapshot remains available — useful for rollback if the resumed workload corrupts something. Wait until Ready before k7 exec. On a 3-node HA cluster Longhorn attach can leave the pod Pending for tens of seconds after k7 resume returns. Poll k7 list until Ready is True (the same wait you need after Client.resume() / k7 restore). k7 snapshot list may show READY False for a few seconds after k7 snapshot create; wait for readyToUse: true before k7 restore.

Fork

k7 fork demo demo-clone
k7 fork demo demo-clone --snapshot custom-name

Fork takes a snapshot of the source sandbox’s PVC, clones it into a new PVC, and creates a new Deployment pointing at the cloned PVC. Implementation steps:

Create a VolumeSnapshot of <source>-root-lh (auto-named or --snapshot).
Wait for readyToUse: true on the snapshot.
Create a new PVC <new>-root-lh from the snapshot via dataSource.
Create a Deployment for the new sandbox using the cloned PVC.
Wait for the new pod to be Ready.
Delete the temporary snapshot (the cloned PVC is independent of it).

Scope: disk only. Fork does not clone:

Memory state (the new sandbox boots fresh)
CPU registers / process state
Network identity (the fork gets a new pod IP)

What a fork inherits on every backend: the source’s egress whitelist and its ingress_ports / ingress_from policy, so a child is never more reachable than its parent (an earlier release left forks fully ingress-closed). It does not inherit an expose_ports NodePort Service — expose the child explicitly if you need it. A --docker source forks with its graph disk (kql: cloned PVC; k7d / k7d-fc: reflinked scratch disk, dockerd keeps running with overlay2); kfd rejects fork. k7 0.4.0 prints a direct “kfd cannot fork” error. On 0.3.1 that rejection was a PVC 404 (Source root PVC <name>-root-lh not found; cannot fork storage) — kfd has no Longhorn PVC.

Observed performance

Single-node medians (Hetzner AX41, Longhorn replicas=1, 2026-08-10) — see PERFORMANCE.md:

Operation	kql	k7d
Cold create → pod ready	17.1 s	2.1 s
Named snapshot ready	6.5 s	n/a (k7d rejects `k7 snapshot`)
Pause	1.3 s (scale to 0)	0.2 s (VM frozen in place)
Resume → exec	4.1 s	0.3 s
Fork → usable	46.7 s (disk clone + cold boot)	2.4 s (warm memory CoW)

On a two-node r=2 cluster (2026-08-18 Show HN cut) kql fork is ~83 s; k7d fork stays ~4 s and is the only backend whose fork inherits live memory (/tmp markers, running processes). k7d-fc (k7d driving stock Firecracker) sits close to k7d: create → Ready 2.1 s and fork → exec ~2.45 s on the same node (2026-09-10). Forking a warm --docker engine takes ~6.8 s (k7d) / ~9.1 s (k7d-fc) to Ready with overlay2 still active in the child — see Docker in a sandbox.

Example: a parallel exploration with fork

A common AI workflow is to set up a heavy environment once, then explore many branches from it.

from k7_sdk import Client

k7 = Client(endpoint="https://node:31007", api_key="...", verify_ssl="./k7-ca.crt")

# Step 1: prepare a base sandbox with a big environment
base = k7.create({
    "name": "base",
    "image": "python:3.12-bookworm",
    "backend": "kata-qemu-longhorn",
    "root_disk_size": "20Gi",
    "before_script": (
        "apt-get update && "
        "apt-get install -y build-essential && "
        "pip install torch transformers datasets"
    ),
    "egress_whitelist": ["pypi.org", "files.pythonhosted.org", "huggingface.co"],
})

# (Wait until base is Ready; do whatever setup you want)

# Step 2: fork the base for each parallel experiment
forks = [base.fork(f"exp-{i}") for i in range(8)]
for f in forks:
    f.exec("python -c 'print(\"branch\", __import__(\"os\").getpid())'")

Or from the CLI on a node with API access:

for i in $(seq 0 7); do
  k7 fork base exp-$i &
done
wait

Each exp-N boots from a clone of base’s disk — the heavy pip install doesn’t run again.

Cleanup

k7 delete demo removes the Deployment, the PVCs (root and, with --docker, the graph disk), and any pause/fork snapshots associated with the sandbox. k7 snapshot gc reaps unreferenced snapshots and now also deletes orphan VolumeSnapshotContent objects and clears the PVC finalizer that used to leave Longhorn volumes stuck. To inspect what’s still around:

sudo k3s kubectl get pvc,volumesnapshot -A | grep demo
sudo k3s kubectl get volumes.longhorn.io,replicas.longhorn.io -n longhorn-system

Integration tests under tests/integration/ exercise pause/resume/fork end-to-end and detect orphaned Longhorn replicas or stuck PVCs.

Reference

Performance baseline: PERFORMANCE.md
Integration tests: tests/integration/

Getting started

Concepts

Using Katakate

API overview

API endpoints

Anatomy of a `kata-qemu-longhorn` sandbox

Pause

Resume

Fork

Observed performance

Example: a parallel exploration with fork

Cleanup

Reference

​Anatomy of a kata-qemu-longhorn sandbox

​Pause

​Resume

​Fork

​Observed performance

​Example: a parallel exploration with fork

​Cleanup

​Reference

Anatomy of a `kata-qemu-longhorn` sandbox

Pause

Resume

Fork

Observed performance

Example: a parallel exploration with fork

Cleanup

Reference