# Katakate Source: https://docs.katakate.org/ export const Tag = ({ children, bg = '#333', color = '#fff' }) => ( {children} ); # Projects Documentation for Katakate projects — K7 sandbox orchestration and the k7d VM runtime. stars Secure forkable VM sandboxed compute for AI agents and workloads.

Kubernetes Firecracker Kata QEMU K3s Longhorn Cilium Ansible

code execution sandboxing Docker in VM

Python

[GitHub →](https://github.com/Katakate/k7)
stars A purpose-built Rust VMM: fork a running Kubernetes cluster (~105 ms minimal guest, ~1.1 s Ubuntu k3s+Cilium+Tetragon) or a single VM in ~5 ms, and warp the guest clock so idle time costs nothing.

KVM virtio containerd K3s Aeneas

cluster fork snapshot tree RL environments

Rust

[GitHub →](https://github.com/katakate/k7d)
# k7 Source: https://docs.katakate.org/k7 > Secure sandboxed compute for AI agents and workloads k7 logo k7 is self-hosted infra for secure VM sandboxes — orchestrated with Kubernetes, driven by a CLI, REST API, and Python SDK. Pick a backend per sandbox: Firecracker via Kata (`kfd`), QEMU + Longhorn (`kql`), [k7d](/k7d/index) for warm CoW forks in milliseconds, or `k7d-fc` — the same k7d daemon driving jailed Firecracker. Run `docker build` inside any of them with `--docker`; sandboxes are locked away from the platform by default and reachable only through the ports you open. Get running in minutes: install, trust the API's CA, create your first sandbox. All `k7` commands with examples. kfd, kql, k7d, k7d-fc — pick per sandbox; what forks, what persists, what is jailed. `k7 create --docker`: pinned dockerd, overlay2 on a block disk, forkable on k7d. Sync and async clients with complete examples. Endpoint overview, request/response formats, and errors. Build a ReAct agent that executes inside a sandbox. # Quickstart Source: https://docs.katakate.org/k7/getting-started/installation > Install k7, prepare your node, start the API, and run your first sandbox k7 lets you run secure VM sandboxes on Kubernetes — Firecracker, QEMU + Longhorn, or [k7d](/k7d/index) (natively or under the Firecracker jailer). This Quickstart gets you from zero to a working sandbox via CLI and Python SDK. If you already installed k7 previously, consider running `make uninstall` before reinstalling to avoid stale cached files in a previous `.deb`. These pages track **k7 0.5.0** (tenants, roles and OIDC login, metering, the signed GHCR `k7-api` image, Helm chart, remote node install; Firecracker v1.16.2). The playbook still pins [k7d 0.7.0](https://github.com/Katakate/k7d/releases/tag/v0.7.0) by default. The Launchpad PPA and PyPI `k7-sdk` publish **0.5.0**. Read the [CHANGELOG](https://github.com/Katakate/k7/blob/main/CHANGELOG.md). ## Requirements - Ubuntu host with hardware virtualization (KVM) - Check: `ls /dev/kvm` should exist - **`kfd` and `kql`**: amd64 or arm64. **`k7d` and `k7d-fc` are amd64 / x86_64 only.** - Cloud guidance: AWS `.metal`, GCP (enable nested virtualization), Azure D/Ev series; typical VPS often lack KVM. Hetzner: Robot dedicated only (not Cloud VPS). - For the `kfd` (Firecracker) backend: one raw, unformatted disk for the LVM thin-pool. `kql`, `k7d`, and `k7d-fc` do not need a spare disk. - **No Docker and no source checkout.** 0.5.0 deploys the signed GHCR `k7-api` image by digest (`images.lock.yaml`, cosign-verified). Building `k7-api:local` from a checkout, which 0.4.0 did by default, is opt-in with `k7 install --api-image local`. - Ansible for the installer (Ubuntu), on the machine that runs `k7 install`: ```bash sudo add-apt-repository universe -y sudo apt update sudo apt install -y ansible ``` - Python 3.10+ on the client for the SDK Tested setup: Hetzner Robot dedicated, Ubuntu 24.04, with a **spare raw NVMe** for the `kfd` thin-pool. Dual-NVMe boxes (no third drive): put Ubuntu on **one** disk (`SWRAID 0`) and leave the other raw — see [Hetzner node setup](https://github.com/Katakate/k7/blob/main/tutorials/k7_hetzner_node_setup.md). Do **not** pin `--disk /dev/nvme1n1`: NVMe names swap across reboots; the playbook auto-detects the empty non-root disk. ## Install the CLI The Launchpad PPA publishes **0.5.0** (`k7 -V` after install). PyPI `k7-sdk` is the same version. ```bash sudo add-apt-repository ppa:katakate.org/k7 sudo apt update sudo apt install k7 k7 -V # 0.5.0 ``` The same package is on the [GitHub release](https://github.com/Katakate/k7/releases/tag/v0.5.0) if you prefer not to add the PPA: ```bash curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.5.0/k7_0.5.0_amd64.deb sudo apt install ./k7_0.5.0_amd64.deb ``` ## Install K7 on your node(s) `k7 install` runs from any directory — no clone, no Docker. It deploys the signed `ghcr.io/katakate/k7-api` image at the digest recorded in the release's `images.lock.yaml`, and verifies the cosign signature before use. ```bash k7 install --backend kfd,kql,k7d,k7d-fc --api-mode development ``` Two flags are **required** (no silent defaults): - `--backend`: `kfd` needs a spare raw disk; `kql` uses Longhorn on the OS disk; `k7d` is the warm-fork daemon; `k7d-fc` adds jailed Firecracker **v1.16.2**; `none` is control-plane only. The playbook pins [k7d 0.7.0](https://github.com/Katakate/k7d/releases/tag/v0.7.0). - `--api-mode`: `production` (every API key belongs to a [tenant](/k7/guides/tenants-and-roles), HTTPS only, metering mandatory) or `development` (unscoped keys keep working, remote install allowed). Documented in [Hardening](/k7/concepts/security-model#production-and-development-modes). That installs and wires up Kubernetes (K3s `v1.36.5+k3s1`, override with `--k3s-version`), Cilium, Kata, Firecracker + Jailer + thin-pool (`kfd`), QEMU + Longhorn (`kql`), and the k7d daemon + `RuntimeClass k7` / `k7-fc` (`k7d`, `k7d-fc`). It also deploys the `k7-api` (2 replicas) over HTTPS with a cluster CA, the non-privileged `k7-agent` DaemonSet, Prometheus metrics and the metering spool. Pass `-v` for verbose output. Two-node / HA inventories: [Multi-node clusters](/k7/guides/multi-node); install-time knobs (`--api-allow-cidr`, `--api-hostname`, `--oidc-*`, `--dns-upstream`, `--metering-*`, `--hubble`): [CLI reference](/k7/guides/cli#install). To install a k7d tarball you built yourself instead of GitHub `v0.7.0`, pass `--k7d-artifact /path/to/k7d-v0.9.0-x86_64-linux.tar.gz`. Already running Kubernetes (RKE2, kubeadm, EKS)? The same stack ships as a Helm chart (`deploy/helm/k7`) that never installs Kubernetes, the CNI or the CSI driver: see [Helm chart](/k7/guides/multi-node#helm-chart-for-existing-clusters). ![Example output: k7 install](/k7/images/ex-install.png) You should see "Installation completed successfully!" when done. Add `-v` for verbose output. ## The API and managing keys `k7 install` deploys the K7 API automatically as a `k7-api` Deployment in `kube-system`. K3s keeps it running and reschedules it on failure — there's no separate "start" step. If you want a CLI-only install with no API deployed, pass `--no-api` to `k7 install`. ### Check API status On a cluster node (needs `kubectl` / `k3s kubectl`): ```bash k7 api status ``` From a laptop after `k7 config set`, the same command probes `GET /health` on the configured URL instead of crashing. Replica counts still need kubectl on the node. ### Get the endpoint ```bash k7 api endpoint # https://:31007 ``` On a node this is `/etc/k7/api_endpoint`. From a laptop it prints the URL you configured. ### Trust the cluster CA from your laptop The default install serves the API over HTTPS with a certificate signed by a cluster CA the playbook minted (Let's Encrypt cannot issue for a bare IP). Copy the CA off the node once and point the CLI at it: ```bash scp root@:/etc/k7/tls/ca.crt ./k7-ca.crt k7 config set api.url https://:31007 k7 config set api.ca ./k7-ca.crt k7 config set api.key # from the next step ``` If you installed with `--api-hostname k7.example.com`, the certificate is publicly trusted and `api.ca` is unnecessary. On the node itself the CLI finds `/etc/k7/api_endpoint` and `/etc/k7/tls/ca.crt` automatically; you still have to `k7 config set api.key` with the token `generate-api-key` printed. ### Generate an API key Every key has a **role** (`--role` is required): `read-only` and `user` belong to a tenant, `operator` and `admin` are cluster-wide. In `--api-mode development` a `user` key without `--tenant` keeps the pre-0.5 "unscoped" behaviour; in `production` every `read-only`/`user` key needs a tenant. ```bash k7 generate-api-key mykey --role admin # cluster-wide, manages keys/tenants/nodes k7 tenant create acme -n acme-dev -n acme-prod # a tenant owns namespaces (and quotas, defaults) k7 generate-api-key acme-ci --role user --tenant acme # reaches every namespace of acme k7 generate-api-key acme-ro --role read-only --tenant acme -n acme-dev # narrowed further to one namespace ``` A tenant key cannot list or mutate another tenant's namespaces and cannot run all-namespaces operations. Details and quotas: [Tenants, roles and OIDC](/k7/guides/tenants-and-roles). **Save the printed key.** `k7 generate-api-key` stores only a hash in the `kube-system/k7-api-keys` Secret — the raw token is shown once and is **not** auto-loaded on the node. Key commands edit that Secret through kubectl, so run them on a server node (or add `--api` to go through k7-api with an `admin` credential). Point the CLI at it (laptop or node): ```bash k7 config set api.url https://:31007 # laptop; on the node this is auto-read from /etc/k7/api_endpoint k7 config set api.ca ./k7-ca.crt # laptop; on the node /etc/k7/tls/ca.crt is used automatically k7 config set api.key k7 list ``` Humans can skip keys altogether: with `k7 install --oidc-issuer … --oidc-client-id …`, `k7 login` signs in through the IdP (device flow by default, `--browser` for Authorization Code + PKCE) and stores a short-lived token; `k7 whoami` shows the resolved role. ![Example: k7 generate-api-key](/k7/images/ex-generate-api-key.png) ### Temporarily disable / re-enable the API ```bash k7 api disable # scale k7-api to 0 (temporary off; records the replica count) k7 api enable # scale back to the recorded count (default 2) ``` - API keys live in the `kube-system/k7-api-keys` Secret (hashes only); every API replica picks up a change within 5 seconds. Keys created before 0.5.0 in `/etc/k7/api_keys.json` were migrated by `k7 install` (`k7 api keys migrate`) and keep working until they expire. Authentication accepts the `X-API-Key` header or `Authorization: Bearer ` (API key or OIDC JWT). - The previous top-level commands `k7 start-api` / `k7 stop-api` / `k7 api-status` / `k7 get-api-endpoint` are hidden deprecated aliases that forward to `k7 api …` with a warning. ## Create your first sandbox via CLI Example `k7.yaml`: ```yaml name: demo image: alpine:3.20 namespace: default backend: k7d # env_file: /root/secrets.env # optional; omit unless that file exists on the node limits: cpu: "1" memory: "1Gi" before_script: | # Installing curl. Egress is open during before_script, then restricted (empty whitelist) afterwards. apk add curl egress_whitelist: [] ``` Do not use `cpu: 100m` / `memory: 128Mi`. Kata Firecracker refuses hypervisor memory below **256Mi** (`io.katacontainers.config.hypervisor.default_memory`) and the pod stays `ContainerCreating` until create times out. Use at least `1Gi` (or omit `limits`). ### Create a sandbox ```bash # Uses k7.yaml in the current directory by default k7 create # Or pick a backend (kfd | kql | k7d | k7d-fc) k7 create -f k7.yaml --backend k7d # Need docker build / docker run inside the sandbox? (see the Docker guide) k7 create --docker --backend k7d --egress-open builder ubuntu:24.04 ``` CLI `k7 create` with no `--egress` / `--egress-open` **blocks all egress**. An API/YAML request that omits `egress_whitelist` leaves egress **open**. Use `--egress-open` or omit the field in YAML when you want the internet. ![Example: k7 create](/k7/images/ex-create.png) ### Shell into your sandbox ```bash k7 exec demo -- echo hello # works from a laptop via the API k7 shell demo # interactive; node-local (needs kubectl) ``` ![Example: k7 shell](/k7/images/ex-shell.png) ### List sandboxes ```bash k7 list ``` ![Example: k7 list](/k7/images/ex-list.png) ### Delete a sandbox ```bash k7 delete my-sandbox-123 ``` ### Delete all sandboxes ```bash k7 delete-all ``` ### Prerequisites for the SDK ```bash # `k7 install` already deployed the API; just grab the endpoint and a key k7 api endpoint k7 generate-api-key my-key --role user # add --tenant in production mode ``` Always pass the cluster CA (`./k7-ca.crt` from your laptop, or `/etc/k7/tls/ca.crt` on the node). `k7 api status` prints an SDK snippet that already includes `verify_ssl`. ## Create your first sandbox via Python SDK Install the SDK on your **client** (laptop), not necessarily on the cluster node: ```bash pip install k7-sdk==0.5.0 ``` A freshly installed Ubuntu node does not include `python3-venv`. If you run the SDK on the node: `sudo apt install -y python3-venv python3-pip`, then `python3 -m venv ~/k7sdk && ~/k7sdk/bin/pip install k7-sdk==0.5.0`. `pip install --break-system-packages k7-sdk==0.5.0` also works on Ubuntu 24.04. Use the synchronous client: ```python from k7_sdk import Client import time # verify_ssl takes the CA bundle path on the default (cluster-CA) install; # leave it at True for a Let's Encrypt (--api-hostname) cluster. k7 = Client(endpoint="https://:31007", api_key="", verify_ssl="./k7-ca.crt") sb = k7.create({ "name": "my-sandbox", "image": "alpine:latest" }) # POST returns once objects exist; poll until Ready before exec. deadline = time.time() + 120 while time.time() < deadline: for s in k7.list(): if s["name"] == "my-sandbox" and s.get("ready") in (True, "True"): break else: time.sleep(2) continue break result = sb.exec('echo "Hello World"') print(result["stdout"]) print(k7.list()) sb.delete() ``` Async variant: ```python import asyncio from k7_sdk import AsyncClient async def main(): k7 = AsyncClient(endpoint="https://:31007", api_key="", verify_ssl="./k7-ca.crt") print(await k7.list()) await k7.aclose() asyncio.run(main()) ``` ## Next steps - Explore the [CLI guide](/k7/guides/cli) - Explore the [Python SDK guide](/k7/guides/python-sdk) - Run [Docker inside a sandbox](/k7/guides/docker) - Integrate with the [REST API](/k7/api/introduction) # Architecture Source: https://docs.katakate.org/k7/concepts/architecture > How the CLI, API, K3s, Kata, k7d, and the four VM backends fit together Katakate is a thin orchestration layer over a curated stack of battle-tested infrastructure. This page explains what the moving parts are and how they talk to each other. ## High-level diagram ```mermaid flowchart TB subgraph Client["Client (your machine)"] direction LR CLI["k7 CLI"] SDK["Python SDK
k7-sdk"] Curl["curl / your
HTTP client"] end subgraph Node["K7 node — Ubuntu + K3s"] direction TB subgraph ControlPlane["K3s control plane"] API["k7-api ×2
FastAPI + Caddy TLS sidecar
kube-system · NodePort 31007 (HTTPS)
roles · tenants · OIDC · metering leader"] K3sAPI["K3s API server"] Sched["scheduler
(nodeSelector by backend)"] API -- "kubernetes_asyncio" --> K3sAPI K3sAPI --- Sched end subgraph DataPlane["Data plane
per node"] direction LR Cilium["Cilium CNI
NetworkPolicy +
CiliumNetworkPolicy +
cluster-wide platform deny"] Containerd["K3s containerd
+ runtime shims
(kata / kata-qemu / k7 / k7-fc)"] Agent["k7-agent
DaemonSet pod, non-privileged
(k7d VM ops · storage stats · usage sampler)"] K7dD["k7d daemon
/run/k7d/k7d.sock"] end subgraph Sandbox["Sandbox VM"] VMM["Firecracker (+ jailer),
QEMU, k7d native VMM,
or k7d-driven Firecracker"] UserCnt["user container"] VMM --- UserCnt end Storage[("Storage backends
LVM thin-pool · Longhorn PVC · k7d disk pool
+ optional Docker graph disk")] K3sAPI -- "schedules pod" --> Containerd Cilium -. "wires netns" .- Containerd Containerd -- "launches" --> VMM API -. "k7d pause/resume/fork
on remote nodes" .-> Agent Agent -. "proxies to" .- K7dD K7dD -- "owns k7d VMs" --> VMM VMM -. "rootfs" .- Storage end CLI -- "local: K7Core direct
remote: HTTPS + API key or OIDC JWT" --> API SDK -- "HTTPS · API key / token_provider" --> API Curl -- "HTTPS · X-API-Key / Bearer" --> API classDef client fill:#0f1a2a,stroke:#3b82f6,color:#fff; classDef ctrl fill:#1a1a1a,stroke:#888,color:#fff; classDef data fill:#0b0b0b,stroke:#ef672b,color:#fff; classDef vm fill:#2a1a05,stroke:#ef672b,color:#fff; classDef store fill:#0b1a0b,stroke:#22c55e,color:#fff; class Client,CLI,SDK,Curl client; class ControlPlane,API,K3sAPI,Sched ctrl; class DataPlane,Cilium,Containerd,Agent,K7dD data; class Sandbox,VMM,UserCnt vm; class Storage store; ``` ## Components ### CLI — `k7` A Typer-based CLI distributed as a Debian package (`apt install k7`). Sandbox-management commands (`create`, `list`, `fork`, …) talk to the **K7 API** by default, with an API key or an OIDC login (`k7 login`). Node-local commands (`install`, `shell`, `top`, `tenant`, `nodes`, and the key commands unless `--api` is passed) still run on the cluster with kubectl. Hidden `--core` bypasses the API and calls `K7Core` in-process. Interactive `k7 shell` shells out to `kubectl exec`. ### API — `k7-api` A FastAPI service that exposes a REST surface for sandbox lifecycle, exec, snapshots, identity (keys, role bindings), tenants and usage. The API runs as a **K3s `Deployment`** in `kube-system` — **2 replicas** spread across nodes with a PodDisruptionBudget, uid 1000, all capabilities dropped — behind a `NodePort` Service (31007), with an in-cluster `ServiceAccount` and a scoped `ClusterRole`. It is stateless: API keys live in the `kube-system/k7-api-keys` Secret (hashes only, snapshot refreshed every 5 s), role bindings in ConfigMap `k7-role-bindings`, tenants as `Tenant` custom resources, and OIDC tokens are validated against the issuer's JWKS with no sessions. The replica holding the `k7-tenant-status` / `k7-metering-leader` Leases reconciles `Tenant.status.usage` and samples storage usage. The image is `ghcr.io/katakate/k7-api` deployed **by digest** from the release's `images.lock.yaml` (cosign-signed); `--api-image local` builds it from a checkout instead. The pod carries a **Caddy TLS sidecar**: FastAPI listens on `:8000` inside the pod, Caddy terminates HTTPS on the NodePort with a cluster CA the installer writes to `/etc/k7/tls/ca.crt` (or a Let's Encrypt / operator-supplied certificate; the Let's Encrypt overlay pins back to one replica on the first master). On Cilium clusters `k7 install --api-allow-cidr` adds a `CiliumNetworkPolicy` (`k7-api-ingress`) restricting who can reach the port. See [API transport](/k7/concepts/security-model#api-transport-tls). Temporarily stop it with `k7 api disable` (scale to 0) and bring it back with `k7 api enable`. The entire `K7Core` is async (`kubernetes_asyncio` + `httpx`), so the API can serve concurrent requests without thread-pool workarounds. Both k7-api and k7-agent serve Prometheus metrics on `:9464` and log JSON lines with a request id that follows each request from the API into the agent ([Observability](/k7/guides/observability-metering)). ### Per-node agent — `k7-agent` A **DaemonSet** in `kube-system` (same image as `k7-api`, running `k7.api.agent:app`) that exposes node-local operations the central API can't do remotely: k7d VM ops (`pause` / `resume` / `fork` / `lookup`, which need the node's `/run/k7d/k7d.sock` and containerd socket), storage-pool utilization (`GET /agent/v1/storage`), and the **usage sampler** that meters every sandbox pod on the node into a local spool. The API always goes through the agent for k7d ops (`K7_K7D_REMOTE_ONLY=1`), authenticated with a shared token provisioned by the installer (`/etc/k7/agent_token`) and locked down by a CiliumNetworkPolicy so only the k7-api pod can reach the agents. If no agent is Ready on the target node, the operation fails loudly — never a silent no-op. The agent is **not privileged**: uid 0 plus the `k7d` group (gid 770 on every node, so the DaemonSet carries one static gid), all capabilities dropped, read-only root filesystem, `RuntimeDefault` seccomp, no ServiceAccount token. Its only host mounts are the containerd socket directory, `/run/k7d`, `/var/lib/k7d` (ro), `/run/k7` (ro) and the usage spool. ### Python SDK — `k7-sdk` A small `requests`/`httpx` wrapper published to PyPI as **`k7-sdk`** (`from k7_sdk import Client`). Both `Client` (sync) and `AsyncClient` (async, `pip install "k7-sdk[async]"`) speak the same JSON envelope as the API, authenticate with an `api_key` or a `token_provider` callable (OIDC), and cover keys, tenants, role bindings, node dedication, usage and remote install as well as the sandbox lifecycle. ### K3s The CNCF-certified Kubernetes distribution we install on every node (pinned `v1.36.5+k3s1`; `--k3s-version` overrides before the first install). K3s is single-binary, supports embedded etcd for HA, and ships its own containerd. Katakate **does not touch your existing Docker or containerd** — K3s installs its own runtime so installation is non-disruptive, and since 0.5.0 the installer needs no Docker at all. Already running Kubernetes? The **Helm chart** (`deploy/helm/k7`) installs k7-api, k7-agent, the k7-node-installer, RuntimeClasses, snapshot GC and the Cilium policies onto RKE2, kubeadm or EKS, and never installs Kubernetes, the CNI or the CSI driver. Node preparation on those clusters is done by the `k7-node-installer` image (the playbook's node-prep role, `k7 install --node-prep-only`). See [Multi-node clusters](/k7/guides/multi-node#helm-chart-for-existing-clusters). ### Tenants A `Tenant` custom resource (`tenants.k7.katakate.org/v1alpha1`, cluster-scoped) groups namespaces and carries dedicated nodes, a default sandbox size, quotas and per-VM limits. k7-api resolves the namespace's tenant on every request, enforces quotas under a per-tenant Lease, and stamps every object it creates with the tenant and the creating key id. The ValidatingAdmissionPolicy `k7-runtime-annotations` makes sure only k7-api can set the `k7d.katakate.org/*` and `io.katacontainers.*` pod annotations the runtimes act on. See [Tenants, roles and OIDC](/k7/guides/tenants-and-roles). ### Cilium Cilium is the default CNI plugin. It enables **FQDN-based egress** (e.g. allow `api.openai.com` instead of a CIDR) via `CiliumNetworkPolicy`. Cilium runs with `kubeProxyReplacement=true` for a leaner stack. Pass `--cni flannel` at install time to fall back to K3s' default CNI (CIDR-only egress). ### Kata Containers Kata is the OCI runtime that turns each pod into a lightweight VM. We install both runtime classes: - `kata` — Firecracker microVMs (used by `kata-firecracker-devmapper`) - `kata-qemu` — QEMU-based VMs (used by `kata-qemu-longhorn`) Kata guest seccomp is enforced (`disable_guest_seccomp = false`) on both classes. The other two runtime classes are not Kata. `k7` and `k7-fc` are both served by the [k7d](/k7d/index) containerd shim (`containerd-shim-k7-v1`) talking to the node-local `k7d` daemon; `k7-fc` differs only by a per-runtime `ConfigPath` (`/etc/k7d/shim-k7-fc.toml`, `backend = "firecracker"`) that tells k7d to drive stock Firecracker under the stock jailer instead of its own VMM. See [k7d backends](/k7d/concepts/backends). ### Backends A **backend** is a tuple of (runtime class, storage driver). There are four: | Backend | Alias | Runtime | Storage | Best for | |---------|-------|---------|---------|----------| | `kata-firecracker-devmapper` | `kfd` | Kata + Firecracker + jailer (`kata`) | LVM thin-pool snapshotter | Lightest footprint, fastest cold boot, ephemeral workloads | | `kata-qemu-longhorn` | `kql` | Kata + QEMU | Longhorn CSI (replicated PVCs) | Persistent volumes, disk-level snapshot/pause/resume/fork | | `k7d` | — | [k7d Rust VMM](/k7d/index) (`runtimeClassName: k7`) | erofs on virtio-blk + host-local state | ~5 ms warm fork, snapshot trees, RL / agent tree search | | `k7d-fc` | `k7-fc` | k7d daemon driving stock Firecracker + jailer (`runtimeClassName: k7-fc`) | erofs on virtio-blk + host-local state | Firecracker isolation (jailer, seccomp) with k7d fork/tree/pool semantics; multi-tenant `--docker` | A node can support **one or more** backends; on a multi-node cluster, different nodes can specialize. See [Backends](/k7/concepts/backends) for a deeper comparison. ### Storage — Longhorn (kata-qemu-longhorn only) [Longhorn](https://longhorn.io/) provides cluster-wide replicated block storage via a Kubernetes CSI driver. Each `kata-qemu-longhorn` sandbox gets its own root PVC (`-root-lh`), backed by `numberOfReplicas` replicas spread across nodes. The default StorageClass uses `WaitForFirstConsumer` binding and `dataLocality: best-effort` so the volume is created on (and one replica stays on) the node where the pod is scheduled. VolumeSnapshots power `pause`/`resume`/`fork`. ### Storage — devmapper thin-pool (kata-firecracker-devmapper only) A raw disk is provisioned as an LVM thin-pool. Containerd's devmapper snapshotter creates per-sandbox logical volumes from this pool, giving each microVM its own block device while sharing the underlying physical storage efficiently. `--docker` sandboxes on this backend get an ephemeral graph LV through the `k7-docker-lvm` StorageClass (OpenEBS LVM LocalPV). ### Storage — Docker graph disks (`--docker`) A sandbox created with `--docker` gets a dedicated block device for Docker's `overlay2` graph: a Longhorn PVC `-docker-lh` on `kql`, an LVM LocalPV on `kfd`, or a per-VM scratch disk on `k7d` / `k7d-fc` that is reflinked on fork. See [Docker in a sandbox](/k7/guides/docker). ## Request lifecycle: `k7 create` 1. **Client → API**: `POST /api/v1/sandboxes` with a `SandboxConfig` JSON body. The API authenticates the API key (SHA-256 hash lookup in the Secret snapshot) or OIDC JWT, resolves the role, checks that the route allows it and that the namespace belongs to the caller's tenant, applies the key's node scope, checks the image registry host against the public allowlist (SSRF guard), fills `limits` from the tenant's `defaultSize`, refuses limits the backend cannot enforce, takes the tenant's quota Lease and checks the quota, then forwards to `K7Core.create_sandbox`. One audit line is written whatever the outcome. 2. **K7Core**: builds a Kubernetes `Deployment` (1 replica, labelled with the tenant, key id and a fresh `sandbox-uid`), an optional `Secret` from `env_file`, an optional `PVC` (`-root-lh`) on `kata-qemu-longhorn`, an optional Docker graph disk, an ingress `NetworkPolicy` (deny-all, or opened for `ingress_ports` / `ingress_from`), an optional NodePort Service for `expose_ports`, and either a CIDR `NetworkPolicy` or a `CiliumNetworkPolicy` for egress (whose DNS rule carries the allowlist's own matchers). The always-on `CiliumClusterwideNetworkPolicy` `k7-sandbox-platform-deny` (installed once per cluster) keeps every sandbox off nodes, the apiserver and platform namespaces regardless of these per-sandbox objects. 3. **K3s scheduler** picks a node based on `nodeSelector` (`k7.katakate.org/backend-=true`, set by the installer per host), the tenant's dedicated nodes (selector + toleration for the `k7.katakate.org/tenant` taint), and an explicit `node_name` pin if any. 4. **Kata or k7d** (via the appropriate containerd shim) launches a VM, mounts the rootfs (devmapper or Longhorn PVC), and starts the user container inside it. 5. **`before_script`** runs via `kubectl exec` while egress is still open. Once it completes, the egress NetworkPolicy is applied and the pod is marked Ready. 6. The API returns `201 Created` with a `Location` header. ## Source code map ``` src/ ├── k7/ │ ├── api/ FastAPI service (Dockerfile + manifests deploy as a K3s Deployment) │ │ ├── main.py routes │ │ ├── authz.py roles, PERMISSIONS table, Principal │ │ ├── oidc.py JWT validation, JWKS cache, role bindings │ │ ├── keystore.py Secret-backed API key store (5 s snapshot) │ │ ├── tenants.py tenant admission on requests │ │ ├── remote_install.py POST /api/v1/install → k7-node-installer pool │ │ ├── agent.py k7-agent app (VM ops, storage, usage sampler) │ │ └── snapshot_gc.py the k7-snapshot-gc CronJob │ ├── cli/ Typer CLI (k7.py, _oidc.py for login), packaged as the k7 .deb │ ├── core/ K7Core: async sandbox lifecycle (Kubernetes + Longhorn + Cilium) │ │ ├── core.py orchestration │ │ ├── models.py SandboxConfig / SandboxInfo / ExecResult dataclasses │ │ ├── tenancy.py Tenant CR, quotas under a Lease, node dedication │ │ ├── docker.py --docker: graph disks, docker-vehicle (Kata) / k7d annotations │ │ └── sidecar.py legacy sidecar registry (kept for the deprecated --sidecar alias) │ ├── metering/ usage sampler, spool, sinks (Prometheus / http / stdout), showback │ └── deploy/ Ansible playbook, roles, manifests, Helm chart │ ├── k7-install-node.yaml the playbook │ ├── roles/ node prep shared with the k7-node-installer │ ├── manifests/ k7-api (Deployment, PDB, agent DaemonSet, Tenant CRD), k7-runtime-annotations admission policy │ ├── helm/k7/ chart for existing clusters (RKE2, kubeadm, EKS) │ ├── observability/ Grafana dashboards, alert rules, runbooks │ ├── k7_api_tls.py Caddy TLS sidecar rendering for k7-api │ ├── k7_api_keys.py key minting, roles, migration │ └── k7d-fc/ install-firecracker.sh (pinned Firecracker + jailer for k7d-fc) └── k7_sdk/ Python SDK (PyPI package k7-sdk) ``` # Backends Source: https://docs.katakate.org/k7/concepts/backends > kfd vs kql vs k7d vs k7d-fc — how to choose A **backend** in Katakate combines a VM runtime with a storage driver. There are four: `kata-firecracker-devmapper` (`kfd`), `kata-qemu-longhorn` (`kql`), `k7d`, and `k7d-fc`. A node can install any combination (`k7 install --backend kfd,kql,k7d,k7d-fc`) and each sandbox picks one (`k7 create --backend …`). **Deprecated aliases:** `fd` and `ql` (and the long names without the `kata-` prefix) are older names for `kfd` / `kata-firecracker-devmapper` and `kql` / `kata-qemu-longhorn`. They still resolve for backwards compatibility, but new configs should use the current names. `k7-fc` is accepted as an alias of `k7d-fc`. ## At a glance | | `kfd` — kata-firecracker-devmapper | `kql` — kata-qemu-longhorn | `k7d` | `k7d-fc` | |--|---|---|---|---| | Runtime class | `kata` | `kata-qemu` | `k7` | `k7-fc` | | Hypervisor | Firecracker (microVM) via Kata | QEMU via Kata | [k7d](/k7d/index) — purpose-built in-process Rust VMM | The same k7d daemon driving **stock Firecracker under the stock jailer** | | Process isolation | **Jailer** (chroot, dropped caps, seccomp) | Kata's built-in sandboxing | KVM hardware isolation; one daemon hosts every VM (see [k7d security model](/k7d/concepts/security)) | KVM **plus** a per-VM jailer (own uid, chroot, pid namespace, cgroup, Firecracker seccomp) | | Storage driver | LVM thin-pool + containerd devmapper snapshotter | Longhorn CSI (replicated PVCs) | erofs images on virtio-blk + reflink-XFS volume images | Same as k7d (erofs / scratch on virtio-blk); **no virtiofs / hostPath** | | Persistent disk per sandbox | Ephemeral (devmapper LV, lost on delete) | Yes — `-root-lh` PVC, sized via `--root-disk-size` | Host-local VM state | Host-local VM state | | Named snapshots (`k7 snapshot`) / `k7 restore` | ❌ Rejected up front (node-local devmapper root) | ✅ Disk-level VolumeSnapshots | ❌ Not yet exposed by K7 (fork the live sandbox instead) | ❌ (same as k7d) | | `k7 fork` | ❌ Rejected | ✅ Disk clone + cold boot (~45 s) | ✅ Warm CoW fork of memory **and** disk (~5 ms at the VMM, ~2.4 s end-to-end) | ✅ Same daemon path (Firecracker Diff snapshot → reflinked base → `LoadSnapshot`), ~2.5 s end-to-end | | `k7 pause` / `resume` | Scale to 0 / 1 (state lost; `--snapshot` rejected up front) | Scale to 0 / 1, disk persists | VM frozen in place: vCPUs stop, memory stays (0.2 s / 0.3 s) | Same daemon path (0.2 s pause; see note below) | | Docker in the VM (`--docker`) | ✅ Privileged `docker-vehicle`, overlay2 on an ephemeral LVM block LV | ✅ Privileged `docker-vehicle`, overlay2 on a Longhorn Block PVC (survives pause / fork / restore) | ✅ Agent-supervised in-guest `dockerd`, overlay2 on virtio-blk, **forkable** | ✅ Same guest `dockerd`, **forkable** | | Multi-node replication | ❌ Node-local thin pool | ✅ Longhorn replicas across nodes | ❌ Host-local trees today | ❌ | | Tenant `limits.network` (`--egress-bps` / `--ingress-bps`) | ✅ Firecracker token buckets | ✅ `tc` on the tap | ❌ `400 LimitNotEnforceable` (also for `disk` quotas; `diskIO` is unenforceable everywhere) | ❌ (same as k7d) | | Cold-create latency | Faster (microVM boot) | Slower (full VM boot + PVC bind), ~17 s | ~2.1 s to a Ready pod | ~2.1 s to a Ready pod | | Privileged container required | No (unless `--docker`) | **Yes** (for the bind-mount persistence wrapper) | No | No | | Architecture | amd64, arm64 | amd64, arm64 | **amd64 / x86_64 only** | **amd64 / x86_64 only** | | Best for | Many small ephemeral workloads, AI agents, CI runners | Long-lived workloads, replicated state | Fork-heavy workflows: RL rollouts, agent tree search, snapshot trees | k7d's fork/pause lifecycle **and** a per-VM jailer — multi-tenant `--docker` | Numbers are medians of 3 on one Hetzner AX41 node; ranges and methodology are in [PERFORMANCE.md](https://github.com/Katakate/k7/blob/main/PERFORMANCE.md). **Privilege separation** is the k7d daemon, not a fifth backend: a root node broker (no listening socket) plus a uid-`k7d` VM broker for the whole node. Both `k7d` and `k7d-fc` sandboxes go through that pair. Tables: [k7d vs k7d-fc](/k7d/concepts/backends#k7d-vs-k7d-fc) and [privilege separation](/k7d/concepts/security#privilege-separation). ## Choosing a backend Pick **`kata-firecracker-devmapper`** when: - You run many short-lived sandboxes per node and want the smallest possible footprint. - You don't need to pause, resume, fork, or persist state between restarts. - You're on a single node and don't need replication. - Security is paramount: Firecracker has the smallest attack surface, and the **jailer** is active (verified by `tests/integration/test_firecracker.py::test_jailer_active`). Pick **`kata-qemu-longhorn`** when: - You need a real persistent disk that survives pod restarts (`/mnt/state` mount inside the sandbox). - You want to `pause` / `resume` / `fork` sandboxes at the disk level, or `k7 snapshot` / `k7 restore` named snapshots. - You run multi-node and want data to survive node failure (Longhorn replicates). - You need bigger root disks (sized with `--root-disk-size`), or a Docker graph that survives pause / fork / restore. Pick **`k7d`** when: - Fork latency matters: k7d warm-forks a running VM in ~5 ms with faithful memory, disk, process, and network state — versus ~45 s for a disk-only Longhorn fork. - You want snapshot trees: fork many branches from a checkpoint, protect winners, prune losers, under RAM/disk budgets. - Your workload is RL training or agent tree search over many resettable environments. - You want a forkable Docker engine: `k7 create --docker --backend k7d` runs `dockerd` inside the guest and `k7 fork` clones it, running inner containers included. Pick **`k7d-fc`** when: - You want k7d's fork / pause lifecycle **and** Firecracker's jailer around every VM — a separate VMM process per sandbox with its own uid, chroot, pid namespace, cgroup, and Firecracker's default seccomp. A guest-to-VMM escape lands in that jail instead of the k7d daemon. - You run `--docker` for mutually distrusting tenants. The guest side is identical to `k7d`, so this is a RuntimeClass switch, not a rewrite. - You can live without virtiofs / hostPath volumes (refused up front on `k7d-fc`) and without k7d's [time warp](/k7d/guides/time-warp). See the [K7d documentation](/k7d/index) for the runtime's own concepts, API, and benchmarks, and [Backends: native vs Firecracker](/k7d/concepts/backends) for how the k7d daemon selects the engine. **Cluster fork is a k7d feature, not a k7 feature.** Forking an entire live Kubernetes cluster (~100 ms for a 3-node cluster) is done by the k7d runtime directly, through its own daemon API — see [Cluster mode](/k7d/guides/cluster-mode). k7's `fork` command clones a single sandbox. ## Specifying the backend ### At install time ```bash k7 install --backend kfd --api-mode development # --backend and --api-mode are both required; kfd auto-detects a raw disk k7 install --backend kfd --disk /dev/nvme2n1 # pin the thin-pool disk explicitly k7 install --backend kql # kata-qemu-longhorn only, no raw disk needed k7 install --backend kql --longhorn-extra-disk /mnt/longhorn-extra # optional extra storage k7 install --backend k7d # k7d runtime (daemon + shim + RuntimeClass k7) k7 install --backend k7d,k7d-fc # add pinned Firecracker + jailer and RuntimeClass k7-fc k7 install --backend k7d --k7d-version 0.7.0 # pin the k7d release (playbook default: 0.7.0; other versions need k7d_sha256=) k7 install --backend k7d --k7d-artifact /root/k7d-v0.9.0-x86_64-linux.tar.gz # install a newer k7d from a local tarball k7 install --backend kfd,kql --longhorn-extra-disk /mnt/longhorn-extra # omit --disk on dual-NVMe (auto-detect) k7 install -i inventory.ini # multi-node, per-host backends k7 install -i inventory.ini --ha # multi-master HA (3+ servers, embedded etcd) ``` ### At sandbox creation time ```bash # Auto-detect (uses /etc/k7/backend on the API node, else kfd) k7 create demo alpine:3.20 --cpu 1 --memory 1Gi # Explicit k7 create demo alpine:3.20 --backend kql --cpu 1 --memory 1Gi k7 create demo alpine:3.20 --backend k7d --cpu 1 --memory 1Gi k7 create builder ubuntu:24.04 --backend k7d-fc --docker --egress-open --cpu 2 --memory 3Gi ``` In YAML: ```yaml backend: kata-qemu-longhorn # or kata-firecracker-devmapper, k7d, k7d-fc — or alias kql / kfd / k7-fc ``` If the requested backend isn't available on any node, you'll see a clear `FailedScheduling` message listing which backends are. ## Backend-specific behavior ### kata-qemu-longhorn - Each sandbox gets a Longhorn PVC named `-root-lh`, defaulting to **10Gi** (override with `--root-disk-size 20Gi` or in YAML). - The container runs **privileged** to allow the persistence wrapper to bind-mount `/mnt/state` from the PVC. - Image base must include `/bin/sh`, `tar`, and `mount` (Debian/Ubuntu/Alpine all work). - Image `ENTRYPOINT`+`CMD` are preserved by default — override with `--entrypoint` / `--cmd`. - `pause` takes a crash-consistent VolumeSnapshot before scaling to 0 (optional; pass `--snapshot `). - `fork` snapshots the source PVC, clones it via Longhorn's CSI VolumeSnapshot+Restore, then creates a new Deployment pointing at the cloned PVC. - `--docker` adds a second Longhorn **Block** PVC (`-docker-lh`, default 20Gi via `--docker-disk`) for the overlay2 graph; it is included in pause / fork / restore. The two VolumeSnapshots are crash-consistent per volume, not cross-volume atomic. ### kata-firecracker-devmapper - Sandbox storage is a per-pod logical volume on the LVM thin-pool — fast to create, lost on delete. - The Firecracker process runs **inside the jailer**: a chroot, with dropped capabilities and a restrictive seccomp filter, as an unprivileged UID. - Kata's guest seccomp is enforced (`disable_guest_seccomp = false`): the container's `RuntimeDefault` seccomp profile is applied by the in-guest runtime, not silently dropped. - `pause` / `resume` only scale the Deployment; `fork`, `k7 snapshot create` and `k7 pause --snapshot` are rejected up front — there's no persistent disk to snapshot. - `--root-disk-size` has no effect (warned by the CLI). - `--docker` puts the overlay2 graph on a generic ephemeral volume from StorageClass `k7-docker-lvm` (OpenEBS LVM LocalPV over `kata-vg/thin-pool`); it is deleted with the pod and `k7 fork` of a kfd `--docker` sandbox is rejected. Docker-heavy kfd nodes share the thin-pool with these LVs — size `kata_thinpool_pv_size` accordingly. ### k7d - Pods run as k7d microVMs via `runtimeClassName: k7` — a small containerd shim talks to the `k7d` daemon over `/run/k7d/k7d.sock`. - Fork copies the **whole VM**: memory, disk, processes, and network identity — not just the disk. Warm fork is ~5 ms. With k7d **0.9.0+** the fork keeps a working NIC (own bridge, NAT), so forked sandboxes have egress like their source; older k7d forks came up without network. - Multi-vCPU guests are supported: the vCPU count is derived from the pod's CPU limit (`cpu: "2"` boots a 2-vCPU guest), with fork/snapshot parity. - **VM ops work for sandboxes on any node.** The k7d daemon and its socket are node-local, but `k7 pause` / `resume` / `fork` through the API are forwarded to the `k7-agent` DaemonSet pod on the sandbox's node when needed. A fork still lands on the **source's node** (cross-node fork is on the k7d roadmap). - `--docker` does **not** add a second container: the pod is stamped `k7d.katakate.org/docker=true` and the guest agent (PID 1) supervises a pinned `dockerd` whose graph lives on a per-sandbox virtio-blk scratch disk (overlay2, default 20Gi). Because it is not a CRI sidecar, `k7 fork` of a `--docker` sandbox works — the child keeps `dockerd` and its running inner containers. Requires k7d **0.6.0+** (payload at `/usr/local/share/k7d/docker`); an older k7d fails loudly with `this k7d has no docker service; upgrade`. - Warm fork works for single-workload sandboxes (including `--docker`); forking a sandbox with a **real CRI sidecar** is rejected (use `kql` for that). - Per-node k7d disk-pool utilization is visible via `k7 nodes storage` (see [CLI reference](/k7/guides/cli)). - See [RuntimeClass k7](/k7d/guides/runtime-class) for how the runtime is registered, and the [feature matrix](/k7d/guides/feature-matrix) for current Kubernetes feature support. ### k7d-fc - Same daemon, shim, guest agent, and `--docker` service as `k7d`; pods use `runtimeClassName: k7-fc`. The daemon boots the guest on **upstream Firecracker under the upstream jailer** (sha-pinned binaries installed to `/usr/local/bin` by the playbook — a different binary from Kata's `/opt/kata/bin` Firecracker used by `kfd`). k7 **0.4.0+** (including 0.5.0) and k7d both pin **v1.16.2**; k7 0.3.1 still vendored v1.16.1. - `k7 install --backend k7d-fc` writes `/etc/k7d/shim-k7-fc.toml` (`backend = "firecracker"`), registers containerd `runtimes.k7-fc` with a `ConfigPath` option (and no `BinaryName`), creates RuntimeClass `k7-fc` with a PodOverhead of `32Mi` / `50m`, and labels the node `k7.katakate.org/backend-k7d-fc=true` (plus `backend-k7d`, so native `k7d` pods still schedule there). - Fork and pause go through the same `/run/k7d/k7d.sock` verbs and `fork-source-*` annotations as `k7d`. Under the hood a fork is a Firecracker Diff snapshot materialised onto a reflinked base image and `LoadSnapshot`ed into a fresh jail (~2.5 s end-to-end via k7; ~2.45 s fork→exec measured). - **hostPath / PVC `volumes` are refused at create time** — Firecracker has no virtiofs. Time warp / dilation is unavailable on this backend. - Pause / resume → exec needs Firecracker **v1.16.2+**, which k7 0.4.0+ installs. k7 0.3.1 still copied v1.16.1, which gated vsock RX after a bare resume (upstream #6100 / k7d CHALLENGES); current k7d then **fails `k7 resume` loudly** (`Firecracker < v1.16.2`). Native `k7d` is unaffected. - Requires k7d **0.6.0+** (Firecracker engine + `ConfigPath`). The stock 0.5.0 tarball does not know `ConfigPath` and the RuntimeClass will not come up. # Networking Source: https://docs.katakate.org/k7/concepts/networking > Cilium, platform isolation, opt-in ingress and NodePort exposure, FQDN-aware egress whitelists without DNS exfiltration, CoreDNS upstreams, bandwidth caps, Hubble Katakate gives every sandbox its own VM-level network namespace and applies opinionated network policies on top. This page explains the model. ## CNI plugin: Cilium by default By default, K3s is installed with **Flannel disabled** (`--flannel-backend=none --disable-network-policy`) and **Cilium** as the CNI. Cilium runs with `kubeProxyReplacement=true` (no kube-proxy), and its eBPF datapath attaches to the host-side veth — including for Kata VMs, whose virtio NICs are bridged to a host veth. ```bash # Cilium is the default k7 install --backend k7d --api-mode development # Cilium # Opt out (CIDR-only egress, no FQDN policies, no platform deny) k7 install --backend k7d --api-mode development --cni flannel ``` The Helm chart, which installs onto a cluster you already run, exposes the same choice as `networkPolicy.provider`: `cilium` (FQDN + CIDR egress, platform deny, agent ingress policy; the pre-install hook requires Cilium's DNS proxy with `dnsProxy.dnsRejectResponseCode=nameError`), `kubernetes` (v1 `NetworkPolicy`, CIDR egress only — a domain entry is `400`), or `none` (no egress policy; any egress field is `400`). The weaker providers need explicit `acknowledge*` values. When the cluster is on Cilium, k7 uses **`CiliumNetworkPolicy`** for FQDN egress. On Flannel it falls back to the standard Kubernetes **`NetworkPolicy`** with CIDR-only `ipBlock` rules. ## Sandboxes never reach the platform (always on, Cilium) A cluster-wide, deny-only `CiliumClusterwideNetworkPolicy` named **`k7-sandbox-platform-deny`** is installed by `k7 install`. It stops every sandbox — in **every** egress mode, including `--egress-open` — from reaching: - the node it runs on and the other nodes (SSH, kubelet, k3s ports), - the Kubernetes API server (`10.43.0.1:443`), - link-local / cloud metadata (`169.254.0.0/16`), - the `kube-system` and `longhorn-system` pods — `k7-api`, `k7-agent`, Longhorn — **except CoreDNS**, which stays reachable so FQDN egress keeps working. It is deny-only (`enableDefaultDeny: false`), so it never narrows what a whitelist allows beyond those targets, and it cannot turn an open sandbox into a default-deny one. **Cilium-only:** `--cni flannel` clusters do not get this isolation. ## Ingress: denied by default, opt-in per sandbox Every sandbox gets a deny-all `{name}-deny-ingress` `NetworkPolicy`. Practical implications: - **Sandbox-to-sandbox networking is denied** — even within the same namespace — until you open a port. - **`kubectl exec` and `k7 shell` still work** — they go through the Kubernetes API, not pod networking. - **`kube-system` traffic is allowed** — needed for cluster services (DNS, metrics, etc.). To accept connections, open TCP ports and say who may connect: | Setting | CLI | YAML / API | Result | |---|---|---|---| | **deny-all** (default) | `k7 create …` (no ingress flag) | omit `ingress_ports` | deny-all ingress NetworkPolicy | | **open TCP ports** | `--ingress-port ` (repeatable) | `ingress_ports: []` | listed ports reachable from sandboxes in the **same namespace** | | **scope the sources** | `--ingress-from ` (repeatable) | `ingress_from: []` | only the listed sources reach those ports | Sources: | Entry | Matches | |---|---| | `sandbox:` | that sandbox, same namespace (`podSelector` on `katakate.org/sandbox`) | | `namespace:` | every pod in that namespace (`namespaceSelector` on `kubernetes.io/metadata.name`) | | `cidr:` | clients in that CIDR (`ipBlock`) | Anything else is a hard error at create time. `--ingress-from` without `--ingress-port` is an error too: sources alone open nothing. `--ingress-port` with no `--ingress-from` means "other sandboxes in this namespace", **never the world** — reaching a sandbox from anywhere requires `--ingress-from cidr:0.0.0.0/0`, typed out in full. These are stock v1 `NetworkPolicy` rules, so ingress works on `--cni flannel` as well. **`cidr:` does not scope in-cluster peers on Cilium.** Cilium does not evaluate an `ipBlock` peer when both ends are Cilium-managed pods, so a rule whose peers are all `cidr:` degrades to a *port-only* allow inside the cluster — the port answers **any sandbox in the cluster**, even when the CIDR contains no cluster IP. `k7 create` warns on stderr whenever `--ingress-from` contains a `cidr:` entry. Use `cidr:` for **external** clients (that is what `--expose-port` needs it for) and `sandbox:` / `namespace:` to scope in-cluster access — those are evaluated by pod identity. A sandbox is a bare Deployment with no Service, so ingress rules only cover in-cluster traffic. To reach a sandbox from outside the cluster, expose it. ## Exposing a sandbox outside the cluster `--expose-port ` (repeatable; `expose_ports` in YAML / API) creates a `NodePort` Service named `{name}-expose` in the sandbox's namespace. `k7 create` prints the resolved `http://:` for each port, and `k7 list` shows the allocated NodePorts. ```bash k7 create web python:3.12-slim \ --backend k7d --egress-open \ --ingress-port 8000 --ingress-from cidr:0.0.0.0/0 --expose-port 8000 \ --before-script 'nohup python3 -m http.server 8000 >/tmp/http.log 2>&1 &' # 🌐 Port 8000 exposed at http://:31234 # That URL is the node *running* the pod. python:3.12-slim does not listen # on 8000 by itself. On k7d/kfd the container pid 1 is `sleep 365d`, so a # server started from --before-script must be `nohup … &` or it dies with # the script's shell. `--cmd` only replaces pid 1 on kql. ``` - **Every `--expose-port` needs a matching `--ingress-port`.** k7 refuses otherwise: a public NodePort in front of a deny-all policy is a confusing no-op, and one in front of an accidentally-open policy is a breach. - The Service uses `externalTrafficPolicy: Local`, so the client's real source IP reaches the pod and `--ingress-from cidr:…` rules mean something. Consequence: the NodePort **only answers on the node running the sandbox** (`k7 list` shows which one). - A `cidr:` allowlist does not constrain clients on the cluster's own nodes; verify it from a genuinely external client — `curl` from a node passes no matter how the rule is written. - On a bare-metal host the node IP is public, so `--expose-port` with `--ingress-from cidr:0.0.0.0/0` publishes an untrusted sandbox to the internet. k7 warns on stderr on every such create. - `k7 delete` removes the Service. A `k7 fork` inherits the source's ingress and egress **policy** but not its Service — a fork is not reachable from outside until you expose it. ## Egress control A sandbox has exactly one of three egress modes: | Mode | CLI | YAML / API | Result | |------|-----|------------|--------| | **block-all** | `k7 create …` (no egress flag) | `egress_whitelist: []` | deny-all egress NetworkPolicy | | **whitelist** | `--egress ` (repeatable) | `egress_whitelist: []` | only listed CIDRs / FQDNs | | **open** | `--egress-open` | omit `egress_whitelist` (or `null`) | no egress policy | `--egress-open` and `--egress` are mutually exclusive. Defaults are **asymmetric** (kept for compatibility): the CLI without flags is **block-all**; an API/YAML request that omits `egress_whitelist` is **open**. | `egress_whitelist` value | Behavior | |-------|----------| | Field omitted (or `null`) | Egress fully open — sandbox can reach the internet | | `[]` (empty list) | Egress fully blocked. **DNS is also blocked.** | | `["10.0.0.0/8", "1.1.1.1/32"]` | CIDR allowlist. DNS is blocked unless explicitly whitelisted. Without Cilium, a CIDR that contains the cluster DNS server is refused (`400`). | | `["api.openai.com", "*.huggingface.co"]` | **FQDN allowlist** (requires Cilium). Cluster CoreDNS is reachable through Cilium's DNS proxy, which answers **only the allowlisted names**; every other query gets `NXDOMAIN` and never leaves the node. | | Mixed CIDRs and FQDNs | Both kinds combined into a single `CiliumNetworkPolicy`. | ### How FQDN egress works under Cilium When the whitelist contains FQDNs, k7 creates a `CiliumNetworkPolicy` like: ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: -egress spec: endpointSelector: matchLabels: katakate.org/sandbox: egress: # Cluster DNS, with rule that observes lookups for FQDN matching - toEndpoints: - matchLabels: "k8s:io.kubernetes.pod.namespace": kube-system "k8s:k8s-app": kube-dns toPorts: - ports: [{ port: "53", protocol: UDP }, { port: "53", protocol: TCP }] rules: dns: # the same matchers as toFQDNs — nothing else resolves - matchName: "api.openai.com" - matchPattern: "**.huggingface.co" # FQDN allowlist - toFQDNs: - matchName: "api.openai.com" - matchPattern: "**.huggingface.co" # Optional CIDR additions - toCIDR: - "10.0.0.5/32" ``` Wildcards (`*.example.com`) are supported via Cilium's `matchPattern`. Bare exact names use `matchName`. A leading `*.` covers subdomains at **any depth** (`registry.docker.com` *and* `production.cloudfront.docker.com`) but **not** the apex `docker.com` — add the apex as its own entry. Under the hood k7 maps `*.` to Cilium's multi-label `**.` `matchPattern` so CDN-backed registries work. The install also sets Cilium `dnsProxy.minTtl=3600` so clients that cache a resolved IP longer than a short CDN TTL keep the FQDN→IP allowance for an hour. **No DNS exfiltration (0.5.0).** The DNS rule to kube-dns carries the allowlist's own matchers, not `matchPattern: "*"` as in 0.4.0. Cilium's DNS proxy forwards only allowlisted names and answers every other query itself with `NXDOMAIN` — the query never reaches CoreDNS or its upstream, so a sandbox cannot tunnel data out through lookups for names it may not reach. A program that resolves a forbidden name now fails at the lookup instead of at connect; in-cluster names (`*.svc.cluster.local`) resolve only if allowlisted. The install sets `dnsProxy.dnsRejectResponseCode=nameError` (the default `REFUSED` makes musl abandon its `ndots:5` search list) and `k7 install` applies it to existing clusters through `cilium upgrade`; k7-api rewrites older sandbox policies at startup. ### When DNS is blocked If the whitelist is `[]` or contains only CIDRs, DNS resolution is **denied** by the policy. To work around this: - Use an FQDN allowlist on Cilium — names you list resolve, nothing else does. - Pre-resolve names in `before_script` (which runs **before** the egress policy is applied). - Embed your dependencies into the image rather than fetching them at runtime. - Whitelist your own internal DNS server / egress proxy and resolve there. **Do not whitelist public DNS resolvers** like `1.1.1.1` or `8.8.8.8` — that re-enables DNS-over-UDP/TCP/443 (DoH) and defeats the lockdown. Prefer a single egress proxy IP and enforce DNS / DoH policy at the proxy. Or, even simpler, just use FQDN egress on Cilium. ### Bandwidth caps `network_limits` (`--egress-bps` / `--ingress-bps`, bits per second) and a tenant's `limits.network` cap a sandbox's bandwidth **per VM at the hypervisor** — Firecracker token buckets on `kfd`, `tc` on the tap for `kql` — so they work whatever the CNI and in both directions. A request may lower a tenant cap, never raise it. Firecracker's bucket starts full, so the first second can burst to twice the cap; the 30-second average stays within 10 %. `k7d` and `k7d-fc` have no limiter yet: a cap configured for them is refused with `400 LimitNotEnforceable` rather than ignored. See [Tenants](/k7/guides/tenants-and-roles#per-vm-limits). ## CoreDNS upstreams By default CoreDNS forwards to the node's own resolvers (k3s's `--resolv-conf` points at systemd-resolved's upstream list). If the provider's resolvers degrade, every pod's lookups fail with them — image pulls, FQDN egress, and `POST /api/v1/sandboxes`, which then returns `503` "Cannot resolve registry host". Pin them instead: ```bash k7 install ... --dns-upstream 1.1.1.1 --dns-upstream 9.9.9.9 # inventory: k7_dns_upstreams under [k7_cluster:vars] ``` 1 to 3 IPv4 addresses (the kubelet passes at most 3 and silently drops the rest); IPv6, `0.x` and `127.x` are refused before any host is touched. The playbook writes `/etc/k7/resolv.conf`, sets `K3S_RESOLV_CONF`, and converges an existing cluster on re-run; removing the variable restores the host's resolvers. Side effect: every `dnsPolicy: Default` pod gets the same resolvers. The Helm chart does not manage CoreDNS — configure upstreams in your distribution (on RKE2, the `rke2-coredns` HelmChartConfig). ## Network policies created per sandbox For each sandbox, k7 creates: 1. A **`{name}-deny-ingress`** `NetworkPolicy` (always; `--ingress-port` adds allow rules to this same object). 2. **Either**: - A `NetworkPolicy` with CIDR-only `ipBlock` egress rules (when only CIDRs are listed, **or** the cluster runs Flannel), **or** - A `CiliumNetworkPolicy` for FQDN+CIDR egress (when FQDNs are present and Cilium is installed). 3. A `{name}-expose` `NodePort` Service, only with `--expose-port`. On top of these, the cluster-wide `k7-sandbox-platform-deny` policy applies to every sandbox, and the `k7-metrics` policy keeps port 9464 of `k7-api` / `k7-agent` to the scraper you named ([Observability](/k7/guides/observability-metering#metrics)). All per-sandbox objects are deleted with the sandbox. A `k7 fork` inherits the source's ingress and egress configuration and is rolled back if its policies cannot be created — a fork of a locked-down sandbox never comes up open. On k7d 0.9.0+, the forked VM also keeps a working NIC (its own bridge, NAT'd into the fork pod's netns), so the inherited policy actually has traffic to govern. Only k7-api may set the runtime annotations these policies rely on: the ValidatingAdmissionPolicy `k7-runtime-annotations` rejects `k7d.katakate.org/*` and `io.katacontainers.*` annotation changes by anyone else, so a namespace admin cannot move a pod's VM onto a shared k7d bridge and bypass Cilium egress. ## Administrative access `kubectl exec`, `k7 shell`, the API's `/exec` endpoint, and `k7 logs` all bypass network policies because they ride the Kubernetes API. Pod networking restrictions never affect operator access. ## Restricting who can reach `k7-api` `k7-api` is a `NodePort` Service on `31007`, served over HTTPS by default (see [Security model](/k7/concepts/security-model#api-transport-tls)). On a dedicated box the node IP is public. `k7 install --api-allow-cidr 203.0.113.4/32 --api-allow-cidr 198.51.100.0/24` (repeatable, Cilium only — `--cni flannel` fails at the CLI) renders a `CiliumNetworkPolicy` `k7-api-ingress` on the API pod and flips the Service to `externalTrafficPolicy: Local` so the pod sees the real client IP. Both are skipped entirely when the flag is unset, which is the default. This is defence in depth for operators who know their client CIDRs; it stacks with TLS and does not replace it. Things worth knowing before you debug it: - With `Local`, the NodePort answers **only** on the node running the `k7-api` pod — the first master, which is the address the playbook writes into `/etc/k7/api_endpoint`. - **A curl from a node is not a test of the allowlist.** Node-local traffic matches Cilium's `host` entity and is SNAT'd to the pod network, so it succeeds however wrong the CIDR list is. Test from off-cluster. - A wrong CIDR only costs API access: the policy selects the `k7-api` pod endpoint, never the host, so SSH, `kubectl`, and on-node `k7 --core` keep working. - Rollback: re-run `k7 install` without the flag, or delete `ciliumnetworkpolicy/k7-api-ingress` in `kube-system` and patch the Service back to `externalTrafficPolicy: Cluster`. ## Debugging policy drops (Hubble) Hubble is **opt-in observability**, not enforcement — off by default, not a security control. Enable it with `k7 install --hubble` (Cilium only; `--hubble --cni flannel` fails loudly). It installs Hubble Relay and a pinned `hubble` CLI; the Hubble UI is **not** installed (no authentication, public node IP). Rollback: `cilium hubble disable`. ```bash export KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium hubble port-forward & # kube-system/hubble-relay → 127.0.0.1:4245 hubble status hubble observe --verdict DROPPED --last 50 hubble observe --namespace --verdict DROPPED --last 50 cilium fqdn cache list # what Cilium learned for FQDN rules cilium ip list # which IPs carry an fqdn: identity ``` When a sandbox with an FQDN whitelist cannot reach a host, the symptom is always a hang — policy drop, DNS, MTU, and a dead upstream look the same from inside the VM. A `Policy denied DROPPED` verdict with the destination still labelled `world` means the FQDN→identity mapping missed (wildcard too narrow, or the learned IP expired). ## Multi-node considerations - Cilium routes pod traffic across nodes natively (no Flannel VXLAN). - `CiliumNetworkPolicy` applies cluster-wide regardless of which node the sandbox runs on. - Longhorn replication traffic between nodes is **not affected** — it goes through the host network (Cilium's host-level rules), not the sandbox's policy. # Security model Source: https://docs.katakate.org/k7/concepts/security-model > VM isolation, jailer, capabilities, non-root, network, API identity (roles, tenants, OIDC), supply chain, and production mode Katakate is built for **executing untrusted code** — model output, third-party scripts, anonymous CI jobs. The threat model assumes the workload inside a sandbox is hostile and tries to escape, exfiltrate data, or attack other tenants. This page summarizes the layered defenses; for hands-on configuration see the [API security & networking reference](/k7/api/security). ## Defense in depth Each boundary below has to be defeated before an attacker reaches the next one. The untrusted workload is at the center; the cluster is the outermost ring. ```mermaid flowchart TB Attacker(["attempted escape
(malicious workload)"]) subgraph Cluster["Cluster boundary — K3s + Cilium"] direction TB C0["Cluster-wide platform deny
sandboxes never reach nodes,
apiserver, metadata, kube-system"] C1["NetworkPolicy
ingress denied by default, opt-in ports"] C2["Egress locked per sandbox
(CIDR or FQDN allowlist)"] C3["Per-sandbox filesystem
devmapper LV / Longhorn PVC / k7d disk image"] subgraph Host["Host boundary — Kata or k7d + KVM"] direction TB H1["Kata Containers or k7d shim
pod = lightweight VM"] H2["Firecracker / QEMU / k7d VMM
hardware virtualization (KVM)"] H3["Firecracker jailer (kfd, k7d-fc)
chroot · dropped caps · seccomp"] H4["Minimal device exposure
(virtio only)"] subgraph Container["Container boundary — pod spec"] direction TB K1["cap_drop: ALL"] K2["allowPrivilegeEscalation: false"] K3["Seccomp: RuntimeDefault"] K4["Optional non-root
UID/GID 65532"] subgraph Workload["Untrusted workload"] W["model output · third-party scripts
anonymous CI jobs"] end end end end Attacker -. "must defeat every layer" .-> Workload classDef cluster fill:#0b1a0b,stroke:#22c55e,color:#fff; classDef host fill:#2a1a05,stroke:#ef672b,color:#fff; classDef container fill:#1a1a1a,stroke:#888,color:#fff; classDef workload fill:#0f1a2a,stroke:#3b82f6,color:#fff; classDef attacker fill:#1a0b0b,stroke:#ef4444,color:#fff; class Cluster,C0,C1,C2,C3 cluster; class Host,H1,H2,H3,H4 host; class Container,K1,K2,K3,K4 container; class Workload,W workload; class Attacker attacker; ``` A successful escape from the workload has to defeat every layer above before it can reach another tenant or the host. ## VM isolation Every sandbox is a Kata Containers pod, which means **a real VM with its own kernel** running on KVM. The workload sees a virtio block device for its rootfs, a virtio NIC, and not much else. The virtual hardware surface is tiny compared to a shared host kernel. - **`kata-firecracker-devmapper`** uses Firecracker — ~50k LoC Rust microVMM, ~5MB resident, ~125ms boot. The Firecracker process runs **inside the jailer**: a chroot jail with an empty filesystem, dropped capabilities, a restrictive seccomp filter, and an unprivileged UID. An integration test (`tests/integration/test_firecracker.py::test_jailer_active`) verifies the jailer is active on every install. - **`kata-qemu-longhorn`** uses QEMU — bigger surface than Firecracker but still hardware-isolated, and Kata applies its own sandboxing primitives. - **`k7d`** uses Katakate's own KVM VMM (`runtimeClassName: k7`). One daemon hosts every VM; [privilege separation](/k7d/concepts/security#privilege-separation) splits that daemon into a root node broker (no listening socket) and a uid-`k7d` VM broker that parses guest and shim input. Live CoW fork still requires sibling forks of one tree to share the VM broker's address space — that is what makes the ~5 ms fork possible, and it means sibling forks of one tree are isolated more weakly than VMs on separate VMMs. See the [k7d security model](/k7d/concepts/security). - **`k7d-fc`** keeps the k7d daemon for fork / pause but boots each guest on **upstream Firecracker under the upstream jailer** (`runtimeClassName: k7-fc`): a separate VMM process per VM with its own uid, chroot, pid namespace, cgroup, and Firecracker's seccomp. A guest→VMM escape lands in that jail, not in the daemon. Pick it for multi-tenant `--docker`. Trade-offs: no hostPath / virtiofs, no time warp, a 32 MiB / 50m PodOverhead. On both Kata backends the pod's `securityContext` reaches the runtime inside the VM: Kata's guest seccomp is enforced (`disable_guest_seccomp = false`), so the container's `RuntimeDefault` profile is applied by the in-guest runtime rather than dropped. The k7d shim likewise passes `linux.seccomp`, `maskedPaths` / `readonlyPaths`, device cgroups, capabilities, and `noNewPrivileges` through to guest `runc`, stripping only what the guest kernel cannot honour (AppArmor profile, host cgroup path, host namespace paths). ## Container hardening Inside the VM, the user container is further locked down: | Knob | Default | What it does | |------|---------|-------------| | `cap_drop` | `["ALL"]` | All Linux capabilities are dropped unless you explicitly add some back via `cap_add`. | | `cap_add` | `[]` | Add back only what your workload needs (e.g. `["CHOWN"]`). | | `allowPrivilegeEscalation` | `false` | Always — `setuid`/file caps cannot raise privileges. | | Seccomp profile | `RuntimeDefault` | Standard runtime seccomp filter, enforced by the in-guest runtime on top of Kata/jailer's own. | | `pod_non_root` | `false` | Set `true` to run the entire pod as UID/GID/FSGroup 65532 (consistent FS ownership). | | `container_non_root` | `false` | Set `true` to run the main container as UID 65532. | Some package managers (`apk add`, `apt-get install`) require root inside the container. Either run them in `before_script` with the defaults, or prebuild your image with the dependencies baked in. `--docker` changes the picture inside the VM. On `k7d` / `k7d-fc` the sandbox container stays unprivileged, but `docker.sock` is guest root — anything that can reach it owns the guest, so the container `securityContext` no longer bounds the VM. On `kfd` / `kql` a privileged `docker-vehicle` container runs in the same Kata VM. In every case the isolation boundary is the VM; see [Docker in a sandbox](/k7/guides/docker). ## Network isolation - **Sandboxes never reach the platform.** A cluster-wide deny-only `CiliumClusterwideNetworkPolicy` (`k7-sandbox-platform-deny`) blocks every sandbox — even `--egress-open` ones — from the nodes, the Kubernetes API server, cloud metadata / link-local, and `kube-system` / `longhorn-system` pods except CoreDNS. Cilium-only; Flannel clusters do not get this. - **Ingress is denied by default** and opt-in per sandbox: `--ingress-port` opens TCP ports, `--ingress-from` (`sandbox:` / `namespace:` / `cidr:`) scopes who may connect, `--expose-port` publishes through a `NodePort` with the real client IP preserved. Opening a port without a source means "other sandboxes in this namespace", never the world. `kubectl exec` / `k7 shell` are unaffected (they use the K8s API). - **Egress is configurable** per sandbox via `egress_whitelist`: open by default, blocked with `[]`, restricted by CIDRs, or restricted by FQDNs (with Cilium). With `[]` or a CIDR list, DNS is blocked. With an FQDN list, Cilium's DNS proxy answers **only the allowlisted names** and returns `NXDOMAIN` for everything else — other queries never reach CoreDNS or its upstream, closing the DNS-exfiltration channel that 0.4.0's `matchPattern: "*"` rule left open. Without Cilium, a CIDR entry that contains the cluster DNS server is refused. Forks inherit the source's policy. - **Only k7 sets runtime annotations.** The ValidatingAdmissionPolicy `k7-runtime-annotations` denies any change to `k7d.katakate.org/*` and `io.katacontainers.*` annotations on pods and pod templates except by k7-api, `system:masters` and the controller manager — so a namespace admin cannot put a Deployment's VM on a shared k7d bridge, fork another VM on the node, or pick a host Kata config. Containerd forwards only the keys k7 sets. - **Per-VM bandwidth caps** (`network_limits`, tenant `limits.network`) are enforced by the hypervisor on kfd and kql; on k7d / k7d-fc a configured cap is refused rather than ignored. - Details and the `cidr:`-on-Cilium caveat: [Networking](/k7/concepts/networking). ## API transport (TLS) `k7 install` serves `k7-api` on NodePort `31007` over **HTTPS** by default. The API container stays HTTP on `:8000` (kubelet probes unchanged); a Caddy sidecar in the same pod terminates TLS. API keys no longer travel in cleartext. | `k7 install` flag | Certificate | |---|---| | (none) | Playbook-minted cluster CA at `/etc/k7/tls/ca.crt` (Let's Encrypt cannot issue for a bare IP). Copy that file to your laptop and `k7 config set api.ca ./ca.crt`, or pass `--api-ca` / `K7_API_CA`. | | `--api-hostname NAME` | Let's Encrypt via Caddy (HTTP-01 on host port 80). `NAME` must be a DNS name whose A record is the first master's IP; a bare IP is refused. Clients use the system trust store. | | `--api-tls-cert` + `--api-tls-key` | Operator-supplied PEM pair. Incompatible with `--api-hostname`. | | `--api-insecure-http` | The previous plain-HTTP NodePort, bit for bit. Keys travel in cleartext. Cannot be combined with the hostname or cert flags. | The CLI and SDK refuse `verify=False` on an `https://` URL — point them at the CA instead. Optionally restrict the NodePort to your client CIDRs with `--api-allow-cidr` ([Networking](/k7/concepts/networking#restricting-who-can-reach-k7-api)). None of this is rate limiting. ## API identity: keys, roles, tenants, OIDC - **Every credential has a role** — `read-only` / `user` per tenant, `operator` / `admin` cluster-wide — checked inside k7-api on every route from one `PERMISSIONS` table. K7 roles are not Kubernetes RBAC; nobody gets a Kubernetes identity through K7. See [Tenants, roles and OIDC](/k7/guides/tenants-and-roles). - **API keys** are `k7__` (256 random bits), shown once. The store holds `sha256(secret)` in the Kubernetes Secret `kube-system/k7-api-keys` — never on a node's disk, never written on authenticate. k7-api serves a snapshot refreshed every 5 s and answers 503 rather than trust one older than 60 s; revocation reaches every replica within one refresh. The `k7-api` ServiceAccount can only `get` / `patch` that Secret. Protect etcd (`--secrets-encryption`, KMS). - **Tenant scope** is the control-plane tenancy boundary: a tenant key reaches its tenant's namespaces (narrowed by `-n`), cross-tenant requests are `403` before any cluster call, fork and restore never leave their namespace, and quotas are enforced under a per-tenant Lease. **Node scope** (`--node`) plus `k7 nodes dedicate` (label + `NoSchedule` taint) is the placement boundary that keeps a tenant off another tenant's k7d daemon. Cluster-scoped reads such as `GET /api/v1/nodes/storage` need `operator`/`admin` (CWE-862 fix, reported by [@skeletonsec](https://github.com/skeletonsec)). - **OIDC** (`k7 install --oidc-*`): humans send `Authorization: Bearer `; k7-api validates statelessly against the issuer's JWKS (RS256/ES256, cache expires, failed refresh is 503), maps groups/users/subjects to roles through `k7-role-bindings`, and keeps no sessions. `deny-subjects` is the immediate stop; a JWT otherwise lives until `exp`. - **Audit log**: one JSON line per mutating request and per 401/403 on k7-api stdout (`k7.audit`) with principal, role, tenant, route, status, latency and client IP; never tokens, groups, bodies or exec text. Emission failure fails the request closed. - **Registry SSRF guard**: control-plane OCI inspection rejects registry hosts that are not on the allowlist (default `registry-1.docker.io`, `ghcr.io`, `quay.io`, `public.ecr.aws`; extend with `K7_REGISTRY_ALLOWLIST`) or that resolve to loopback / private / link-local / reserved addresses. Redirects are disabled. - **Remote install** (`POST /api/v1/install`) is off by default (404) and, when enabled with `--api-remote-install` in development mode, takes only node names from a cluster-wide `admin`: the installer image and config are fixed at deploy time, and the API never runs Ansible. Up to 0.4.0 any valid key could post playbook content. The API runs as a K3s Deployment (2 replicas, uid 1000, all capabilities dropped) with its own ServiceAccount and a scoped ClusterRole — no admin kubeconfig is ever mounted. `k7-agent` is **not privileged**: uid 0 with the `k7d` group (gid 770), all capabilities dropped, read-only root filesystem, `RuntimeDefault` seccomp, no ServiceAccount token, and only the containerd socket directory, `/run/k7d`, `/var/lib/k7d` (ro) and `/run/k7` (ro) mounted. ## Production and development modes `k7 install --api-mode` (chart `api.mode`) is required and never defaulted. **`production`** refuses to start while the key store holds a key without a tenant or cluster role (pre-role, migrated `unscoped_legacy`, or namespace-only) and names each one; returns `403` to such keys on every route; requires CPU and memory limits on every sandbox (filled from the tenant's `defaultSize`); refuses `--api-insecure-http` and remote install; and makes metering explicit (`--metering-sample-interval`). **`development`** keeps unscoped keys working as cluster `admin`. Every response carries `X-K7-Mode`. ## Supply chain Tags build `ghcr.io/katakate/k7-api` and `ghcr.io/katakate/k7-node-installer`, scan them with Trivy, sign them with cosign keyless, and attach an SPDX SBOM and SLSA provenance; both digests go into the release's `images.lock.yaml`, which is what `k7 install` deploys (`imagePullPolicy: IfNotPresent`, signature verified). The `.deb` and wheel ship with `sha256sums.txt` and cosign bundles. Every install download (K3s script, CNI, Firecracker, Kata, k7d, Cilium CLI, Hubble, Longhorn, Caddy by digest) is checked against a pinned sha256; a mismatch is fatal. Building `k7-api:local` from a checkout (`--api-image local`) is the unsigned opt-out and there is no silent fallback between the two. ## Operational guidance 1. **Always lock down egress** for untrusted workloads. Prefer FQDN allowlists (Cilium) — much less error-prone than tracking CIDR ranges that change. 2. **Don't whitelist public DNS resolvers** (`1.1.1.1`, `8.8.8.8`) — that re-enables DNS exfiltration. 3. **Rotate API keys** with `k7 revoke-api-key` + `k7 generate-api-key --role …`. Treat them as credentials. Give machines tenant keys (`--role user --tenant T`, `--expires-days`) and humans OIDC; keep one break-glass `admin` key offline; run `--api-mode production` once no key is flagged `unscoped_legacy`. 4. **Keep TLS on and distribute the CA** (default install) — or use `--api-hostname` for a publicly trusted certificate. Add `--api-allow-cidr` when you know your client CIDRs. Do not fall back to `--api-insecure-http` on a public node. 5. **Be deliberate with ingress.** Open only the ports you need, scope them with `sandbox:` / `namespace:`, and treat `--expose-port` + `cidr:0.0.0.0/0` as publishing to the internet. 6. **Pin image tags** — `:latest` is convenient but non-reproducible. K7 itself pins every image and Kubernetes manifest version it ships; do the same in your sandbox configs. 7. **Dedicate nodes to tenants** that must not share a k7d daemon (`k7 nodes dedicate`), and keep tenant boxes in `[k7_agents]`: root on a `[k7_servers]` node is cluster-admin, root on an agent is that node only (`k7-agent` has no ServiceAccount token). 8. **Set quotas** per tenant and ship the audit log and usage events ([Observability and metering](/k7/guides/observability-metering)). ## Known limitations - **Pre-1.0 software** under active security review. Avoid for hyper-sensitive workloads until 1.0. - **No API rate limiting yet** — put a proxy with rate limits in front for public exposure. TLS and `--api-allow-cidr` are not a "secure API" claim. - **Platform isolation, FQDN egress and `--api-allow-cidr` are Cilium-only** — `--cni flannel` clusters (and the chart's `networkPolicy.provider: kubernetes`) keep per-sandbox CIDR policies but not the cluster-wide deny or domain allowlists. - **Quota enforcement is at the K7 API**; `kubectl apply` by a cluster admin bypasses it (single-namespace tenants get a `ResourceQuota` backstop). `diskIO` limits are accepted by the Tenant CRD but no backend enforces them yet; k7d / k7d-fc cannot enforce `network` caps or `disk` quotas — such creates are refused (`LimitNotEnforceable`) rather than run unlimited. - **Host compromise is cluster-wide on servers.** `--node` / `dedicate` isolate k7d placement, not a root shell on `[k7_servers]`. - **An OIDC token is valid until it expires**; `deny-subjects` is the only revocation. Entra ID group overage (> ~200 groups) is not followed — bind by `user` / `subject` or use app roles. - **No independent security audit yet.** - **AppArmor profiles** for the sandbox container are on the roadmap (the k7d Ubuntu guest ships its own). - **TEE (Trusted Execution Environment)** support is on the roadmap. - Tetragon (runtime enforcement) was evaluated and **rejected** for now; Hubble observability is available opt-in. See [SECURITY.md](https://github.com/Katakate/k7/blob/main/SECURITY.md) for the full disclosure policy and reporting channel (security@katakate.org). # CLI reference Source: https://docs.katakate.org/k7/guides/cli > All k7 commands with options and examples Use `k7 -h` for built-in help. Below are the primary commands. ## Talking to the K7 API By default, every sandbox-management command (`create`, `list`, `delete`, `pause`, `resume`, `fork`, `restore`, `exec`, `logs`, `snapshot *`) talks to the **K7 API** over HTTPS. The CLI resolves the URL, key, and CA in this order: 1. `--api-url` / `--api-key` / `--api-ca` flags (global, before the command). 2. `K7_API_URL` / `K7_API_KEY` / `K7_API_CA` environment variables. 3. `~/.config/k7/config.toml` — written by `k7 config set api.url ...` / `k7 config set api.key ...` / `k7 config set api.ca ...`. 4. On a cluster node only: `/etc/k7/api_endpoint` and `/etc/k7/tls/ca.crt`. The API **key** is never stored in the clear on disk — paste the token `k7 generate-api-key` printed into `k7 config set api.key`. Humans on an OIDC-enabled cluster can use `k7 login` instead of a key: the CLI caches the IdP tokens in `~/.config/k7/credentials.json` (0600), refreshes them itself and sends `Authorization: Bearer `. A configured `api.key` and a cached login are both accepted; `k7 whoami` tells you which one is in effect and what role it resolves to. ```bash # One-time setup on your laptop. The default install mints a cluster CA; # copy it off the node so the CLI can verify the API's certificate. scp root@node:/etc/k7/tls/ca.crt ./k7-ca.crt k7 config set api.url https://:31007 k7 config set api.ca ./k7-ca.crt k7 config set api.key '' # The kube-system/k7-api-keys Secret stores SHA-256 hashes, not the raw token. # Then everything Just Works without SSH'ing to the node. k7 list k7 create demo alpine:3.20 k7 fork demo demo-1 k7 exec demo -- echo hello k7 logs demo --tail 50 ``` An `https://` URL is always verified: there is no `--insecure`. If the cluster was installed with `--api-hostname` (Let's Encrypt) the system trust store is enough and `api.ca` is not needed; with `--api-insecure-http` use an `http://` URL. ### `--core` (debugging / on-node CI only) A hidden global flag `--core` skips the API and calls `K7Core` directly in-process. Useful when the API itself is misbehaving and you have kubeconfig access on the node. Most users should never need it. ```bash k7 --core list # bypass API; talk to K3s directly k7 --core logs demo --follow # required if you want live --follow today ``` ## config ```bash k7 config set KEY VALUE # KEY ∈ {api.url, api.key, api.ca} k7 config get KEY k7 config show # api.key is redacted ``` Writes `~/.config/k7/config.toml` with mode `0600`. Same posture as `~/.kube/config` and `~/.docker/config.json`. `api.ca` is the path of the CA bundle to trust for the `https://` API URL (the cluster CA from `/etc/k7/tls/ca.crt` on the default install). ## install Install K7 components on host node(s). ```bash k7 install --backend kfd,kql,k7d,k7d-fc --api-mode development # both flags required; no silent default k7 install --backend k7d --api-mode development --k7d-artifact /root/k7d-v0.9.0-x86_64-linux.tar.gz k7 install -i inventory.ini --ha --api-mode production --metering-sample-interval 15 # multi-node HA (3+ servers) k7 install --role agent --join https://master:6443 --join-token TOKEN k7 install --hubble --api-allow-cidr 203.0.113.4/32 --api-mode development # observability + API source allowlist k7 install --api-hostname k7.example.com --api-mode production --metering-sample-interval 15 \ --oidc-issuer https://login.example.com --oidc-audience k7 --oidc-client-id k7-cli \ --oidc-groups-claim groups --oidc-username-claim email # Let's Encrypt cert + OIDC login ``` Since 0.5.0, `k7 install` runs from any directory: it deploys the signed GHCR `k7-api` image by digest and needs neither a checkout nor Docker on the node (`--api-image local` restores the 0.4.0 build-from-cwd path). - **-v**: verbose output - **--backend, -b**: required. Comma-separated backends: `kata-firecracker-devmapper` (`kfd`), `kata-qemu-longhorn` (`kql`), `k7d`, `k7d-fc`, or `none` (control plane only, no sandbox runtime). Omit is an error. `k7d-fc` adds jailed Firecracker (installs pinned Firecracker **v1.16.2** + jailer, RuntimeClass `k7-fc`). - **--api-mode**: required with the API (no default). `production`: every `read-only`/`user` key needs a tenant, every sandbox needs CPU and memory limits (filled from the tenant's `defaultSize`), HTTPS only, metering mandatory, remote install refused. `development`: unscoped keys keep working as `admin`. See [Production and development modes](/k7/concepts/security-model#production-and-development-modes). - **--disk**: block device for the kfd LVM thin-pool (auto-detected when omitted). On dual-NVMe boxes leave this unset — NVMe names swap across reboots. Inventory equivalent: `k7_disk_source=spare-disk|loopback|device:/dev/`. - **--k7d-version**: k7d release to download (k7d backends only). Playbook default is **`0.7.0`**; the playbook only knows that version's sha256, so another version needs `k7d_sha256=` as an inventory/extra var or `--k7d-artifact`. - **--k3s-version**: override the pinned K3s (`v1.36.5+k3s1`). Only applies when K3s is not installed yet. - **--k7d-artifact**: local k7d release tarball to install instead of downloading (k7d backends only). - **-i, --inventory**: Ansible inventory for multi-node installs (per-host `k7_backends` is authoritative — don't combine with `--backend`). - **--ha**: multi-master HA with embedded etcd (inventory with 3+ servers). - **--role / --join / --join-token**: add a single node to an existing cluster. - **--cni**: `cilium` (default; FQDN egress, platform isolation, `--api-allow-cidr`) or `flannel` (CIDR-only egress). - **--hubble**: opt-in Cilium Hubble flow observability (relay + pinned `hubble` CLI, no UI). Requires Cilium. Not a security control. - **--dns-upstream IP**: repeatable (IPv4, at most 3). Upstream resolvers for CoreDNS instead of the host's `/etc/resolv.conf` (inventory `k7_dns_upstreams`). See [Networking](/k7/concepts/networking#coredns-upstreams). - **--metrics-scraper NS:label=value[,…]**: which pods may scrape `/metrics` on `k7-api` and `k7-agent` port `9464`. Without it no pod can reach the port. - **--monitoring**: also install the pinned kube-prometheus-stack (namespace `monitoring`) with K7's dashboards and alerts; implies the scraper selector. A convenience for single-box installs. See [Observability and metering](/k7/guides/observability-metering). - **--metering-sample-interval N**: usage sample period in seconds (5–60). Required with `--api-mode production`; development uses 15. Related: `--metering-storage-interval` (60), `--metering-spool-size` (1Gi), `--metering-spool-high-water` (80), `--metering-require-delivery` (503 on creates when a node's spool is above its high-water mark), `--metering-prometheus-url` (for `k7 usage`; `--monitoring` sets its own), `--metering-sinks-file` (http/OpenMeter or stdout sinks). - **--oidc-issuer / --oidc-audience / --oidc-client-id / --oidc-groups-claim / --oidc-username-claim**: all five or none; `https://` issuers only. Enables `k7 login` and bearer JWTs. `--oidc-jwks-ttl` (3600 s, min 10) is optional. Reinstalling without them turns OIDC off. - **--api-image**: `local` builds `k7-api:local` from the checkout in the cwd (unsigned, needs Docker) or `name@sha256:`. Default: the digest in the release's `images.lock.yaml`, cosign-verified. - **--api-remote-install**: enable `POST /api/v1/install` (cluster-wide `admin`), deploying the `k7-node-installer` DaemonSet in `k7-system`. Development mode only. `--node-installer-image` and `--node-installer-allow-server-restart` go with it. Re-installing without the flag removes the stack again. - **--api-allow-cidr CIDR**: repeatable. Restrict the `k7-api` NodePort to these source CIDRs via a `CiliumNetworkPolicy` and `externalTrafficPolicy: Local`. Off by default; Cilium only; `0.0.0.0/0` warns. - **--api-hostname NAME**: DNS name pointing at the first master; Caddy obtains a Let's Encrypt certificate (HTTP-01 on `:80`). A bare IP is refused. - **--api-tls-cert / --api-tls-key**: operator-supplied server certificate + key (PEM). Both required together; incompatible with `--api-hostname`. - **--api-insecure-http**: serve `k7-api` as a plain HTTP NodePort (no Caddy sidecar). Keys travel in cleartext. Cannot be combined with the hostname / cert flags. - **--no-api**: CLI-only install without the k7-api Deployment. - **--node-prep-only**: run only the node-prep role (no K3s install, no cluster resources) — what the Helm chart's installer image does on nodes of an existing cluster. See [Multi-node clusters](/k7/guides/multi-node) for the inventory format and topology, and [Security model](/k7/concepts/security-model#api-transport-tls) for how the TLS options fit together. ## version Check version of installed K7 .deb package ```bash k7 -V ``` ## create Create a sandbox from a YAML file or flags. ```bash k7 create -f k7.yaml # or k7 create my-sb alpine:latest \ --backend k7d --cpu 1 --memory 1Gi --storage 2Gi \ --env-file .env --egress 10.0.0.5/32 \ --before-script "apk add curl" # Egress: no flag = block-all; --egress CIDR|FQDN (repeatable); --egress-open = no policy k7 create open alpine:latest --egress-open # Pin to a node (name from `k7 nodes list`) and cap bandwidth (kfd/kql; bits per second) k7 create pinned alpine:latest --backend k7d --node k7-node-01 k7 create capped alpine:latest --backend kfd --egress-open --egress-bps 100M --ingress-bps 100M # Docker inside the sandbox (pinned dockerd, overlay2 on a 20Gi block disk) k7 create --docker --backend k7d --egress-open --docker-disk 40Gi builder ubuntu:24.04 # Ingress: deny-all by default; open a port to one peer sandbox, then publish it k7 create --ingress-port 8000 --ingress-from sandbox:frontend api python:3.12-slim k7 create --ingress-port 8000 --ingress-from cidr:203.0.113.0/24 --expose-port 8000 web python:3.12-slim # 🌐 Port 8000 exposed at http://:31234 # That IP is the node *running* the pod (`externalTrafficPolicy: Local`). # Curling a different node — including the first master — will not answer. # python:3.12-slim is not a web server; start one with # `--before-script 'nohup python3 -m http.server 8000 >/tmp/http.log 2>&1 & sleep 1'` # (k7d/kfd pid 1 is sleep 365d; a bare `&` is killed when before_script exits. # The trailing `sleep 1` lets before_script finish so the Ready probe can # fire — a NodePort stays in Cilium "maintenance" until Ready is True, and # curling it then times out.) ``` ## pause On **kql**, scale a sandbox to 0 replicas, optionally taking a crash-consistent `VolumeSnapshot` of its root PVC (`longhorn` `VolumeSnapshotClass`) — and of the `-docker-lh` graph PVC when the sandbox was created with `--docker`. On **k7d** / **k7d-fc**, pause freezes the live VM in place (vCPUs stop, memory stays) — no snapshot. `--snapshot` is kata-qemu-longhorn-only. ```bash k7 pause NAME # scale to 0, no snapshot k7 pause NAME --snapshot # auto-named snapshot, then scale to 0 k7 pause NAME --snapshot=demo-v1 # custom-named snapshot, then scale to 0 k7 pause NAME -n my-ns --snapshot # with namespace ``` - `--snapshot` (optional, takes optional value): when present, take a `VolumeSnapshot` of the sandbox's root PVC (`-root-lh`) before scaling down. Bare flag → snapshot named `-paused-`; with value → snapshot named as given. - Pause-time snapshots persist as `VolumeSnapshot` objects in the same namespace until you delete them. ## resume ```bash k7 resume NAME [-n NAMESPACE] ``` On **kql**, scales the Deployment back to 1; state on the root PVC survives. On **k7d** and **k7d-fc**, restarts the vCPU loop — in-memory state is intact, and `k7 exec` works right after resume (the Firecracker vsock gating that broke this on 0.3.1 is fixed by the pinned v1.16.2 — see [Backends](/k7/concepts/backends#k7d-fc)). ## restore Boot a brand-new sandbox from a standalone `VolumeSnapshot`. Unlike `k7 fork`, this does **not** require the original sandbox's Deployment to still exist — it only needs the snapshot. ```bash k7 restore SNAPSHOT_NAME NEW_SANDBOX_NAME [-n NAMESPACE] [--image IMAGE] # override (required if snapshot lacks k7.io/source-image) [--root-disk-size 20Gi] [--cpu 2] [--memory 4Gi] [--storage 10Gi] [--sidecar docker] # deprecated alias of --docker [--cmd ...] [--entrypoint ...] # repeatable [--keep-snapshot/--no-keep-snapshot] ``` Image / backend / docker / limits / root-disk-size default to the source sandbox's values via the `k7.io/source-*` annotations stamped on the snapshot at creation time (pause / named only — fork-temp snapshots are auto-deleted). Any flag above overrides the corresponding annotation. `--no-keep-snapshot` deletes the source snapshot after the new sandbox is Ready. Restore is kata-qemu-longhorn-only. ```bash # Happy path: snapshot has source annotations, no flags needed. k7 pause demo --snapshot=v1 k7 delete demo k7 restore v1 demo-v1 # returns once objects exist; wait for Ready before exec # Override the image (e.g. base-OS upgrade while keeping the persisted state). k7 restore v1 demo-v1 --image python:3.12-slim ``` ## fork Clone an existing sandbox. Behavior depends on the backend: - **`k7d`** / **`k7d-fc`** — warm CoW fork of **disk and memory** (~5 ms at the VMM, ~2–4 s end-to-end to a Ready pod). The child inherits files, tmpfs, and running processes — including a live `--docker` engine and its inner containers — and, with k7d 0.9.0+, keeps its network (the fork gets its own bridge and NAT, so source and child have egress at the same time). Lands on the source's node. Sandboxes with a real CRI sidecar cannot be forked. - **`kql` / `kata-qemu-longhorn`** — disk-only clone via a crash-consistent VolumeSnapshot of `-root-lh` (and `-docker-lh` with `--docker`), then a cold boot (~45 s single-node; longer with Longhorn replicas ≥ 2). - **`kfd`** — rejected. The fork inherits the source's egress **and** ingress policy (but not its `--expose-port` Service) and is rolled back if the policies cannot be created. Fork never leaves the source's namespace, and counts against the tenant's quotas like a create. ```bash k7 fork source-sandbox new-sandbox [--snapshot custom-name] ``` On kql only: - `--snapshot` (optional): friendly name for the temporary VolumeSnapshot. If omitted, an auto-named `-fork-` snapshot is created and **auto-deleted** once the new sandbox's PVC is bound. - Pass `--snapshot=NAME` to opt the snapshot out of GC (it becomes a `named` snapshot — persistent). ## snapshot Manage `VolumeSnapshot` objects directly. See [`/k7/api/endpoints/snapshots`](/k7/api/endpoints/snapshots) for the lifetime rules and the three kinds (`pause`, `fork`, `named`). ```bash k7 snapshot list [-n NS] [-A] # list (per-namespace or cluster-wide) k7 snapshot list --sandbox demo # only snapshots tied to a sandbox k7 snapshot list --kind pause # filter by kind: pause / fork / named k7 snapshot inspect NAME [-n NS] # full JSON dump k7 snapshot create SANDBOX SNAP_NAME [-n NS] # named snapshot of a running sandbox k7 snapshot delete NAME [-n NS] [--yes] # delete one snapshot (refuses non-TTY without --yes) k7 snapshot gc [-n NS] [-A] [--keep-fork-for 10m] [--dry-run] ``` `gc` only ever touches `kind=fork` snapshots older than `--keep-fork-for`. Pause and named snapshots are never collected, even with `--keep-fork-for=0s`. A CronJob (`k7-snapshot-gc` in `kube-system`) runs the same logic every 10 minutes as a backstop, in case the inline cleanup after `k7 fork` ever misses a snapshot. GC also reaps orphan `VolumeSnapshotContent` objects whose `VolumeSnapshot` is gone or deleting and clears the PVC source-protection finalizer, so a deleted namespace no longer sticks in `Terminating` on `volumesnapshotcontent-bound-protection`. ### YAML configuration reference All fields map to the server-side `SandboxConfig`: - **name** (string, required): unique sandbox name. - **image** (string, required): container image, e.g. `alpine:latest`. Registry host must be public and on the allowlist (`registry-1.docker.io`, `ghcr.io`, `quay.io`, `public.ecr.aws`; extend with `K7_REGISTRY_ALLOWLIST`). - **backend** (string, optional): `kata-firecracker-devmapper` / `kfd`, `kata-qemu-longhorn` / `kql`, `k7d`, or `k7d-fc`. - **node_name** (string, optional): Kubernetes node to pin the sandbox to (`k7 nodes list`); required when the API key is scoped to several nodes. - **network_limits** (object, optional): `egressBps` / `ingressBps` in bits per second (`"100M"`). kfd and kql only (Firecracker token buckets / `tc` on the tap); on k7d / k7d-fc the create is refused with 400 `LimitNotEnforceable` rather than running unlimited. May only lower a tenant cap. - **docker** (boolean, default `false`): run a pinned `dockerd` inside the sandbox with `overlay2` on a block graph disk — see [Docker in a sandbox](/k7/guides/docker). - **docker_disk** (string, default `20Gi`): graph disk size for `docker: true`. - **sidecar** (string, optional): deprecated; `docker` is an alias of `docker: true`. - **namespace** (string, default `default`): Kubernetes namespace. - **env_file** (string, optional): (absolute) path to an env file on the host node. - **egress_whitelist** (array of CIDR or FQDN strings, optional): omit / `null` = open egress; `[]` = block-all; listed entries = allowlist. FQDNs need Cilium (the default CNI). CLI `k7 create` without `--egress` / `--egress-open` is block-all. - **ingress_ports** (int[], optional): TCP ports to open for inbound traffic. Omit / `[]` = deny all ingress (default). - **ingress_from** (string[], optional): who may reach `ingress_ports` — `sandbox:`, `namespace:`, `cidr:`. Omitted with ports set = sandboxes in the same namespace. Requires `ingress_ports`. - **expose_ports** (int[], optional): publish these TCP ports outside the cluster via a `NodePort` Service; each must also be in `ingress_ports`. - **volumes** (object[], optional): hostPath / PVC entries. Refused on `k7d-fc` (Firecracker has no virtiofs). - **limits** (object, optional; required in production mode unless the tenant sets `defaultSize`): resource limits: - **cpu** (string): cores or millicores, e.g. `"1"` or `"500m"`. - **memory** (string): e.g. `"1Gi"`, `"512Mi"`. - **ephemeral-storage** (string): e.g. `"2Gi"`. - **root_disk_size** (string, `kql` only): Longhorn root PVC size. Default: the tenant's `defaultSize.disk`, else `10Gi`. - **before_script** (string, optional): shell script run once after container starts. - Runs via `kubectl exec` before egress netpol is applied; readiness waits for completion. - **pod_non_root** (boolean, optional): run Pod as non-root (UID/GID/FSGroup 65532). - **container_non_root** (boolean, optional): run container as non-root (UID 65532), no privilege escalation. - **cap_add** (string[], optional): add back Linux capabilities (default policy drops ALL). - **cap_drop** (string[], optional): override drop policy. If omitted, `ALL` is dropped by default. Example `k7.yaml`: ```yaml name: project-build image: alpine:latest backend: k7d # or kfd / kql / k7d-fc namespace: default egress_whitelist: - "10.0.0.5/32" # Private egress proxy/gateway ingress_ports: [8000] # reachable from other sandboxes in this namespace only limits: cpu: "1" memory: "1Gi" ephemeral-storage: "2Gi" before_script: | # Non-root friendly example: create a working dir and print versions mkdir -p "$HOME/work" && cd "$HOME/work" echo "PATH=$PATH" echo "whoami: $(whoami)" pod_non_root: false container_non_root: false cap_add: - CHOWN ``` Do not whitelist public DNS resolvers (e.g., 1.1.1.1, 8.8.8.8). Doing so re-enables DNS exfiltration (UDP/TCP 53 and DoH over 443). Prefer whitelisting only your own egress proxy IP and enforce DNS/DoH policies at the proxy. If using package managers that require root (e.g., `apk add`, `apt-get install`) in `before_script` make sure you didn't add security policies that prevent it such as running the pod or container as non-root. Check Security & Networking section in the API reference for more. ## list ```bash k7 list [-n NAMESPACE] ``` Lists sandboxes with status, readiness, restarts, age, image, backend, node, and the NodePorts allocated by `--expose-port` (empty when the sandbox is not exposed). The table truncates names; `k7 exec NAME` still needs the full name you passed to `create`. ## delete ```bash k7 delete NAME [-n NAMESPACE] ``` Deletes one sandbox. ## delete-all ```bash k7 delete-all [-n NAMESPACE] ``` Deletes all sandboxes in a namespace (with confirmation). ## shell ```bash k7 shell NAME [-n NAMESPACE] ``` Opens an interactive shell in the sandbox pod. This is **node-local** (`kubectl exec`); it does not go through the K7 API. From a laptop use `k7 exec NAME -- 'echo hello'` — the CLI joins the command and the API runs it under `sh -c`, so pass redirects as **one** quoted string (`k7 exec NAME -- 'echo hi > /tmp/state.txt'`). Do not add an extra `sh -c` or the quotes are lost. ## exec ```bash k7 exec NAME -- COMMAND... k7 exec demo -- echo hello k7 exec demo -- 'echo hi > /tmp/state.txt' ``` ## logs ```bash k7 logs NAME [-n NAMESPACE] [--tail 200] [-f] ``` Shows a snapshot of the CRI container's stdout (the `sandbox` container). Empty output with exit 0 means that container has not written anything — it is **not** a failure. `k7 exec` output does **not** appear here (exec goes through the agent / Kubernetes API). On `k7d` / `k7d-fc`, a sleep/alpine sandbox that you have only `exec`'d into commonly has empty logs; `before_script` on Kata backends is more likely to show up. `--follow` is `--core` only (node-local `kubectl logs -f`). ## top ```bash k7 top [-n NAMESPACE] [--refresh-interval 1] ``` Top-like view of CPU and memory usage. ## nodes storage ```bash k7 nodes storage # per-node table k7 nodes storage --json # raw JSON ``` Per-node storage-pool utilization, collected from the `k7-agent` DaemonSet on every node: - **`kata_thinpool`** — the kfd LVM thin-pool (`lvs` size, data%, metadata%). - **`k7d_disks`** — the k7d XFS disk pool at `/var/lib/k7d/disks` (`df` size / used / avail). A node whose agent is unreachable shows a loud `error` entry — nodes are never silently omitted. Also available as `GET /api/v1/nodes/storage` ([API reference](/k7/api/endpoints/nodes)) and `Client.nodes_storage()` in the SDK. This is cluster-wide topology: it needs the `operator` or `admin` role, and a tenant key gets `403`. ## nodes list / dedicate / undedicate ```bash k7 nodes list [--json] # NAME, HOSTNAME, BACKENDS, TENANT k7 nodes dedicate k7-node-01 --tenant acme # label + NoSchedule taint k7.katakate.org/tenant, adds to Tenant.spec.nodes k7 nodes undedicate k7-node-01 ``` `NAME` is the K3s Node name (the machine hostname) — what `k7 create --node`, `k7 generate-api-key --node` and `dedicate` take; it is not an inventory label. Dedication is what keeps other tenants' sandboxes off a node's k7d daemon; a key's `--node` pin alone is a placement restriction on that key, not an exclusion of others. Both commands accept `--api` to go through k7-api (`operator`/`admin`) instead of a node kubeconfig. ## api The K7 API is deployed automatically by `k7 install` and runs as a Kubernetes Deployment (`k7-api` in `kube-system`). K3s keeps it running and reschedules it on failure — there is no separate "start" step. The `k7 api` sub-app provides read-only diagnostics and a feature toggle: ```bash k7 api status # node: kubectl replica counts; laptop: GET /health on the configured URL k7 api endpoint # https://:31007 (configured URL if kubectl is missing) k7 api enable # scale back to the replica count recorded by `disable` (default 2; --replicas N overrides) k7 api disable # scale to 0 (temporary off; needs kubectl) k7 api keys migrate # copy /etc/k7/api_keys.json into the k7-api-keys Secret and give every key a role (idempotent; `k7 install` runs it) ``` `k7-api` runs **2 replicas** spread across nodes with a PodDisruptionBudget (`minAvailable: 1`), as uid 1000; the replica holding the `k7-tenant-status` / `k7-metering-leader` Leases writes `Tenant.status.usage` and the storage usage samples. The Let's Encrypt (`--api-hostname`) overlay pins back to one replica on the first master (hostPort 80 + certificate storage). The endpoint is HTTPS by default, terminated by a Caddy sidecar in the `k7-api` pod. The cluster CA lives at `/etc/k7/tls/ca.crt` on the node — copy it to clients (`k7 config set api.ca`). `k7 install --api-hostname` (Let's Encrypt), `--api-tls-cert/--api-tls-key`, and `--api-insecure-http` change what is served; see [install](#install). To install without the API (CLI-only deployment, e.g. for local-only test rigs), pass `--no-api` to `k7 install`. Re-run `k7 install` later (without `--no-api`) to add it; the playbook is idempotent. ```bash k7 install --no-api # provision cluster without the API k7 install # re-run later to add it ``` The previous top-level commands `k7 start-api` / `k7 stop-api` / `k7 api-status` / `k7 get-api-endpoint` are hidden deprecated aliases that print a one-line warning and forward to the new home. ## API keys ```bash k7 generate-api-key NAME --role read-only|user|operator|admin [--tenant T] [-n NAMESPACE]... [--node NODE]... [--expires-days 365] [--api] k7 list-api-keys [--api] k7 revoke-api-key NAME [--api] ``` Tokens are `k7__`; only `sha256(secret)` is stored, in the `kube-system/k7-api-keys` Secret, and every API replica picks up a change within 5 seconds. Use with `X-API-Key` or `Authorization: Bearer`. - **--role** (required): `read-only` and `user` are **per tenant** (`--tenant` required in production mode; in development a `user` key without a tenant is an unscoped key with the pre-0.5 behaviour); `operator` (cluster-wide read, snapshot GC, node storage and dedication; no exec, no keys) and `admin` (everything, including keys, tenants, role bindings and remote install) are cluster-wide and refuse `--tenant`. - **-n / --namespace** (repeatable): narrows the key further, inside its tenant. A narrowed key cannot read or mutate another namespace and cannot run all-namespaces list/GC. - **--node** (repeatable): the key may only create/restore/fork onto these Kubernetes nodes (a single node auto-pins; several require `node_name`). Pair with `k7 nodes dedicate` for tenant isolation on k7d. - Key commands edit the Secret through kubectl, so they run on a server node (`sudo`); with `--api` they go through k7-api instead, using an `admin` key or `k7 login`. - `k7 list-api-keys` shows role, tenant and scopes, and warns on `unscoped_legacy` keys (pre-0.5 unscoped keys migrated as `admin`): re-issue them with a real role before switching to production mode. ## login / logout / whoami ```bash k7 login [--api URL] [--browser] [--scopes "openid profile email groups"] k7 whoami k7 logout ``` `k7 login` discovers the IdP from `GET /api/v1/auth/oidc`, runs the OIDC device flow (or Authorization Code + PKCE with `--browser`), and caches the tokens in `~/.config/k7/credentials.json`. Which role a human gets comes from the role bindings below. `k7 whoami` prints the resolved principal, role(s) and tenant. `k7 logout` deletes the cache and revokes the token at the IdP. The cluster must have been installed with the `--oidc-*` flags; otherwise `login` fails loudly. ## role-bindings ```bash k7 role-bindings get > bindings.yaml # current bindings + deny-subjects as YAML k7 role-bindings apply -f bindings.yaml # replace them all (admin only) ``` Maps OIDC groups, usernames or subjects to roles (per tenant for `read-only`/`user`, cluster-wide for `operator`/`admin`); lives in ConfigMap `k7-role-bindings` and is read on every request, so `deny-subjects` is the immediate stop for a compromised account. A write that would leave no `admin` binding while no unexpired `admin` key exists is refused. See [Tenants, roles and OIDC](/k7/guides/tenants-and-roles). ## tenant ```bash k7 tenant create acme -n acme-dev -n acme-prod \ --default-size cpu=1,memory=2Gi,disk=10Gi \ --quota sandboxes=20,vcpu=16,memory=32Gi,snapshotStorage=200Gi \ --max-size cpu=8,memory=32Gi --network egressBps=100M,ingressBps=100M k7 tenant update acme --quota sandboxes=40 --clear nodes # each flag replaces its whole section k7 tenant get acme | k7 tenant list [--json] # quotas and reconciled usage k7 tenant delete acme k7 tenant adopt --from-keys [--apply] # migrate pre-0.5 namespace-scoped keys onto tenants (dry run without --apply) ``` A `Tenant` (`tenants.k7.katakate.org/v1alpha1`, cluster-scoped) claims namespaces (created if missing; a namespace another tenant already claims is refused), dedicated nodes, a `defaultSize`, quotas and per-VM limits. Quotas are **reserved, not used** (a paused sandbox keeps its vCPU/memory/disk) and are checked under a per-tenant Lease, so concurrent creates on any number of API replicas cannot overshoot; over-quota is 403 `QuotaExceeded`. `--disk-io` limits exist in the CRD but no backend enforces them yet, so a tenant that sets them gets 400 `LimitNotEnforceable` on create instead of running unlimited. Full semantics: [Tenants, roles and OIDC](/k7/guides/tenants-and-roles). ## usage ```bash k7 usage [--tenant acme] [--from 24h] [--to now] [--group-by tenant,namespace,backend,state] [--json] ``` Showback from Prometheus: sandbox-, vCPU- and memory-seconds, egress bytes, disk and snapshot GiB·s, per tenant / namespace / backend / state. Needs a Prometheus that scrapes K7 (`--monitoring`, or `--metering-prometheus-url`); a tenant key may only ask about its own tenant. Also `GET /api/v1/usage`. See [Observability and metering](/k7/guides/observability-metering). ### Flag reference (create) - **-n, --namespace**: Kubernetes namespace (default `default`). - **-f, --file**: YAML config file (defaults to `k7.yaml` in the current directory when present). - **name** / **image** (positional): `k7 create NAME IMAGE`. There is no `--name` / `--image` flag. - **-b, --backend**: `kfd` / `kql` / `k7d` / `k7d-fc` (alias `k7-fc`, or the long names). - **--docker**: Run a pinned `dockerd` (overlay2 on a block graph disk). k7d / k7d-fc: guest agent service, forkable. Kata (kfd / kql): privileged `docker-vehicle` in the VM; kql persists the graph, kfd is ephemeral. - **--docker-disk**: Docker graph disk size for `--docker` (e.g. `20Gi`, `40Gi`). Default `20Gi` on all backends. - **--sidecar**: Deprecated. `--sidecar docker` is an alias of `--docker`. - **--cpu**: CPU limit (e.g., `1`, `500m`). - **--memory**: Memory limit (e.g., `1Gi`, `512Mi`). - **--storage**: Ephemeral storage limit (e.g., `2Gi`). - **--root-disk-size**: Longhorn root PVC size (`kql` only; default: tenant `defaultSize.disk`, else `10Gi`). - **--node**: pin to this Kubernetes node name (`k7 nodes list`). YAML: `node_name`. - **--egress-bps / --ingress-bps**: cap guest bandwidth in bits/s (`100M`). kfd/kql only; may only lower a tenant cap. YAML: `network_limits`. - **--env-file**: Path to env file on the host node injected as a Secret. - **--egress ENTRY**: Repeatable; whitelist CIDR or FQDN. Default (no flag): **block all**. - **--egress-open**: Allow all egress. Mutually exclusive with `--egress`. - **--ingress-port PORT**: Repeatable; TCP port to open for inbound traffic. Default (no flag): **deny all ingress**. - **--ingress-from SOURCE**: Repeatable; `sandbox:` | `namespace:` | `cidr:`. Default when `--ingress-port` is given: sandboxes in the same namespace. Reaching the sandbox from anywhere needs `cidr:0.0.0.0/0`, spelled out. Requires `--ingress-port`. - **--expose-port PORT**: Repeatable; publish a TCP port outside the cluster through a `NodePort` Service (`externalTrafficPolicy: Local`). Requires a matching `--ingress-port`. On a public node the node IP is public too. - **--before-script**: Shell script to run once after container start; runs with open egress before lockdown. - **--entrypoint**: Repeatable; override image ENTRYPOINT (ordered). - **--cmd**: Repeatable; override image CMD (ordered). - **--pod-non-root / --no-pod-non-root**: Pod-level non-root defaults. - **--container-non-root / --no-container-non-root**: Container runs as UID 65532, no privilege escalation. - **--cap-add CAP**: Repeatable; add back Linux capabilities (default drop ALL). - **--cap-drop CAP**: Repeatable; override default drop policy. Package installs like `apk add` require root inside the container. Either leave `container_non_root` disabled for setup or prebuild an image. See [Security & networking](/k7/api/security). For `kata-qemu-longhorn`, the persistence wrapper requires `/bin/sh`, `tar`, and `mount` in the image. Use a Debian/Ubuntu-like base (or ensure those tools are installed). For `kata-qemu-longhorn`, the main container runs privileged to allow bind mounts for persistence. For `kata-qemu-longhorn`, the wrapper preserves the image ENTRYPOINT+CMD by default. Use `--entrypoint` and/or `--cmd` to override. # Python SDK Source: https://docs.katakate.org/k7/guides/python-sdk > Use the k7-sdk package to manage sandboxes from Python The **`k7-sdk`** PyPI package (`import k7_sdk`) wraps the K7 HTTP API in two clients. The legacy **`katakate`** name still installs but emits a deprecation warning. - `Client` — synchronous, backed by `requests`. - `AsyncClient` — `asyncio`-native, backed by `httpx`. ## Install ```bash pip install k7-sdk==0.5.0 # sync client (requests only) pip install "k7-sdk[async]==0.5.0" # also installs httpx for AsyncClient ``` Run this on your **laptop / client**. A k7 node (fresh Ubuntu 24.04) does not ship `python3-venv`; `python3 -m venv …` fails until `sudo apt install -y python3-venv python3-pip`. The async client raises `RuntimeError` at construction time if `httpx` isn't installed. Always pass `verify_ssl` as the cluster CA path on the default HTTPS install (`./k7-ca.crt`, or `/etc/k7/tls/ca.crt` on the node). `k7 api status` prints a ready-to-paste snippet that includes it. ## Quickstart ```python from k7_sdk import Client k7 = Client(endpoint="https://node:31007", api_key="", verify_ssl="./k7-ca.crt") sb = k7.create({ "name": "demo", "image": "alpine:latest", "before_script": "apk add --no-cache curl", }) print(sb.exec("uname -a")) print(k7.list()) sb.delete() ``` Discover the endpoint with `k7 api endpoint` and the key with `k7 generate-api-key --role user --tenant ` (`--role` is required since 0.5.0; see [Tenants and roles](/k7/guides/tenants-and-roles)). Humans can use an OIDC token instead of a key: pass `token_provider=` a callable returning the bearer token (for example a reader for the token in `~/.config/k7/credentials.json` after `k7 login`, a GitHub Actions ID token, or a projected ServiceAccount token). Exactly one of `api_key` / `token_provider` must be given. On the default install the API presents a certificate signed by the cluster CA (`/etc/k7/tls/ca.crt` on the node) — copy it to the client and pass its path as `verify_ssl`. Clusters installed with `--api-hostname` (Let's Encrypt) work with the default `verify_ssl=True`. See the [CLI guide](/k7/guides/cli) for full details. ## `Client` reference | Method | Returns | Notes | |--------|---------|-------| | `Client(endpoint, api_key=None, verify_ssl=True, token_provider=None)` | — | Exactly one of `api_key` (sent as `X-API-Key`) or `token_provider` (a `Callable[[], str]` called per request, sent as `Authorization: Bearer`). `verify_ssl` is `True` (system trust store) or a **CA bundle path** (cluster CA). `verify_ssl=False` raises `ValueError` on an `https://` URL — there is no insecure mode; use `http://` only against a `--api-insecure-http` development install. | | `whoami()` | `dict` | The caller's identity: subject, role, tenant, namespaces, nodes ([`GET /api/v1/auth/whoami`](/k7/api/endpoints/identity)). | | `create(config: dict)` | `SandboxProxy` | `config` matches [`SandboxConfig`](/k7/api/endpoints/sandboxes#request-body-schema). | | `list(namespace=None)` | `list[dict]` | Omit `namespace` to list all. | | `delete(name, namespace="default")` | `dict` | | | `delete_all(namespace="default")` | `dict` | Returns `{message, results: [{name, success, error}]}`. | | `get_metrics(namespace=None)` | `list[dict]` | Per-sandbox `cpu_usage` / `memory_usage`. | | `install(nodes: list[str])` | `dict` | Remote node install through the k7-node-installer (`admin`, development mode, `--api-remote-install`). Returns the `202` body with the operation id. The pre-0.5.0 playbook-upload form is gone. | | `install_status(operation)` | `dict` | Per-node `pending` / `running` / `succeeded` / `failed` for an install operation. | | `pause(name, snapshot=None, namespace=...)` | `dict` | Scale to 0; optional VolumeSnapshot name | | `resume(name, namespace=...)` | `dict` | Scale back to 1 | | `fork(source, new_name, namespace=..., snapshot=None)` | `SandboxProxy` | kql: disk clone + cold boot; k7d / k7d-fc: warm CoW fork (memory + disk) | | `restore(snapshot, new_name, overrides=None, keep_snapshot=True)` | `SandboxProxy` | New sandbox from a snapshot | | `list_snapshots(...)` / `get_snapshot(...)` / `create_snapshot(...)` / `delete_snapshot(...)` / `gc_snapshots(...)` | varies | Snapshot lifecycle (see [Snapshots API](/k7/api/endpoints/snapshots)) | | `exec(name, command, namespace="default")` | `dict` | Run a command without a proxy | | `logs(name, tail=200, container="sandbox", since=0, namespace=...)` | `str` | Snapshot of the pod logs | | `nodes_storage()` | `dict` | Per-node kfd thin-pool + k7d disk-pool utilization (see [Nodes API](/k7/api/endpoints/nodes)). `operator` / `admin` only (`403` for tenant roles). | | `usage(start, end, tenant=None, group_by=None)` | `dict` | Showback over `[start, end]` (unix seconds) from the cluster's Prometheus; `group_by` is a comma list of `tenant,namespace,backend,state`. Tenant roles see their own tenant only. See [Observability and metering](/k7/guides/observability-metering). | | `create_api_key(name, *, role, expires_days, tenant=None, namespaces=None, nodes=None)` | `dict` | `admin`: mint a key; the response carries the one-time `token`. | | `list_api_keys()` / `revoke_api_key(name)` | `list[dict]` / `dict` | Key inventory (hashes never returned) and revocation. | | `dedicate_node(node, tenant)` / `undedicate_node(node)` | `dict` | Reserve a node for one tenant / release it (`admin`). | | `get_role_bindings()` / `put_role_bindings(bindings, deny_subjects)` | `dict` | Read / replace the OIDC role bindings ConfigMap (`admin`). | `SandboxProxy` (returned by `create`) exposes: - `exec(command: str) -> dict` — `{exit_code, stdout, stderr, duration_ms}` - `delete() -> dict` — shorthand for `client.delete(name, namespace)` - `pause` / `resume` / `fork` — same as the client methods, scoped to this sandbox - `snapshot(name)` — named VolumeSnapshot without pausing - `logs(tail=200, container="sandbox", since=0)` — log snapshot The proxy keeps a reference to the `Client`, so calling `sb.exec(...)` after the parent client closes will raise. ## All sandbox config fields The `create()` payload accepts every field from the API schema. Common ones: ```python sb = k7.create({ "name": "secure-sb", "image": "alpine:latest", "namespace": "default", # Backend selection "backend": "kata-qemu-longhorn", # or "kata-firecracker-devmapper" / "k7d" / "k7d-fc" "root_disk_size": "20Gi", # kata-qemu-longhorn only # Docker inside the sandbox (pinned dockerd, overlay2 on a block disk) "docker": True, "docker_disk": "40Gi", # default 20Gi # Non-root + minimal capabilities "pod_non_root": True, "container_non_root": True, "cap_add": ["CHOWN"], # FQDN-aware egress (Cilium); falls back to CIDR if all entries are CIDRs "egress_whitelist": ["api.openai.com", "*.huggingface.co"], # Ingress is deny-all unless you open ports; scope callers; optionally publish "ingress_ports": [8000], "ingress_from": ["sandbox:frontend"], # or "namespace:" / "cidr:" # "expose_ports": [8000], # NodePort; node_ports comes back on list() # Resource budget "limits": {"cpu": "1", "memory": "1Gi", "ephemeral-storage": "2Gi"}, # Setup commands run with open egress before lockdown "before_script": "pip install openai", # Override entrypoint/cmd "entrypoint": ["/usr/bin/python3"], "cmd": ["-c", "import time; time.sleep(3600)"], }) ``` See [Sandboxes API](/k7/api/endpoints/sandboxes#request-body-schema) for the full table. `env_file` points to a path **on the API node**, not the client. To inject runtime env vars from the client, build them into your `before_script` (`echo "KEY=value" > /etc/sandbox.env`) or commit them to your sandbox image. ## Wait until ready HTTP `POST /sandboxes` (and `Client.create` / `k7 create`) returns once Kubernetes objects exist — often while `ready` is still `False`. Poll `list()` before exec or curling a NodePort: ```python import time def wait_until_ready(client, name, namespace="default", timeout=120): deadline = time.time() + timeout while time.time() < deadline: for s in client.list(namespace=namespace): ready = s.get("ready") if s["name"] == name and s.get("status") == "Running" and ready in (True, "True"): return time.sleep(2) raise TimeoutError(f"{name} did not become Ready") wait_until_ready(k7, "secure-sb") ``` ## Pause, resume, fork `pause`, `resume`, and `fork` work on `Client` and on the `SandboxProxy` from `create`. Behavior depends on the backend: - **`kata-qemu-longhorn`** — fork clones the root disk via Longhorn `VolumeSnapshot` (the fork cold-boots, ~45 s); pause/resume scale the Deployment and the PVC survives. A `docker: True` sandbox's graph PVC is cloned too. - **`k7d`** / **`k7d-fc`** — fork is a warm CoW copy of the whole VM (memory + disk + processes — including a live `docker: True` engine, ~5 ms at the VMM, ~2 s end-to-end to a Ready pod); pause/resume freeze and thaw the live VM in place. Works for sandboxes on any node — the API forwards to the per-node `k7-agent` when needed. Forks inherit the source's egress and ingress policy, but not its `expose_ports` Service. ```python base = k7.create({ "name": "demo", "image": "python:3.12-slim", "backend": "kata-qemu-longhorn", "before_script": "pip install numpy pandas", }) wait_until_ready(k7, "demo") base.pause(snapshot="demo-v1") base.resume() forks = [base.fork(f"exp-{i}") for i in range(8)] for f in forks: print(f.exec("python -c 'import numpy; print(numpy.__version__)'")["stdout"]) f.delete() ``` See [Snapshots & fork](/k7/guides/snapshots-fork) for timing notes and CLI equivalents. ## Snapshots and restore ```python base.snapshot("demo-experiment-1") for s in k7.list_snapshots(sandbox="demo"): print(s["name"], s["kind"], s["age"]) k7.delete_snapshot("demo-experiment-1") k7.gc_snapshots(all_namespaces=True, keep_fork_for="10m", dry_run=True) base.pause(snapshot="demo-v1") base.delete() restored = k7.restore("demo-v1", "demo-restored") wait_until_ready(k7, "demo-restored") # restore returns once objects exist; boot is async print(restored.exec("ls /")["stdout"]) ``` Full HTTP detail: [Snapshots API](/k7/api/endpoints/snapshots). ## Async client ```python import asyncio from k7_sdk import AsyncClient async def main(): k7 = AsyncClient(endpoint="https://node:31007", api_key="", verify_ssl="./k7-ca.crt") try: await k7.create({"name": "async-sb", "image": "alpine:latest"}) for _ in range(60): sbs = await k7.list() if any(s["name"] == "async-sb" and s["ready"] == "True" for s in sbs): break await asyncio.sleep(2) out = await k7.exec("async-sb", "echo from async") print(out["stdout"]) finally: await k7.delete("async-sb") await k7.aclose() asyncio.run(main()) ``` `AsyncClient` mirrors the sync client surface (`create`, `list`, `delete`, `delete_all`, `exec`, `get_metrics`, `aclose`). Note that `AsyncClient.exec` is a top-level method (`await k7.exec(name, command)`) rather than a proxy method. ### Concurrent sandbox fan-out ```python import asyncio from k7_sdk import AsyncClient async def run_one(k7, idx): name = f"worker-{idx}" await k7.create({"name": name, "image": "python:3.12-slim"}) try: for _ in range(60): if any( s["name"] == name and s["ready"] == "True" for s in await k7.list() ): break await asyncio.sleep(2) result = await k7.exec(name, f"python -c 'print({idx} ** 2)'") return result["stdout"].strip() finally: await k7.delete(name) async def main(): k7 = AsyncClient(endpoint="https://node:31007", api_key="", verify_ssl="./k7-ca.crt") try: results = await asyncio.gather(*[run_one(k7, i) for i in range(10)]) print(results) finally: await k7.aclose() asyncio.run(main()) ``` ## Errors `requests.HTTPError` / `httpx.HTTPStatusError` are raised on non-2xx responses. The error body follows the API envelope: ```json { "error": { "code": "BadRequest", "message": "name already exists" } } ``` Recover the structured error: ```python import requests try: sb = k7.create({"name": "demo", "image": "alpine"}) except requests.HTTPError as e: body = e.response.json() print(body["error"]["code"], body["error"]["message"]) ``` ## Tips - Always pass `namespace=` if you use anything other than `default`. A **tenant** key (`k7 generate-api-key NAME --role user --tenant team-a`, optionally narrowed with `-n`) cannot list all namespaces or touch a namespace outside its tenant (`403`); only `operator` / `admin` keys see the whole cluster. - Rotate API keys with `k7 revoke-api-key ` and `k7 generate-api-key `. - The same operations are available via CLI (`k7 pause`, `k7 fork`, `k7 snapshot`, `k7 restore`) — default CLI path is the HTTP API; use `--core` only on the cluster node. # Multi-node clusters Source: https://docs.katakate.org/k7/guides/multi-node > Inventory format, HA, dual-backend nodes, Longhorn replication, the Helm chart for existing clusters, and remote node install Katakate scales horizontally via multi-node K3s clusters, configured with an Ansible-style inventory. This guide explains the inventory format, the HA story, and how to mix backends across nodes. ## Why multi-node? - **Horizontal scale** — more nodes, more sandboxes. - **Replicated storage** — Longhorn keeps multiple replicas of every volume across nodes (`numberOfReplicas` configurable). Data survives a node failure. - **Workload locality** — pods land on nodes whose backend matches the requested `--backend`. - **High availability of the control plane** — embedded etcd across 3+ K3s servers. ## Topology at a glance A typical 3-server + 1-agent cluster with mixed backends (including all-backend nodes running k7d) and Longhorn replication: ```mermaid flowchart TB Client["k7 CLI · SDK · curl
HTTPS · X-API-Key"] subgraph Cluster["K3s cluster (k7.katakate.org)"] direction TB subgraph CP["Embedded etcd"] direction LR E1[("etcd
node1")] E2[("etcd
node2")] E3[("etcd
node3")] E1 --- E2 --- E3 --- E1 CPNote["HA quorum
across servers"] end subgraph Nodes["Nodes"] direction LR subgraph N1["node1 — server"] direction TB N1Tag["backends: kql + kfd + k7d"] A1["k7-api
(Deployment)"] G1["k7-agent
(DaemonSet pod)"] S1["sandbox pods
kata / kata-qemu / k7"] K1["k7d daemon
/run/k7d/k7d.sock"] D1[("LVM thin-pool
/dev/nvme1n1")] P1[("k7d disk pool
/var/lib/k7d/disks")] L1[("Longhorn replica")] end subgraph N2["node2 — server"] direction TB N2Tag["backends: kql + k7d"] G2["k7-agent
(DaemonSet pod)"] S2["sandbox pods
kata-qemu / k7"] K2["k7d daemon"] P2[("k7d disk pool")] L2[("Longhorn replica")] end subgraph N3["node3 — server"] direction TB N3Tag["backend: kql"] G3["k7-agent
(DaemonSet pod)"] S3["sandbox pods
kata-qemu"] L3[("Longhorn replica")] end subgraph N4["node4 — agent"] direction TB N4Tag["backend: kfd"] G4["k7-agent
(DaemonSet pod)"] S4["sandbox pods
kata"] D4[("LVM thin-pool
/dev/nvme1n1")] end end N1 -. "etcd" .- CP N2 -. "etcd" .- CP N3 -. "etcd" .- CP A1 -. "k7d VM ops + storage stats
(agent token · CiliumNetworkPolicy)" .-> G2 L1 <-. "replicate
(longhorn_replicas=2)" .-> L2 L2 <-. "replicate" .-> L3 L1 <-. "replicate" .-> L3 end Client -- "NodePort 31007
(any server)" --> A1 classDef client fill:#0f1a2a,stroke:#3b82f6,color:#fff; classDef server fill:#1a1a1a,stroke:#888,color:#fff; classDef agent fill:#0b0b0b,stroke:#ef672b,color:#fff; classDef store fill:#0b1a0b,stroke:#22c55e,color:#fff; classDef etcd fill:#2a1a05,stroke:#ef672b,color:#fff; classDef k7d fill:#1a0b1a,stroke:#a855f7,color:#fff; class Client client; class N1,N2,N3,A1,S1,S2,S3,N1Tag,N2Tag,N3Tag,G1,G2,G3 server; class N4,S4,N4Tag,G4 agent; class D1,D4,L1,L2,L3,P1,P2 store; class CP,E1,E2,E3,CPNote etcd; class K1,K2 k7d; ``` What this shows: - **Any server** can serve the API NodePort — clients can hit any of `node1..3` on `:31007` over HTTPS (one cluster CA, `/etc/k7/tls/ca.crt` on the first master, covers every node IP). If you installed with `--api-allow-cidr`, the Service switches to `externalTrafficPolicy: Local`, so only the node actually running the `k7-api` pod answers — point clients at that node (or run a load balancer in front that health-checks `:31007`). - **etcd quorum** spans the three servers. One server can fail without losing the control plane. - **Backend labels** decide where a sandbox lands. `--backend kfd` is restricted to `node1` and `node4`; `--backend kql` to `node1`, `node2`, `node3`; `--backend k7d` (and `k7d-fc`, which shares the daemon and adds the `k7-fc` RuntimeClass) to `node1` and `node2`. - **k7d nodes** run the node-local `k7d` daemon (systemd unit, owns every `runtimeClassName: k7` and `k7-fc` microVM on that node) plus its XFS disk pool at `/var/lib/k7d/disks`. `k7d-fc` nodes additionally carry the pinned Firecracker + jailer binaries (`/usr/local/bin/firecracker`, `jailer`) and `/etc/k7d/shim-k7-fc.toml`. - **The `k7-agent` DaemonSet** runs on every node. The k7-api forwards k7d `pause` / `resume` / `fork` to the agent on the sandbox's node (shared-token auth, ingress restricted to the k7-api pod by a CiliumNetworkPolicy), so k7d VM ops work through the API **regardless of which node hosts the sandbox**. It also serves per-node storage-pool stats for `k7 nodes storage`. - **Longhorn replicas** of each `kata-qemu-longhorn` PVC are spread across `kql`-capable nodes (`longhorn_replicas=2` keeps two copies; `dataLocality: best-effort` keeps one local to the pod). - **Devmapper thin-pools and k7d disk pools** are per-node and not replicated — `kfd` sandboxes are ephemeral by design, and k7d fork/snapshot trees are host-local (a fork lands on the source's node; cross-node fork is on the k7d roadmap). ## Inventory format `k7 install -i inventory.ini` reads a standard Ansible INI inventory. Each host declares which backend(s) it supports via `k7_backends` (comma-separated). A complete example ([`src/k7/deploy/inventory.ini.example`](https://github.com/Katakate/k7/blob/main/src/k7/deploy/inventory.ini.example)): ```ini [k7_servers] node1 ansible_host=203.0.113.10 k7_backends=kata-qemu-longhorn,kata-firecracker-devmapper,k7d k7_disk_source=device:/dev/nvme1n1 node2 ansible_host=203.0.113.11 k7_backends=kata-qemu-longhorn,k7d node3 ansible_host=203.0.113.12 k7_backends=kata-qemu-longhorn [k7_agents] node4 ansible_host=203.0.113.13 k7_backends=kata-firecracker-devmapper k7_disk_source=device:/dev/nvme1n1 [k7_cluster:children] k7_servers k7_agents [k7_cluster:vars] ansible_user=root ansible_ssh_private_key_file=~/.ssh/id_ed25519 longhorn_replicas=2 ``` For a 3-node HA cluster with **all backends on every node** (`kfd`, `kql`, `k7d`, `k7d-fc`), one command from the first master brings everything up. Install the 0.5.0 CLI from the PPA or `.deb` on that master and run `k7 install` from any directory — the signed `k7-api` image is pulled from GHCR by digest, so no checkout and no Docker are needed. Fresh nodes have no SSH host keys yet — either `ssh-keyscan` the peers into `known_hosts` or set `ansible_ssh_common_args` as below: ```ini [k7_servers] k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc k7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d,k7d-fc [k7_cluster:children] k7_servers [k7_cluster:vars] ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_ed25519 ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null longhorn_replicas=3 ``` ```bash k7 install -i inventory.ini --ha --api-mode development --k7d-version 0.7.0 ``` `--api-mode` is required (`production` or `development`; see [Production and development modes](/k7/concepts/security-model#production-and-development-modes)). `--k7d-version 0.7.0` is the playbook default and downloads the public GitHub release; another release needs its checksum (`k7d_sha256=`), and a tarball you built from k7d source goes through `--k7d-artifact /root/k7d-v0.9.0-x86_64-linux.tar.gz`. k7 0.4.0+ copies Firecracker **v1.16.2** for `k7d-fc`. Two-node (one server + one agent, **no `--ha`** — embedded etcd needs 3+ servers). After each box has Ubuntu on one NVMe and a raw spare (`TWO_DISK`): ```ini [k7_servers] k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc [k7_agents] k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc [k7_cluster:children] k7_servers k7_agents [k7_cluster:vars] ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_ed25519 ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null longhorn_replicas=2 ``` ```bash k7 install -i inventory.ini --api-mode development --k7d-version 0.7.0 ``` Servers that must not run sandboxes (typical masters in a cluster that has agents) declare `k7_backends=none` — omitting `k7_backends` is an error, not an implicit empty set. Tenant isolation comes later with `k7 nodes dedicate NAME --tenant ID` (see [Tenants and roles](/k7/guides/tenants-and-roles)); do not dedicate the first master. Don't pass `--backend` alongside `-i`: the per-host `k7_backends` in the inventory is authoritative (an explicit `--backend` overrides it). On dual-NVMe boxes where the OS lives on one disk and the other is a raw spare, **omit `k7_disk_source`** (the default `spare-disk`): NVMe enumeration (`nvme0n1` vs `nvme1n1`) is not stable across reboots, so a hardcoded device can point at the OS disk after a reboot. The playbook auto-detects the empty non-root whole disk. Set it only when a node has several spare disks and you must pick a specific one. ### Per-host variables | Variable | Required | Description | |----------|----------|-------------| | `ansible_host` | yes | SSH target IP or hostname | | `ansible_user` | yes | SSH user (commonly `root` on bare-metal) | | `k7_backends` | yes | Comma-separated list of backends supported by this node. Values: `kata-qemu-longhorn` (`kql`), `kata-firecracker-devmapper` (`kfd`), `k7d`, `k7d-fc` (`k7-fc`) | | `k7_disk_source` | optional | Where the kfd thin-pool and k7d disk image live: `spare-disk` (default — auto-detect the empty non-root whole disk), `device:` (e.g. `device:/dev/nvme1n1`; wipe with `utils/wipe-disk.sh` first), or `loopback` (XFS image only — no kfd thin-pool, so `kfd` is rejected). The legacy `k7_devmapper_disk=` still means `device:`. On dual-NVMe (`TWO_DISK`) boxes leave it unset so a swapped `nvme0n1`/`nvme1n1` cannot point at the OS disk. | | `longhorn_extra_disk` | optional, `kata-qemu-longhorn` only | Existing **mount path** (e.g. `/mnt/longhorn-extra`) to register as an extra Longhorn disk. **Not** a `/dev/...` device — format and mount it yourself first. Without this, Longhorn uses `longhorn_data_path` (default `/var/lib/longhorn`) on the root filesystem. | | `kata_thinpool_pv_size` | optional, `kata-firecracker-devmapper` only | LVM PV size for the kfd thin-pool (default `100G`) | | `k7d_disks_image_size` | optional, `k7d` only | Sparse XFS image size for the k7d volume pool at `/var/lib/k7d/disks` (default `32G`) | ### Group variables (`[k7_cluster:vars]`) | Variable | Default | Description | |----------|---------|-------------| | `ansible_user` | `root` | SSH user for all hosts | | `ansible_ssh_private_key_file` | `~/.ssh/id_rsa` | Private key used for SSH | | `longhorn_replicas` | `min(3, node count)` | Longhorn replica count | | `longhorn_data_path` | `/var/lib/longhorn` | Longhorn data directory | | `k3s_version` | `v1.36.5+k3s1` (playbook pin) | K3s to install on fresh nodes; `k7 install --k3s-version` overrides. k7 never upgrades a running K3s. | | `k7_dns_upstreams` | the host's resolvers | CoreDNS upstream resolvers, 1–3 IPv4 comma-separated (e.g. `1.1.1.1,9.9.9.9`); same as `k7 install --dns-upstream`. See [CoreDNS upstreams](/k7/concepts/networking#coredns-upstreams). | | `k7_distro` | set by `k7 install` (`k3s`) | Selects the node-prep adapter (`roles/k7_node_prep/vars/.yml`); the Helm chart's installer uses `rke2`, `kubeadm` or `eks-al2023`. | ## Roles: servers vs agents - **`[k7_servers]`** — K3s control-plane nodes. The **first host listed becomes the cluster-init server**; subsequent servers join it (HA via embedded etcd). - **`[k7_agents]`** — K3s worker nodes. They join the first server with the K3s join token. Servers can also schedule sandboxes (they aren't tainted). ### Single-master vs HA - **1 server**: the simplest setup; no HA. Failure of the server takes the cluster down. - **2 servers**: **avoid** — embedded etcd has no fault tolerance with even counts. Better: 1 server + 1 agent. - **3+ servers** (odd numbers): full HA. Quorum tolerates `(N-1)/2` server failures. ## Mixed backends per node A node can advertise both backends in `k7_backends`: ```ini node1 ansible_host=... k7_backends=kata-qemu-longhorn,kata-firecracker-devmapper k7_disk_source=device:/dev/nvme1n1 ``` The playbook installs both Kata runtime classes (`kata` and `kata-qemu`), merges containerd configs, and labels the node with **both**: ``` k7.katakate.org/backend-kata-qemu-longhorn=true k7.katakate.org/backend-kata-firecracker-devmapper=true ``` When you `k7 create --backend kql ...`, the scheduler uses `nodeSelector: { k7.katakate.org/backend-kata-qemu-longhorn: "true" }` to land the pod on a compatible node. If no compatible node exists, k7 detects the `FailedScheduling` event and returns a clear error listing which backends are actually available. ## Longhorn topology and replication Longhorn is configured to be topology-aware: - **`volumeBindingMode: WaitForFirstConsumer`** — the PVC binds when the pod is scheduled, ensuring the volume is created on the node where the pod will run. - **`dataLocality: best-effort`** — Longhorn keeps one replica on the same node as the pod (reduces network reads). - **`replica-auto-balance: best-effort`** — replicas auto-spread across nodes. Practical effect: writing to `/mnt/state` from inside the sandbox stays on the local node when possible, and the data has redundant replicas elsewhere in the cluster. If the pod's node fails, the Deployment reschedules onto another node and Longhorn re-attaches the volume from a surviving replica. ## Joining nodes incrementally You don't have to install everything in one shot. To add an agent to a running cluster: ```bash # On the new node, after `apt install k7`: k7 install \ --role agent \ --join https://master.example.com:6443 \ --join-token "$(ssh master 'sudo cat /var/lib/rancher/k3s/server/node-token')" \ --backend kql ``` To add a new server (HA): ```bash k7 install \ --role server \ --join https://master.example.com:6443 \ --join-token "..." \ --ha ``` For a fresh **3-server HA cluster**, prefer the inventory-based install — it handles token distribution and join ordering automatically. ## CNI choice across the cluster Cilium (the default) is cluster-wide and applies its eBPF datapath uniformly across nodes. FQDN egress works regardless of which node the sandbox lands on. If you pass `--cni flannel`, every node uses Flannel and FQDN egress is unavailable cluster-wide. ## Verification After `k7 install -i inventory.ini`: ```bash # Cluster nodes sudo k3s kubectl get nodes -o wide # Per-backend labels sudo k3s kubectl get nodes --show-labels | grep katakate # Cilium health sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium # Longhorn sudo k3s kubectl -n longhorn-system get nodes,replicas # k7-api pod + per-node k7-agent DaemonSet sudo k3s kubectl -n kube-system get deploy/k7-api ds/k7-agent -o wide k7 api status # Per-node storage-pool utilization (thin-pool + k7d disk pool, via the agents) k7 nodes storage ``` A clean cluster will show all nodes `Ready`, Cilium pods `Running`, the k7-api Deployment with `2/2 ready` (two replicas behind a PodDisruptionBudget since 0.5.0), and Longhorn replicas distributed across nodes. `k7 nodes list` shows every node with its backends and tenant dedication. ## Helm chart for existing clusters Since 0.5.0, `deploy/helm/k7` installs K7 on a Kubernetes cluster you already run — validated on **RKE2 v1.36.5 + Cilium 1.19.2**, with `kubeadm` and `eks-al2023` node-prep adapters. The chart deploys k7-api, k7-agent, the k7-node-installer DaemonSet, the RuntimeClasses, snapshot GC and the Cilium policies; it does **not** install the CNI, Longhorn, the CSI snapshot controller or OpenEBS LVM LocalPV (kfd `--docker`) — those are the cluster's own. `values.schema.json` rejects unknown keys, and several values are **required with no default**: | Value | Choices | Notes | |-------|---------|-------| | `networkPolicy.provider` | `cilium` / `kubernetes` / `none` | `cilium` gives FQDN + CIDR egress and the sandbox→platform deny (pre-install hook requires the DNS proxy with `dnsRejectResponseCode=nameError`). `kubernetes` is CIDR-only (domain egress → 400) and needs `acknowledgeNoPlatformDeny: true`; `none` also needs `acknowledgeNoEgressPolicy: true`. | | `tls.mode` | `certManager` / `secret` / `insecureHTTP` | `insecureHTTP` is refused with `api.mode=production`. | | `api.mode` | `production` / `development` | Same semantics as `k7 install --api-mode`. | | `nodeInstaller.distro` | `k3s` / `rke2` / `kubeadm` / `eks-al2023` | Node-prep adapter for nodes labelled `k7.katakate.org/node-pool=true`. | | `nodeInstaller.diskSource` | `spare-disk` / `loopback` / `device:/dev/` | Same meaning as the inventory's `k7_disk_source`. | | `images.*.digest` | `sha256:<64 hex>` | The release pipeline fills them (`images.lock.yaml`); the chart is not published to an OCI registry yet — install from the checkout. | | `backends.*.enabled` | at least one | `kql` also needs `storageClass` and `volumeSnapshotClass`. | | `observability.scraperSelector` | namespace + pod labels | Required while `observability.metrics` is on (default); only those pods may reach port 9464. | | `metering.sampleInterval` | 5–60 s | Required with `api.mode=production`. | Other notable values: `api.replicas` (2), `api.service.type` (`ClusterIP` / `NodePort` / `LoadBalancer`), `api.remoteInstall` (development mode only), `nodeInstaller.allowServerRestart` (the installer restarts containerd, which restarts the apiserver on server nodes — refused unless `true`), `nodeInstaller.wipeDisks` / `forceCleanup` for uninstall, `observability.serviceMonitor` / `prometheusRule` / `grafanaDashboards`, `auth.oidc.*` and `metering.*` (same documents as the `k7 install` flags). The release namespace runs the privileged installer, so label it `pod-security.kubernetes.io/enforce=privileged`. Hooks refuse a cluster that cannot enforce the values, create the agent-token and API-key Secrets, and undo node prep on uninstall; `helm test` creates, execs and forks one sandbox per backend. Pass helm a generous `--timeout` (e.g. `35m`) — node prep is bounded by `hookTimeoutSeconds`. ```bash kubectl label ns k7-system pod-security.kubernetes.io/enforce=privileged kubectl label node worker-1 k7.katakate.org/node-pool=true helm install k7 deploy/helm/k7 -n k7-system --create-namespace --timeout 35m \ --set networkPolicy.provider=cilium --set tls.mode=certManager \ --set tls.certManager.issuerRef.name=letsencrypt --set 'tls.certManager.dnsNames={k7.example.com}' \ --set api.mode=production --set metering.sampleInterval=15 \ --set nodeInstaller.distro=rke2 --set nodeInstaller.diskSource=spare-disk \ --set backends.kfd.enabled=true --set backends.k7d.enabled=true \ --set observability.scraperSelector.namespace=monitoring \ --set 'observability.scraperSelector.podLabels.app\.kubernetes\.io/name=prometheus' helm test k7 -n k7-system --timeout 35m ``` The full RKE2 walk-through (host packages, RKE2 without kube-proxy, Cilium, Longhorn, OpenEBS, TLS, chart, upgrade, uninstall) is [`deploy/helm/k7/ci/rke2-hetzner.md`](https://github.com/Katakate/k7/blob/main/deploy/helm/k7/ci/rke2-hetzner.md); the values file it uses is `ci/rke2-cilium-values.yaml`. ## Remote node install through the API `POST /api/v1/install` lets a cluster-wide `admin` add nodes without SSH from the controller. It is **off by default (404)** and **development mode only** — `k7 install --api-mode production --api-remote-install` (and the chart with `api.mode=production` + `api.remoteInstall=true`) refuse the combination before deploying. ```bash k7 install --backend kfd,k7d --api-mode development --api-remote-install \ [--node-installer-image local|ghcr.io/katakate/k7-node-installer@sha256:…] \ [--node-installer-allow-server-restart] ``` The API never runs Ansible (its image has none): the flag deploys the **k7-node-installer** DaemonSet in `k7-system`, configured from the first master's distro, backends and disk settings, with its image pinned at deploy time (default from `images.lock.yaml`). A request is exactly `{"nodes": ["worker-3"]}` — any other key (`playbook`, `image`, `inventory`, `config`) is a 400, an unknown node 404, a K3s server node without `--node-installer-allow-server-restart` 409, a drifted DaemonSet image 500. The API labels the nodes `k7.katakate.org/node-pool=true` and returns **202 with an operation id**; `GET /api/v1/install/{id}` reports each node `pending` / `running` / `succeeded` / `failed`. In the SDK: `Client.install(nodes)` and `Client.install_status(id)` (the pre-0.5.0 playbook-upload form is gone). Running `k7 install` again **without** `--api-remote-install` converges: pool labels, installer DaemonSet, RBAC, admission policies and the operation records are removed (unless Helm owns them). Endpoint details: [Tenants, nodes and install](/k7/api/endpoints/tenants#remote-install). ## Reference - Example inventory: [`src/k7/deploy/inventory.ini.example`](https://github.com/Katakate/k7/blob/main/src/k7/deploy/inventory.ini.example) - Ansible playbook: [`src/k7/deploy/k7-install-node.yaml`](https://github.com/Katakate/k7/blob/main/src/k7/deploy/k7-install-node.yaml) # Snapshots & fork Source: https://docs.katakate.org/k7/guides/snapshots-fork > Pause, resume, and fork sandboxes via crash-consistent VolumeSnapshots The `kata-qemu-longhorn` backend supports **disk-level snapshots** of running sandboxes via Longhorn's CSI VolumeSnapshot API. This unlocks three workflows: - **Pause** — scale to 0 and (optionally) snapshot the disk for safekeeping. - **Resume** — scale back to 1; the same disk re-attaches. - **Fork** — clone a sandbox's disk into a brand-new sandbox with its own pod, IP, and lifecycle. Disk-level `k7 snapshot` requires the `kata-qemu-longhorn` backend. The `kata-firecracker-devmapper` backend uses ephemeral devmapper LVs and doesn't support persistent snapshots or fork. On `k7d` and `k7d-fc`, `k7 pause` / `k7 resume` / `k7 fork` are VM-level (memory + disk, no VolumeSnapshot) and `k7 snapshot` is rejected loudly. **Looking for memory-faithful or whole-cluster forking?** That's the [k7d runtime](/k7d/index), not k7's Longhorn-based fork. k7d warm-forks a running VM in ~5 ms (memory, disk, processes, and network identity all survive) and can fork an entire live Kubernetes cluster in ~100 ms — see [CoW fork](/k7d/concepts/cow-fork) and [Cluster mode](/k7d/guides/cluster-mode). The fork described on this page is **disk-only**: the clone boots fresh from a copy of the source's disk. ## Anatomy of a `kata-qemu-longhorn` sandbox When you create a `kata-qemu-longhorn` sandbox, k7 provisions: - A **Longhorn PVC** named `-root-lh`, sized via `--root-disk-size` (default `10Gi`). - A **Deployment** with one replica, mounting the PVC at `/mnt/state` (with persistence wrapper bind-mounts). - An ingress `NetworkPolicy` (deny-all unless `ingress_ports` is set) and an egress policy (CIDR or FQDN). - An optional `Secret` from `env_file`. - With `--docker`, a second Longhorn PVC `-docker-lh` holding Docker's `overlay2` graph (default `20Gi`, `--docker-disk`). It is snapshotted, cloned and re-attached alongside the root PVC on pause/fork/restore. State written to `/mnt/state` (and to the rootfs via the bind-mount overlay) survives pod restarts because it lives on the PVC. ## Pause ```bash k7 pause demo # scale to 0, no snapshot k7 pause demo --snapshot # auto-named -paused- k7 pause demo --snapshot=demo-snap-1 # take a VolumeSnapshot, then scale to 0 ``` `--snapshot` is the only pause flag besides `-n`. Bare `--snapshot` auto-names; `--snapshot=NAME` uses your name. There is no `--pvc` or `--snapshot-class` on the CLI (those were removed). Restore is `k7 restore SNAPSHOT_NAME NEW_SANDBOX_NAME` — there is no `--latest`. The snapshot is **crash-consistent**: it captures the disk state at the moment of the snapshot, not the in-memory process state. The sandbox process is then terminated as the Deployment scales to 0. Earlier releases could hand a kql fork a half-cloned disk on a replicated cluster (the child failed `fsck` and never went Ready): the source VolumeSnapshot was deleted while Longhorn was still copying it. Since 0.5.0 the snapshot is only removed once the Longhorn clone reports `completed`, so a fork of a busy volume comes up clean. The clone is still crash-consistent — pause the sandbox first when you need a quiesced copy, and prefer **`k7d` warm fork** when latency matters. The snapshot is stored as a `VolumeSnapshot` object in the same namespace and persists until you delete it (`kubectl delete volumesnapshot `). ## Resume ```bash k7 resume demo ``` Scales the Deployment back to 1. The pod re-attaches to the same PVC; data on `/mnt/state` is exactly as it was at pause time. If the pause took a snapshot, the snapshot remains available — useful for rollback if the resumed workload corrupts something. **Wait until Ready before `k7 exec`.** On a 3-node HA cluster Longhorn attach can leave the pod `Pending` for tens of seconds after `k7 resume` returns. Poll `k7 list` until `Ready` is `True` (the same wait you need after `Client.resume()` / `k7 restore`). `k7 snapshot list` may show `READY False` for a few seconds after `k7 snapshot create`; wait for `readyToUse: true` before `k7 restore`. ## Fork ```bash k7 fork demo demo-clone k7 fork demo demo-clone --snapshot custom-name ``` Fork takes a snapshot of the source sandbox's PVC, **clones it into a new PVC**, and creates a new Deployment pointing at the cloned PVC. Implementation steps: 1. Create a `VolumeSnapshot` of `-root-lh` (auto-named or `--snapshot`). 2. Wait for `readyToUse: true` on the snapshot. 3. Create a new PVC `-root-lh` from the snapshot via `dataSource`. 4. Create a Deployment for the new sandbox using the cloned PVC. 5. Wait for the new pod to be Ready. 6. Delete the temporary snapshot (the cloned PVC is independent of it). **Scope: disk only.** Fork does not clone: - Memory state (the new sandbox boots fresh) - CPU registers / process state - Network identity (the fork gets a new pod IP) **What a fork inherits on every backend:** the source's egress whitelist *and* its `ingress_ports` / `ingress_from` policy, so a child is never more reachable than its parent (an earlier release left forks fully ingress-closed). It does **not** inherit an `expose_ports` NodePort Service — expose the child explicitly if you need it. A `--docker` source forks with its graph disk (`kql`: cloned PVC; `k7d` / `k7d-fc`: reflinked scratch disk, dockerd keeps running with `overlay2`); `kfd` rejects fork up front with a direct "kfd cannot fork" error (kfd has no Longhorn PVC). With k7d 0.9.0+ a `k7d` / `k7d-fc` fork also keeps a working NIC (own bridge, NAT), so the child has egress like its parent; older k7d forks had none. In production mode a fork counts against the tenant's `sandboxes` / `vcpu` / `memory` quota like a create (`403 QuotaExceeded`). ## Observed performance Single-node medians (Hetzner AX41, Longhorn `replicas=1`, 2026-08-10) — see [PERFORMANCE.md](https://github.com/Katakate/k7/blob/main/PERFORMANCE.md): | Operation | kql | k7d | |-----------|-----|-----| | Cold create → pod ready | **17.1 s** | **2.1 s** | | Named snapshot ready | 6.5 s | n/a (k7d rejects `k7 snapshot`) | | Pause | 1.3 s (scale to 0) | **0.2 s** (VM frozen in place) | | Resume → exec | 4.1 s | **0.3 s** | | **Fork → usable** | **46.7 s** (disk clone + cold boot) | **2.4 s** (warm memory CoW) | On a two-node r=2 cluster (2026-08-18 Show HN cut) kql fork is ~83 s; k7d fork stays ~4 s and is the only backend whose fork inherits live memory (`/tmp` markers, running processes). `k7d-fc` (k7d driving stock Firecracker) sits close to `k7d`: create → Ready 2.1 s and fork → exec ~2.45 s on the same node (2026-09-10). Forking a warm `--docker` engine takes ~6.8 s (`k7d`) / ~9.1 s (`k7d-fc`) to Ready with `overlay2` still active in the child — see [Docker in a sandbox](/k7/guides/docker). ## Example: a parallel exploration with fork A common AI workflow is to set up a heavy environment once, then explore many branches from it. ```python from k7_sdk import Client k7 = Client(endpoint="https://node:31007", api_key="...", verify_ssl="./k7-ca.crt") # Step 1: prepare a base sandbox with a big environment base = k7.create({ "name": "base", "image": "python:3.12-bookworm", "backend": "kata-qemu-longhorn", "root_disk_size": "20Gi", "before_script": ( "apt-get update && " "apt-get install -y build-essential && " "pip install torch transformers datasets" ), "egress_whitelist": ["pypi.org", "files.pythonhosted.org", "huggingface.co"], }) # (Wait until base is Ready; do whatever setup you want) # Step 2: fork the base for each parallel experiment forks = [base.fork(f"exp-{i}") for i in range(8)] for f in forks: f.exec("python -c 'print(\"branch\", __import__(\"os\").getpid())'") ``` Or from the CLI on a node with API access: ```bash for i in $(seq 0 7); do k7 fork base exp-$i & done wait ``` Each `exp-N` boots from a clone of `base`'s disk — the heavy `pip install` doesn't run again. ## Cleanup `k7 delete demo` removes the Deployment, the PVCs (root and, with `--docker`, the graph disk), and any pause/fork snapshots associated with the sandbox. `k7 snapshot gc` reaps unreferenced snapshots and now also deletes orphan `VolumeSnapshotContent` objects and clears the PVC finalizer that used to leave Longhorn volumes stuck. To inspect what's still around: ```bash sudo k3s kubectl get pvc,volumesnapshot -A | grep demo sudo k3s kubectl get volumes.longhorn.io,replicas.longhorn.io -n longhorn-system ``` Integration tests under `tests/integration/` exercise pause/resume/fork end-to-end and detect orphaned Longhorn replicas or stuck PVCs. ## Reference - Performance baseline: [`PERFORMANCE.md`](https://github.com/Katakate/k7/blob/main/PERFORMANCE.md) - Integration tests: [`tests/integration/`](https://github.com/Katakate/k7/tree/main/tests/integration) # Docker in a sandbox Source: https://docs.katakate.org/k7/guides/docker > k7 create --docker — a pinned dockerd with overlay2 on a block disk, on every backend Some workloads need more than a shell — they need `docker build`, `docker run`, or `docker compose` inside the sandbox. `k7 create --docker` gives every backend the same contract: - `docker`, `docker compose`, and `docker buildx` on `PATH` (pinned Docker **27.5.1**, Compose 2.32.4, Buildx 0.20.0) — your image does **not** need the Docker CLI. - A pinned `dockerd` with `overlay2` on a **block** graph disk (default **20Gi**, `--docker-disk 40Gi` to change). Never vfs, never a graph on virtio-fs or emptyDir. - `DOCKER_HOST` pre-set in the sandbox container; the socket is shared as a directory mount. - Everything runs **inside the sandbox VM** — the isolation boundary is the VM, not the container. ```bash k7 create --docker --backend k7d --egress-open builder ubuntu:24.04 k7 exec builder -- docker info | grep 'Storage Driver' # overlay2 k7 exec builder -- docker pull alpine:3.21 # Interactive `k7 shell` is node-local; from a laptop keep using exec. ``` Or in YAML / the API: ```yaml name: builder image: ubuntu:24.04 backend: k7d # kfd / kql / k7d / k7d-fc docker: true docker_disk: 40Gi # optional, default 20Gi ``` `--sidecar docker` (and `sidecar: docker`) is a **deprecated alias** of `--docker`. It still resolves, but the old dind sidecar with a tmpfs / subPath graph is gone: the alias now provisions the same block-backed overlay2 engine described here. ## How it runs on each backend The user-visible contract is identical; what differs is where `dockerd` lives and what happens to the graph disk across pause, fork, and restore. ```mermaid flowchart LR subgraph K7D["k7d / k7d-fc — one VM, one container"] direction TB Agent["k7-agent (guest PID 1)
supervises pinned dockerd"] Scratch[("virtio-blk graph disk
overlay2, reflinked on fork")] User1["sandbox container
docker CLI · your code"] Agent --- Scratch User1 -- "/run/k7d/docker/docker.sock" --> Agent end subgraph KATA["kfd / kql — one Kata VM, two containers"] direction TB Vehicle["docker-vehicle (privileged)
docker:27.5.1-dind → dockerd"] Block[("block device /dev/k7docker
kfd: ephemeral LVM LV
kql: Longhorn Block PVC")] User2["sandbox container (unchanged securityContext)
docker CLI · your code"] Vehicle --- Block User2 -- "/run/k7/docker/docker.sock" --> Vehicle end classDef vm fill:#0b0b0b,stroke:#ef672b,stroke-width:1px,color:#fff; classDef cnt fill:#1a1a1a,stroke:#888,color:#fff; classDef disk fill:#0b1a0b,stroke:#22c55e,color:#fff; class K7D,KATA vm; class Agent,Vehicle,User1,User2 cnt; class Scratch,Block disk; ``` | Backend | Where `dockerd` runs | Graph disk | Pause / resume | `k7 fork` | `k7 restore` | |---|---|---|---|---|---| | `k7d` | **Guest agent service** — no extra CRI container, the sandbox container is not privileged. Pod is stamped `k7d.katakate.org/docker=true` | Per-sandbox virtio-blk scratch image (`k7d.katakate.org/docker-disk`), deleted with the VM | ✅ VM frozen in place; graph intact | ✅ **Warm CoW fork of the live engine** — the child keeps `dockerd`, its images, and running inner containers (graph disk is reflinked) | ❌ (k7d has no named snapshots) | | `k7d-fc` | Same guest service, VM under the Firecracker jailer | Same | ✅ graph intact; resume→exec needs Firecracker **v1.16.2+** (see [k7d-fc](/k7/concepts/backends#k7d-fc)) | ✅ Same, child stays overlay2 | ❌ | | `kql` | Privileged **`docker-vehicle`** container in the same Kata VM; the sandbox container's security context is unchanged | Second Longhorn **Block** PVC `-docker-lh` (ext4, overlay2) | ✅ Graph persists on the PVC; `--snapshot` snapshots both volumes | ✅ Disk clone of root **and** graph PVC, then cold boot | ✅ Both PVCs restored | | `kfd` | Privileged `docker-vehicle` in the same Kata VM | Generic ephemeral volume from StorageClass `k7-docker-lvm` (OpenEBS LVM LocalPV over `kata-vg/thin-pool`), deleted with the pod | Scale to 0 / 1 — graph lost | ❌ Rejected (`k7 fork of a kfd --docker sandbox is not supported`) | ❌ | On kql the two VolumeSnapshots (root + docker graph) are crash-consistent **per volume**, not atomic across both. k7 syncs the vehicle before snapshotting, but an inner container mid-write can still land on either side of the cut. ### k7d / k7d-fc: dockerd as a guest service The k7d guest agent (PID 1) starts a pinned `dockerd` from a read-only, host-enforced payload image and puts its graph on a dedicated scratch disk. Inner containers share the guest kernel and network namespace with the sandbox container, and the sandbox paths `/home`, `/root`, `/tmp`, `/opt`, and `/workspace` are exported into the engine so `docker run -v "$PWD:/src"` from your working directory works as expected. Because `dockerd` is not a Kubernetes sidecar, `k7 fork` clones it like any other in-VM process: a warm fork of a `--docker` sandbox comes up with `Storage Driver: overlay2` and the source's images and containers already present. This is the only backend where the Docker engine itself is forkable. Requires k7d **0.6.0+** (the release tarball ships the guest docker payload; `k7 install --backend k7d` installs it to `/usr/local/share/k7d/docker`). An older k7d fails create loudly: `this k7d has no docker service; upgrade`. Inside the guest, `docker.sock` is guest root — the sandbox `securityContext` no longer bounds what runs in the VM. For mutually distrusting tenants, use **`k7d-fc`**: identical guest side, plus a per-VM jailer around the VMM. See the [k7d security notes on `--docker`](/k7d/concepts/security#docker-as-a-guest-service). ### kfd / kql: the docker-vehicle On the Kata backends k7 injects one privileged `docker-vehicle` container (`docker:27.5.1-dind`, digest-pinned) into the pod. It formats and mounts the block device `/dev/k7docker`, then runs `dockerd --storage-driver=overlay2` on a directory socket at `/run/k7/docker/docker.sock`. An init step copies the pinned CLI and plugins into the sandbox container. Within one Kata VM the vehicle and the sandbox are a single trust domain; the isolation boundary is the VM. The sandbox waits for the vehicle's readiness probe (`docker info` reporting `overlay2`) before `before_script` runs. ## Measured Medians of 3 on one Hetzner AX41 node (Ryzen 5 3600, 64 GiB, NVMe), 3Gi / 4 CPU guests, `DOCKER_BUILDKIT=0`. Full tables and ranges: [PERFORMANCE.md](https://github.com/Katakate/k7/blob/main/PERFORMANCE.md). | Operation | `k7d` | `k7d-fc` | `kql` (overlay2 on Longhorn block) | |---|---|---|---| | `docker pull debian:12-slim` | 11.3 s | 8.16 s | — | | `docker build` (no cache) | 52.0 s | 51.6 s | — | | `docker build` (cached) | 469 ms | 470 ms | — | | run io (2k small files + 512 MB fsync) | 1.18 s | 1.28 s | 4.24 s | | fork of a warm engine → Ready + overlay2 child | 6.79 s | 9.13 s | n/a | The `run io` row is the point of the block graph: the old dind sidecar on a tmpfs / subPath graph fell back to `vfs` and took ~37 s on the same workload. ## Network considerations Inner containers share the sandbox's network namespace. **Traffic spawned by `dockerd`** (image pulls, containers dialing out) flows through the VM's network stack and is subject to the sandbox's egress policy. `k7 create` without an egress flag is **block-all**, so pulls will hang; whitelist the registry: ```yaml docker: true egress_whitelist: - "registry-1.docker.io" - "auth.docker.io" - "*.docker.com" # any-depth subdomains, not the apex - "docker.com" - "ghcr.io" - "*.pkg.github.com" ``` `*.docker.com` matches `production.cloudflare.docker.com` (and any other CDN host under that suffix) but **not** `docker.com` itself — add the apex separately. See [Networking](/k7/concepts/networking). ## Reference - Pins and helpers: [`src/k7/core/docker.py`](https://github.com/Katakate/k7/blob/main/src/k7/core/docker.py) - Kata vehicle script: [`src/k7/assets/k7-docker-vehicle.sh`](https://github.com/Katakate/k7/blob/main/src/k7/assets/k7-docker-vehicle.sh) - Backend comparison: [Backends](/k7/concepts/backends) # Tenants, roles and OIDC Source: https://docs.katakate.org/k7/guides/tenants-and-roles > Multi-tenancy in K7 0.5.0: Tenant objects with quotas and defaults, the four roles, API keys, OIDC login and role bindings, production vs development mode K7 0.5.0 turns the API into a multi-tenant control plane. Three things changed from 0.4.0: every credential now has a **role**; namespaces are grouped into **tenants** that carry quotas, defaults and dedicated nodes; and humans can sign in with **OIDC** instead of holding a key. Keys moved from `/etc/k7/api_keys.json` on a node to the `kube-system/k7-api-keys` Secret, so every API replica sees the same store. K7 roles are **application roles inside k7-api, not Kubernetes RBAC**. The `k7-api` ServiceAccount stays the only Kubernetes actor; nobody gets a Kubernetes identity through K7, and a K7 `admin` cannot `kubectl` anything. Protect kubeconfigs and node root separately. ## Roles | Role | Scope | Can | |---|---|---| | `read-only` | one tenant | list/get sandboxes, metrics and snapshots | | `user` | one tenant | `read-only` plus create/delete, exec, logs, pause/resume, fork, snapshots | | `operator` | cluster | read everything, snapshot GC, node storage, node dedication, tenant read; no exec, no keys | | `admin` | cluster | everything, including API keys, tenants, role bindings and remote install | One table (`PERMISSIONS` in `src/k7/api/authz.py`) maps every route to its roles; a route missing from it answers 500 and fails CI. Checks compose — route role, then tenant, then the key's namespace/node narrowing — and the first failure is a `403` that names the missing role or scope. `k7 whoami` / `GET /api/v1/auth/whoami` shows what a credential resolves to. ## Tenants A `Tenant` (`tenants.k7.katakate.org/v1alpha1`, cluster-scoped) claims namespaces and owns everything a tenant key may do: ```bash k7 tenant create acme -n acme-dev -n acme-prod \ --default-size cpu=1,memory=2Gi,disk=10Gi \ --quota sandboxes=20,vcpu=16,memory=32Gi,snapshotStorage=200Gi \ --max-size cpu=8,memory=32Gi \ --network egressBps=100M,ingressBps=100M k7 nodes dedicate k7-node-01 --tenant acme k7 generate-api-key acme-ci --role user --tenant acme k7 tenant list # quotas and reconciled usage ``` - **Namespaces** are created if missing and labelled `k7.katakate.org/tenant`. A tenant key reaches every namespace of its tenant (intersected with its own `-n` list), read from the Tenant on every request — a namespace added later works without reissuing the key, and a key whose tenant is deleted reaches nothing. A namespace claimed by two Tenants (only possible by writing CRs by hand) fails every request there with 500 `TenantConflict`. - **Dedicated nodes** (`spec.nodes`, or `k7 nodes dedicate`): the tenant's sandboxes land only there (node selector + toleration), and the `NoSchedule` taint keeps everyone else off. This is what gives tenant isolation on k7d, whose daemon is shared per node — a key's `--node` pin alone restricts that key, not the others. - **`defaultSize`** (`cpu`, `memory`, `disk`) fills creates without limits. Production mode requires CPU and memory on every sandbox, so set it or make callers pass `limits`. - **Quotas** (`sandboxes`, `vcpu`, `memory`, `disk`, `snapshotStorage`) are checked at K7's two choke points — the sandbox Deployment create and the VolumeSnapshot create — so create, fork, restore, pause-with-snapshot and named snapshots are all covered. The check and the create run under a per-tenant Lease (`k7-quota-`), so concurrent creates on any number of API replicas cannot overshoot; a Lease not acquired in 5 s is 503 and creates nothing. Over-quota is `403 QuotaExceeded` with the dimension, limit, usage and request, and is audit-logged. Usage is **reserved, not used**: a paused sandbox keeps its vCPU, memory and disk until deleted. The leader replica writes `Tenant.status.usage` every 60 s. - **Per-VM limits** : `maxSandboxSize` caps a request (400 above it). `network` (`egressBps` / `ingressBps`, bits/s) is enforced by the hypervisor — Firecracker token buckets on kfd, `tc` on the tap for kql — so it works whatever the CNI; a request may lower a cap, never raise it. A backend that **cannot** enforce a configured limit refuses the create with `400 LimitNotEnforceable` instead of running unlimited: `diskIO` on every backend (no block rate limiter in Kata 3.24 or k7d yet), `network` on k7d / k7d-fc, and a `disk` quota or `maxSandboxSize.disk` on k7d / k7d-fc (K7 cannot read a k7d VM's writable-disk size). - **Enforcement is at the K7 API.** `kubectl apply` by a cluster admin bypasses it. For a single-namespace tenant a Kubernetes `ResourceQuota` (`k7-tenant-quota`) is the backstop; multi-namespace tenants have none, since a ResourceQuota cannot span namespaces. - **Migrating from 0.4.0 namespace-scoped keys:** `k7 tenant adopt --from-keys` proposes one tenant per connected group of key namespaces (dry run), `--apply` creates them and binds the keys. ## API keys ```bash k7 generate-api-key NAME --role read-only|user|operator|admin [--tenant T] [-n NS]... [--node NODE]... [--expires-days 365] k7 list-api-keys # role, tenant, scopes; warns on unscoped_legacy k7 revoke-api-key NAME ``` Tokens are `k7__`; the store holds `sha256(secret)` (the secret is 256 random bits, so a slow KDF adds nothing) in the `kube-system/k7-api-keys` Secret. The API never writes the store on authenticate: it serves a snapshot refreshed every 5 s and answers 503 rather than trust one older than 60 s, so revocation reaches every replica within one refresh. Key commands edit the Secret through kubectl (run them on a server node), or go through k7-api with `--api` using an `admin` credential (`POST /api/v1/keys`). Protect etcd (`--secrets-encryption` on K3s, KMS on managed clusters): anyone with `get secrets` in `kube-system` reads the hashes. **Upgrading from 0.4.0:** `k7 install` runs `k7 api keys migrate`, which copies `/etc/k7/api_keys.json` into the Secret and gives every pre-role key a role: `admin: true` → `admin`, namespace-scoped → `user` (still narrowed), unscoped → `admin` flagged `unscoped_legacy`. Such keys keep working in development mode; re-issue them with a real role before switching to production. ## OIDC login Install with all five settings (or none; `https://` issuers only): ```bash k7 install ... \ --oidc-issuer https://idp.example.com/realms/k7 \ --oidc-audience k7-api \ --oidc-client-id k7-cli \ --oidc-groups-claim groups \ --oidc-username-claim email ``` Then `k7 login` (device flow; `--browser` for Authorization Code + PKCE on a loopback redirect) caches the tokens in `~/.config/k7/credentials.json` (0600) and refreshes them itself; `k7 logout` deletes the cache and revokes at the IdP. Clients send `Authorization: Bearer `; k7-api validates statelessly — signature against the issuer's JWKS (cached `--oidc-jwks-ttl` seconds, default 3600; a failed refresh is 503, never a stale allow), `alg` RS256/ES256 only, `iss`, `aud`, `exp` (60 s leeway) and `sub` required, `email_verified: true` when the username claim is `email`. k7-api stores no sessions and never sees refresh tokens. Register `k7-cli` as a **public** client with the device grant enabled. Per-IdP claim settings (Keycloak, Entra ID, Okta, Auth0, Dex) are in [docs/HARDENING.md](https://github.com/Katakate/k7/blob/main/docs/HARDENING.md#idp-claim-settings). ### Role bindings Groups, usernames or subjects map to roles through ConfigMap `k7-role-bindings`, read on every request: ```yaml # k7 role-bindings apply -f bindings.yaml bindings: - {group: k7-admins, role: admin} - {group: sre, role: operator} - {group: acme-devs, role: user, tenant: acme} - {group: acme-viewers, role: read-only, tenant: acme} deny_subjects: [] # OIDC `sub` or username; blocks on the next request ``` A subject in several groups gets the highest role per tenant plus the highest cluster role. An unbound subject gets a 403 that echoes its username and groups, to debug claim shapes. A write that leaves no `admin` binding while no unexpired `admin` key exists is refused. A JWT is valid until `exp` — configure 1 h or shorter tokens at the IdP; `deny_subjects` is the immediate stop. Keep one **break-glass `admin` key** offline: keys keep working during an IdP outage, and `k7 generate-api-key` with the node kubeconfig always works. ## Production and development modes `--api-mode` is chosen at install and never defaulted (chart: `api.mode`). Every response carries `X-K7-Mode`; `/health` and `k7_api_info{mode}` report it. | | `production` | `development` | |---|---|---| | Keys without a tenant or cluster role (`unscoped_legacy`, pre-role, namespace-only) | refused at startup (k7-api names each one) and `403` on every route | work as cluster `admin` | | `read-only` / `user` keys | need `--tenant` | may be unscoped | | Sandbox CPU and memory limits | required (from `limits` or the tenant's `defaultSize`) | optional | | `--api-insecure-http` | refused | allowed | | Metering | `--metering-sample-interval` required | 15 s default | | `--api-remote-install` | refused before deploying | allowed | Switch a development cluster to production by re-running `k7 install --api-mode production …` once every key has been re-scoped (`k7 list-api-keys` shows none flagged `unscoped_legacy`). ## Checklist 1. Migrate keys (`k7 api keys migrate` runs on install) and re-issue every `unscoped_legacy` key with the narrowest real role. 2. Give machines tenant keys with `--expires-days`; give humans OIDC. 3. Create tenants with `defaultSize` and quotas; dedicate nodes for tenants that must not share a k7d daemon. 4. Bind groups, not people; keep a break-glass `admin` key offline. 5. Ship the audit log (k7-api stdout, logger `k7.audit`). 6. Install with `--api-mode production`. Reference: [SECURITY.md](https://github.com/Katakate/k7/blob/main/SECURITY.md), [docs/HARDENING.md](https://github.com/Katakate/k7/blob/main/docs/HARDENING.md), API: [Identity](/k7/api/endpoints/identity), [Tenants](/k7/api/endpoints/tenants). # Observability and metering Source: https://docs.katakate.org/k7/guides/observability-metering > Prometheus metrics, JSON logs with request ids, dashboards and alerts, the opt-in monitoring stack, and per-tenant usage metering with showback Since 0.5.0, `k7-api` and `k7-agent` export Prometheus metrics, log JSON lines, and measure what every tenant and sandbox uses. K7 ships Grafana dashboards, alert rules with a runbook per alert, and `k7 install --monitoring` can install a monitoring stack for a single-box install. Nothing is priced: rating and invoices belong to whatever consumes the usage events. ## Metrics Both processes serve `/metrics` (and nothing else) as plain HTTP on port **9464**. Only the pods you name may reach that port: | Install | Setting | |---|---| | `k7 install` | `--metrics-scraper :