Skip to main content

Use k7 -h for built-in help. Below are the primary commands.

Talking to the K7 API#

By default, every sandbox-management command (create, list, delete, pause, resume, fork, restore, exec, logs, snapshot *) talks to the K7 API over HTTPS. The CLI resolves the URL, key, and CA in this order:

  1. --api-url / --api-key / --api-ca flags (global, before the command).
  2. K7_API_URL / K7_API_KEY / K7_API_CA environment variables.
  3. ~/.config/k7/config.toml — written by k7 config set api.url ... / k7 config set api.key ... / k7 config set api.ca ....
  4. On a cluster node only: /etc/k7/api_endpoint and /etc/k7/tls/ca.crt. The API key is never stored in the clear on disk — paste the token k7 generate-api-key printed into k7 config set api.key.

Humans on an OIDC-enabled cluster can use k7 login instead of a key: the CLI caches the IdP tokens in ~/.config/k7/credentials.json (0600), refreshes them itself and sends Authorization: Bearer <JWT>. A configured api.key and a cached login are both accepted; k7 whoami tells you which one is in effect and what role it resolves to.

An https:// URL is always verified: there is no --insecure. If the cluster was installed with --api-hostname (Let’s Encrypt) the system trust store is enough and api.ca is not needed; with --api-insecure-http use an http:// URL.

--core (debugging / on-node CI only)#

A hidden global flag --core skips the API and calls K7Core directly in-process. Useful when the API itself is misbehaving and you have kubeconfig access on the node. Most users should never need it.

config#

Writes ~/.config/k7/config.toml with mode 0600. Same posture as ~/.kube/config and ~/.docker/config.json. api.ca is the path of the CA bundle to trust for the https:// API URL (the cluster CA from /etc/k7/tls/ca.crt on the default install).

install#

Install K7 components on host node(s).

Since 0.5.0, k7 install runs from any directory: it deploys the signed GHCR k7-api image by digest and needs neither a checkout nor Docker on the node (--api-image local restores the 0.4.0 build-from-cwd path).

  • -v: verbose output
  • —backend, -b: required. Comma-separated backends: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d, k7d-fc, or none (control plane only, no sandbox runtime). Omit is an error. k7d-fc adds jailed Firecracker (installs pinned Firecracker v1.16.2 + jailer, RuntimeClass k7-fc).
  • —api-mode: required with the API (no default). production: every read-only/user key needs a tenant, every sandbox needs CPU and memory limits (filled from the tenant’s defaultSize), HTTPS only, metering mandatory, remote install refused. development: unscoped keys keep working as admin. See Production and development modes.
  • —disk: block device for the kfd LVM thin-pool (auto-detected when omitted). On dual-NVMe boxes leave this unset — NVMe names swap across reboots. Inventory equivalent: k7_disk_source=spare-disk|loopback|device:/dev/<disk>.
  • —k7d-version: k7d release to download (k7d backends only). Playbook default is 0.7.0; the playbook only knows that version’s sha256, so another version needs k7d_sha256=<hex> as an inventory/extra var or --k7d-artifact.
  • —k3s-version: override the pinned K3s (v1.36.5+k3s1). Only applies when K3s is not installed yet.
  • —k7d-artifact: local k7d release tarball to install instead of downloading (k7d backends only).
  • -i, —inventory: Ansible inventory for multi-node installs (per-host k7_backends is authoritative — don’t combine with --backend).
  • —ha: multi-master HA with embedded etcd (inventory with 3+ servers).
  • —role / —join / —join-token: add a single node to an existing cluster.
  • —cni: cilium (default; FQDN egress, platform isolation, --api-allow-cidr) or flannel (CIDR-only egress).
  • —hubble: opt-in Cilium Hubble flow observability (relay + pinned hubble CLI, no UI). Requires Cilium. Not a security control.
  • —dns-upstream IP: repeatable (IPv4, at most 3). Upstream resolvers for CoreDNS instead of the host’s /etc/resolv.conf (inventory k7_dns_upstreams). See Networking.
  • —metrics-scraper NS:label=value[,…]: which pods may scrape /metrics on k7-api and k7-agent port 9464. Without it no pod can reach the port.
  • —monitoring: also install the pinned kube-prometheus-stack (namespace monitoring) with K7’s dashboards and alerts; implies the scraper selector. A convenience for single-box installs. See Observability and metering.
  • —metering-sample-interval N: usage sample period in seconds (5–60). Required with --api-mode production; development uses 15. Related: --metering-storage-interval (60), --metering-spool-size (1Gi), --metering-spool-high-water (80), --metering-require-delivery (503 on creates when a node’s spool is above its high-water mark), --metering-prometheus-url (for k7 usage; --monitoring sets its own), --metering-sinks-file (http/OpenMeter or stdout sinks).
  • —oidc-issuer / —oidc-audience / —oidc-client-id / —oidc-groups-claim / —oidc-username-claim: all five or none; https:// issuers only. Enables k7 login and bearer JWTs. --oidc-jwks-ttl (3600 s, min 10) is optional. Reinstalling without them turns OIDC off.
  • —api-image: local builds k7-api:local from the checkout in the cwd (unsigned, needs Docker) or name@sha256:<digest>. Default: the digest in the release’s images.lock.yaml, cosign-verified.
  • —api-remote-install: enable POST /api/v1/install (cluster-wide admin), deploying the k7-node-installer DaemonSet in k7-system. Development mode only. --node-installer-image and --node-installer-allow-server-restart go with it. Re-installing without the flag removes the stack again.
  • —api-allow-cidr CIDR: repeatable. Restrict the k7-api NodePort to these source CIDRs via a CiliumNetworkPolicy and externalTrafficPolicy: Local. Off by default; Cilium only; 0.0.0.0/0 warns.
  • —api-hostname NAME: DNS name pointing at the first master; Caddy obtains a Let’s Encrypt certificate (HTTP-01 on :80). A bare IP is refused.
  • —api-tls-cert / —api-tls-key: operator-supplied server certificate + key (PEM). Both required together; incompatible with --api-hostname.
  • —api-insecure-http: serve k7-api as a plain HTTP NodePort (no Caddy sidecar). Keys travel in cleartext. Cannot be combined with the hostname / cert flags.
  • —no-api: CLI-only install without the k7-api Deployment.
  • —node-prep-only: run only the node-prep role (no K3s install, no cluster resources) — what the Helm chart’s installer image does on nodes of an existing cluster.

See Multi-node clusters for the inventory format and topology, and Security model for how the TLS options fit together.

version#

Check version of installed K7 .deb package

create#

Create a sandbox from a YAML file or flags.

pause#

On kql, scale a sandbox to 0 replicas, optionally taking a crash-consistent VolumeSnapshot of its root PVC (longhorn VolumeSnapshotClass) — and of the <name>-docker-lh graph PVC when the sandbox was created with --docker. On k7d / k7d-fc, pause freezes the live VM in place (vCPUs stop, memory stays) — no snapshot. --snapshot is kata-qemu-longhorn-only.

  • --snapshot (optional, takes optional value): when present, take a VolumeSnapshot of the sandbox’s root PVC (<sandbox>-root-lh) before scaling down. Bare flag → snapshot named <sandbox>-paused-<unix-ts>; with value → snapshot named as given.
  • Pause-time snapshots persist as VolumeSnapshot objects in the same namespace until you delete them.

resume#

On kql, scales the Deployment back to 1; state on the root PVC survives. On k7d and k7d-fc, restarts the vCPU loop — in-memory state is intact, and k7 exec works right after resume (the Firecracker vsock gating that broke this on 0.3.1 is fixed by the pinned v1.16.2 — see Backends).

restore#

Boot a brand-new sandbox from a standalone VolumeSnapshot. Unlike k7 fork, this does not require the original sandbox’s Deployment to still exist — it only needs the snapshot.

Image / backend / docker / limits / root-disk-size default to the source sandbox’s values via the k7.io/source-* annotations stamped on the snapshot at creation time (pause / named only — fork-temp snapshots are auto-deleted). Any flag above overrides the corresponding annotation. --no-keep-snapshot deletes the source snapshot after the new sandbox is Ready. Restore is kata-qemu-longhorn-only.

fork#

Clone an existing sandbox. Behavior depends on the backend:

  • k7d / k7d-fc — warm CoW fork of disk and memory (~5 ms at the VMM, ~2–4 s end-to-end to a Ready pod). The child inherits files, tmpfs, and running processes — including a live --docker engine and its inner containers — and, with k7d 0.9.0+, keeps its network (the fork gets its own bridge and NAT, so source and child have egress at the same time). Lands on the source’s node. Sandboxes with a real CRI sidecar cannot be forked.
  • kql / kata-qemu-longhorn — disk-only clone via a crash-consistent VolumeSnapshot of <sandbox>-root-lh (and <sandbox>-docker-lh with --docker), then a cold boot (~45 s single-node; longer with Longhorn replicas ≥ 2).
  • kfd — rejected.

The fork inherits the source’s egress and ingress policy (but not its --expose-port Service) and is rolled back if the policies cannot be created. Fork never leaves the source’s namespace, and counts against the tenant’s quotas like a create.

On kql only:

  • --snapshot (optional): friendly name for the temporary VolumeSnapshot. If omitted, an auto-named <source>-fork-<unix> snapshot is created and auto-deleted once the new sandbox’s PVC is bound.
  • Pass --snapshot=NAME to opt the snapshot out of GC (it becomes a named snapshot — persistent).

snapshot#

Manage VolumeSnapshot objects directly. See /k7/api/endpoints/snapshots for the lifetime rules and the three kinds (pause, fork, named).

gc only ever touches kind=fork snapshots older than --keep-fork-for. Pause and named snapshots are never collected, even with --keep-fork-for=0s. A CronJob (k7-snapshot-gc in kube-system) runs the same logic every 10 minutes as a backstop, in case the inline cleanup after k7 fork ever misses a snapshot. GC also reaps orphan VolumeSnapshotContent objects whose VolumeSnapshot is gone or deleting and clears the PVC source-protection finalizer, so a deleted namespace no longer sticks in Terminating on volumesnapshotcontent-bound-protection.

YAML configuration reference#

All fields map to the server-side SandboxConfig:

  • name (string, required): unique sandbox name.
  • image (string, required): container image, e.g. alpine:latest. Registry host must be public and on the allowlist (registry-1.docker.io, ghcr.io, quay.io, public.ecr.aws; extend with K7_REGISTRY_ALLOWLIST).
  • backend (string, optional): kata-firecracker-devmapper / kfd, kata-qemu-longhorn / kql, k7d, or k7d-fc.
  • node_name (string, optional): Kubernetes node to pin the sandbox to (k7 nodes list); required when the API key is scoped to several nodes.
  • network_limits (object, optional): egressBps / ingressBps in bits per second ("100M"). kfd and kql only (Firecracker token buckets / tc on the tap); on k7d / k7d-fc the create is refused with 400 LimitNotEnforceable rather than running unlimited. May only lower a tenant cap.
  • docker (boolean, default false): run a pinned dockerd inside the sandbox with overlay2 on a block graph disk — see Docker in a sandbox.
  • docker_disk (string, default 20Gi): graph disk size for docker: true.
  • sidecar (string, optional): deprecated; docker is an alias of docker: true.
  • namespace (string, default default): Kubernetes namespace.
  • env_file (string, optional): (absolute) path to an env file on the host node.
  • egress_whitelist (array of CIDR or FQDN strings, optional): omit / null = open egress; [] = block-all; listed entries = allowlist. FQDNs need Cilium (the default CNI). CLI k7 create without --egress / --egress-open is block-all.
  • ingress_ports (int[], optional): TCP ports to open for inbound traffic. Omit / [] = deny all ingress (default).
  • ingress_from (string[], optional): who may reach ingress_ports — sandbox:<name>, namespace:<ns>, cidr:<cidr>. Omitted with ports set = sandboxes in the same namespace. Requires ingress_ports.
  • expose_ports (int[], optional): publish these TCP ports outside the cluster via a NodePort Service; each must also be in ingress_ports.
  • volumes (object[], optional): hostPath / PVC entries. Refused on k7d-fc (Firecracker has no virtiofs).
  • limits (object, optional; required in production mode unless the tenant sets defaultSize): resource limits:
    • cpu (string): cores or millicores, e.g. "1" or "500m".
    • memory (string): e.g. "1Gi", "512Mi".
    • ephemeral-storage (string): e.g. "2Gi".
  • root_disk_size (string, kql only): Longhorn root PVC size. Default: the tenant’s defaultSize.disk, else 10Gi.
  • before_script (string, optional): shell script run once after container starts.
    • Runs via kubectl exec before egress netpol is applied; readiness waits for completion.
  • pod_non_root (boolean, optional): run Pod as non-root (UID/GID/FSGroup 65532).
  • container_non_root (boolean, optional): run container as non-root (UID 65532), no privilege escalation.
  • cap_add (string[], optional): add back Linux capabilities (default policy drops ALL).
  • cap_drop (string[], optional): override drop policy. If omitted, ALL is dropped by default.

Example k7.yaml:

Warning

Do not whitelist public DNS resolvers (e.g., 1.1.1.1, 8.8.8.8). Doing so re-enables DNS exfiltration (UDP/TCP 53 and DoH over 443). Prefer whitelisting only your own egress proxy IP and enforce DNS/DoH policies at the proxy.

If using package managers that require root (e.g., apk add, apt-get install) in before_script make sure you didn’t add security policies that prevent it such as running the pod or container as non-root. Check Security & Networking section in the API reference for more.

list#

Lists sandboxes with status, readiness, restarts, age, image, backend, node, and the NodePorts allocated by --expose-port (empty when the sandbox is not exposed). The table truncates names; k7 exec NAME still needs the full name you passed to create.

delete#

Deletes one sandbox.

delete-all#

Deletes all sandboxes in a namespace (with confirmation).

shell#

Opens an interactive shell in the sandbox pod. This is node-local (kubectl exec); it does not go through the K7 API. From a laptop use k7 exec NAME -- 'echo hello' — the CLI joins the command and the API runs it under sh -c, so pass redirects as one quoted string (k7 exec NAME -- 'echo hi > /tmp/state.txt'). Do not add an extra sh -c or the quotes are lost.

exec#

logs#

Shows a snapshot of the CRI container’s stdout (the sandbox container). Empty output with exit 0 means that container has not written anything — it is not a failure. k7 exec output does not appear here (exec goes through the agent / Kubernetes API). On k7d / k7d-fc, a sleep/alpine sandbox that you have only exec’d into commonly has empty logs; before_script on Kata backends is more likely to show up.

--follow is --core only (node-local kubectl logs -f).

top#

Top-like view of CPU and memory usage.

nodes storage#

Per-node storage-pool utilization, collected from the k7-agent DaemonSet on every node:

  • kata_thinpool — the kfd LVM thin-pool (lvs size, data%, metadata%).
  • k7d_disks — the k7d XFS disk pool at /var/lib/k7d/disks (df size / used / avail).

A node whose agent is unreachable shows a loud error entry — nodes are never silently omitted. Also available as GET /api/v1/nodes/storage (API reference) and Client.nodes_storage() in the SDK. This is cluster-wide topology: it needs the operator or admin role, and a tenant key gets 403.

nodes list / dedicate / undedicate#

NAME is the K3s Node name (the machine hostname) — what k7 create --node, k7 generate-api-key --node and dedicate take; it is not an inventory label. Dedication is what keeps other tenants’ sandboxes off a node’s k7d daemon; a key’s --node pin alone is a placement restriction on that key, not an exclusion of others. Both commands accept --api to go through k7-api (operator/admin) instead of a node kubeconfig.

api#

The K7 API is deployed automatically by k7 install and runs as a Kubernetes Deployment (k7-api in kube-system). K3s keeps it running and reschedules it on failure — there is no separate “start” step. The k7 api sub-app provides read-only diagnostics and a feature toggle:

k7-api runs 2 replicas spread across nodes with a PodDisruptionBudget (minAvailable: 1), as uid 1000; the replica holding the k7-tenant-status / k7-metering-leader Leases writes Tenant.status.usage and the storage usage samples. The Let’s Encrypt (--api-hostname) overlay pins back to one replica on the first master (hostPort 80 + certificate storage).

The endpoint is HTTPS by default, terminated by a Caddy sidecar in the k7-api pod. The cluster CA lives at /etc/k7/tls/ca.crt on the node — copy it to clients (k7 config set api.ca). k7 install --api-hostname (Let’s Encrypt), --api-tls-cert/--api-tls-key, and --api-insecure-http change what is served; see install.

To install without the API (CLI-only deployment, e.g. for local-only test rigs), pass --no-api to k7 install. Re-run k7 install later (without --no-api) to add it; the playbook is idempotent.

The previous top-level commands k7 start-api / k7 stop-api / k7 api-status / k7 get-api-endpoint are hidden deprecated aliases that print a one-line warning and forward to the new home.

API keys#

Tokens are k7_<id>_<secret>; only sha256(secret) is stored, in the kube-system/k7-api-keys Secret, and every API replica picks up a change within 5 seconds. Use with X-API-Key or Authorization: Bearer.

  • —role (required): read-only and user are per tenant (--tenant required in production mode; in development a user key without a tenant is an unscoped key with the pre-0.5 behaviour); operator (cluster-wide read, snapshot GC, node storage and dedication; no exec, no keys) and admin (everything, including keys, tenants, role bindings and remote install) are cluster-wide and refuse --tenant.
  • -n / —namespace (repeatable): narrows the key further, inside its tenant. A narrowed key cannot read or mutate another namespace and cannot run all-namespaces list/GC.
  • —node (repeatable): the key may only create/restore/fork onto these Kubernetes nodes (a single node auto-pins; several require node_name). Pair with k7 nodes dedicate for tenant isolation on k7d.
  • Key commands edit the Secret through kubectl, so they run on a server node (sudo); with --api they go through k7-api instead, using an admin key or k7 login.
  • k7 list-api-keys shows role, tenant and scopes, and warns on unscoped_legacy keys (pre-0.5 unscoped keys migrated as admin): re-issue them with a real role before switching to production mode.

login / logout / whoami#

k7 login discovers the IdP from GET /api/v1/auth/oidc, runs the OIDC device flow (or Authorization Code + PKCE with --browser), and caches the tokens in ~/.config/k7/credentials.json. Which role a human gets comes from the role bindings below. k7 whoami prints the resolved principal, role(s) and tenant. k7 logout deletes the cache and revokes the token at the IdP. The cluster must have been installed with the --oidc-* flags; otherwise login fails loudly.

role-bindings#

Maps OIDC groups, usernames or subjects to roles (per tenant for read-only/user, cluster-wide for operator/admin); lives in ConfigMap k7-role-bindings and is read on every request, so deny-subjects is the immediate stop for a compromised account. A write that would leave no admin binding while no unexpired admin key exists is refused. See Tenants, roles and OIDC.

tenant#

A Tenant (tenants.k7.katakate.org/v1alpha1, cluster-scoped) claims namespaces (created if missing; a namespace another tenant already claims is refused), dedicated nodes, a defaultSize, quotas and per-VM limits. Quotas are reserved, not used (a paused sandbox keeps its vCPU/memory/disk) and are checked under a per-tenant Lease, so concurrent creates on any number of API replicas cannot overshoot; over-quota is 403 QuotaExceeded. --disk-io limits exist in the CRD but no backend enforces them yet, so a tenant that sets them gets 400 LimitNotEnforceable on create instead of running unlimited. Full semantics: Tenants, roles and OIDC.

usage#

Showback from Prometheus: sandbox-, vCPU- and memory-seconds, egress bytes, disk and snapshot GiB·s, per tenant / namespace / backend / state. Needs a Prometheus that scrapes K7 (--monitoring, or --metering-prometheus-url); a tenant key may only ask about its own tenant. Also GET /api/v1/usage. See Observability and metering.

Flag reference (create)#

  • -n, —namespace: Kubernetes namespace (default default).
  • -f, —file: YAML config file (defaults to k7.yaml in the current directory when present).
  • name / image (positional): k7 create NAME IMAGE. There is no --name / --image flag.
  • -b, —backend: kfd / kql / k7d / k7d-fc (alias k7-fc, or the long names).
  • —docker: Run a pinned dockerd (overlay2 on a block graph disk). k7d / k7d-fc: guest agent service, forkable. Kata (kfd / kql): privileged docker-vehicle in the VM; kql persists the graph, kfd is ephemeral.
  • —docker-disk: Docker graph disk size for --docker (e.g. 20Gi, 40Gi). Default 20Gi on all backends.
  • —sidecar: Deprecated. --sidecar docker is an alias of --docker.
  • —cpu: CPU limit (e.g., 1, 500m).
  • —memory: Memory limit (e.g., 1Gi, 512Mi).
  • —storage: Ephemeral storage limit (e.g., 2Gi).
  • —root-disk-size: Longhorn root PVC size (kql only; default: tenant defaultSize.disk, else 10Gi).
  • —node: pin to this Kubernetes node name (k7 nodes list). YAML: node_name.
  • —egress-bps / —ingress-bps: cap guest bandwidth in bits/s (100M). kfd/kql only; may only lower a tenant cap. YAML: network_limits.
  • —env-file: Path to env file on the host node injected as a Secret.
  • —egress ENTRY: Repeatable; whitelist CIDR or FQDN. Default (no flag): block all.
  • —egress-open: Allow all egress. Mutually exclusive with --egress.
  • —ingress-port PORT: Repeatable; TCP port to open for inbound traffic. Default (no flag): deny all ingress.
  • —ingress-from SOURCE: Repeatable; sandbox:<name> | namespace:<ns> | cidr:<cidr>. Default when --ingress-port is given: sandboxes in the same namespace. Reaching the sandbox from anywhere needs cidr:0.0.0.0/0, spelled out. Requires --ingress-port.
  • —expose-port PORT: Repeatable; publish a TCP port outside the cluster through a NodePort Service (externalTrafficPolicy: Local). Requires a matching --ingress-port. On a public node the node IP is public too.
  • —before-script: Shell script to run once after container start; runs with open egress before lockdown.
  • —entrypoint: Repeatable; override image ENTRYPOINT (ordered).
  • —cmd: Repeatable; override image CMD (ordered).
  • —pod-non-root / —no-pod-non-root: Pod-level non-root defaults.
  • —container-non-root / —no-container-non-root: Container runs as UID 65532, no privilege escalation.
  • —cap-add CAP: Repeatable; add back Linux capabilities (default drop ALL).
  • —cap-drop CAP: Repeatable; override default drop policy.
Info

Package installs like apk add require root inside the container. Either leave container_non_root disabled for setup or prebuild an image. See Security & networking.

Info

For kata-qemu-longhorn, the persistence wrapper requires /bin/sh, tar, and mount in the image. Use a Debian/Ubuntu-like base (or ensure those tools are installed).

Info

For kata-qemu-longhorn, the main container runs privileged to allow bind mounts for persistence.

Info

For kata-qemu-longhorn, the wrapper preserves the image ENTRYPOINT+CMD by default. Use --entrypoint and/or --cmd to override.