Security model - Katakate

Katakate is built for executing untrusted code — model output, third-party scripts, anonymous CI jobs. The threat model assumes the workload inside a sandbox is hostile and tries to escape, exfiltrate data, or attack other tenants.

This page summarizes the layered defenses; for hands-on configuration see the API security & networking reference.

Defense in depth#

Each boundary below has to be defeated before an attacker reaches the next one. The untrusted workload is at the center; the cluster is the outermost ring.

A successful escape from the workload has to defeat every layer above before it can reach another tenant or the host.

VM isolation#

Every sandbox is a Kata Containers pod, which means a real VM with its own kernel running on KVM. The workload sees a virtio block device for its rootfs, a virtio NIC, and not much else. The virtual hardware surface is tiny compared to a shared host kernel.

kata-firecracker-devmapper uses Firecracker — ~50k LoC Rust microVMM, ~5MB resident, ~125ms boot. The Firecracker process runs inside the jailer: a chroot jail with an empty filesystem, dropped capabilities, a restrictive seccomp filter, and an unprivileged UID. An integration test (tests/integration/test_firecracker.py::test_jailer_active) verifies the jailer is active on every install.
kata-qemu-longhorn uses QEMU — bigger surface than Firecracker but still hardware-isolated, and Kata applies its own sandboxing primitives.
k7d uses Katakate’s own KVM VMM (runtimeClassName: k7). One daemon hosts every VM; privilege separation splits that daemon into a root node broker (no listening socket) and a uid-k7d VM broker that parses guest and shim input. Live CoW fork still requires sibling forks of one tree to share the VM broker’s address space — that is what makes the ~5 ms fork possible, and it means sibling forks of one tree are isolated more weakly than VMs on separate VMMs. See the k7d security model.
k7d-fc keeps the k7d daemon for fork / pause but boots each guest on upstream Firecracker under the upstream jailer (runtimeClassName: k7-fc): a separate VMM process per VM with its own uid, chroot, pid namespace, cgroup, and Firecracker’s seccomp. A guest→VMM escape lands in that jail, not in the daemon. Pick it for multi-tenant --docker. Trade-offs: no hostPath / virtiofs, no time warp, a 32 MiB / 50m PodOverhead.

On both Kata backends the pod’s securityContext reaches the runtime inside the VM: Kata’s guest seccomp is enforced (disable_guest_seccomp = false), so the container’s RuntimeDefault profile is applied by the in-guest runtime rather than dropped. The k7d shim likewise passes linux.seccomp, maskedPaths / readonlyPaths, device cgroups, capabilities, and noNewPrivileges through to guest runc, stripping only what the guest kernel cannot honour (AppArmor profile, host cgroup path, host namespace paths).

Container hardening#

Inside the VM, the user container is further locked down:

Knob	Default	What it does
`cap_drop`	`["ALL"]`	All Linux capabilities are dropped unless you explicitly add some back via `cap_add`.
`cap_add`	`[]`	Add back only what your workload needs (e.g. `["CHOWN"]`).
`allowPrivilegeEscalation`	`false`	Always — `setuid`/file caps cannot raise privileges.
Seccomp profile	`RuntimeDefault`	Standard runtime seccomp filter, enforced by the in-guest runtime on top of Kata/jailer’s own.
`pod_non_root`	`false`	Set `true` to run the entire pod as UID/GID/FSGroup 65532 (consistent FS ownership).
`container_non_root`	`false`	Set `true` to run the main container as UID 65532.

Info

Some package managers (apk add, apt-get install) require root inside the container. Either run them in before_script with the defaults, or prebuild your image with the dependencies baked in.

Warning

--docker changes the picture inside the VM. On k7d / k7d-fc the sandbox container stays unprivileged, but docker.sock is guest root — anything that can reach it owns the guest, so the container securityContext no longer bounds the VM. On kfd / kql a privileged docker-vehicle container runs in the same Kata VM. In every case the isolation boundary is the VM; see Docker in a sandbox.

Network isolation#

Sandboxes never reach the platform. A cluster-wide deny-only CiliumClusterwideNetworkPolicy (k7-sandbox-platform-deny) blocks every sandbox — even --egress-open ones — from the nodes, the Kubernetes API server, cloud metadata / link-local, and kube-system / longhorn-system pods except CoreDNS. Cilium-only; Flannel clusters do not get this.
Ingress is denied by default and opt-in per sandbox: --ingress-port opens TCP ports, --ingress-from (sandbox: / namespace: / cidr:) scopes who may connect, --expose-port publishes through a NodePort with the real client IP preserved. Opening a port without a source means “other sandboxes in this namespace”, never the world. kubectl exec / k7 shell are unaffected (they use the K8s API).
Egress is configurable per sandbox via egress_whitelist: open by default, blocked with [], restricted by CIDRs, or restricted by FQDNs (with Cilium). With [] or a CIDR list, DNS is blocked. With an FQDN list, Cilium’s DNS proxy answers only the allowlisted names and returns NXDOMAIN for everything else — other queries never reach CoreDNS or its upstream, closing the DNS-exfiltration channel that 0.4.0’s matchPattern: "*" rule left open. Without Cilium, a CIDR entry that contains the cluster DNS server is refused. Forks inherit the source’s policy.
Only k7 sets runtime annotations. The ValidatingAdmissionPolicy k7-runtime-annotations denies any change to k7d.katakate.org/* and io.katacontainers.* annotations on pods and pod templates except by k7-api, system:masters and the controller manager — so a namespace admin cannot put a Deployment’s VM on a shared k7d bridge, fork another VM on the node, or pick a host Kata config. Containerd forwards only the keys k7 sets.
Per-VM bandwidth caps (network_limits, tenant limits.network) are enforced by the hypervisor on kfd and kql; on k7d / k7d-fc a configured cap is refused rather than ignored.
Details and the cidr:-on-Cilium caveat: Networking.

API transport (TLS)#

k7 install serves k7-api on NodePort 31007 over HTTPS by default. The API container stays HTTP on :8000 (kubelet probes unchanged); a Caddy sidecar in the same pod terminates TLS. API keys no longer travel in cleartext.

`k7 install` flag	Certificate
(none)	Playbook-minted cluster CA at `/etc/k7/tls/ca.crt` (Let’s Encrypt cannot issue for a bare IP). Copy that file to your laptop and `k7 config set api.ca ./ca.crt`, or pass `--api-ca` / `K7_API_CA`.
`--api-hostname NAME`	Let’s Encrypt via Caddy (HTTP-01 on host port 80). `NAME` must be a DNS name whose A record is the first master’s IP; a bare IP is refused. Clients use the system trust store.
`--api-tls-cert` + `--api-tls-key`	Operator-supplied PEM pair. Incompatible with `--api-hostname`.
`--api-insecure-http`	The previous plain-HTTP NodePort, bit for bit. Keys travel in cleartext. Cannot be combined with the hostname or cert flags.

The CLI and SDK refuse verify=False on an https:// URL — point them at the CA instead. Optionally restrict the NodePort to your client CIDRs with --api-allow-cidr (Networking). None of this is rate limiting.

API identity: keys, roles, tenants, OIDC#

Every credential has a role — read-only / user per tenant, operator / admin cluster-wide — checked inside k7-api on every route from one PERMISSIONS table. K7 roles are not Kubernetes RBAC; nobody gets a Kubernetes identity through K7. See Tenants, roles and OIDC.
API keys are k7_<id>_<secret> (256 random bits), shown once. The store holds sha256(secret) in the Kubernetes Secret kube-system/k7-api-keys — never on a node’s disk, never written on authenticate. k7-api serves a snapshot refreshed every 5 s and answers 503 rather than trust one older than 60 s; revocation reaches every replica within one refresh. The k7-api ServiceAccount can only get / patch that Secret. Protect etcd (--secrets-encryption, KMS).
Tenant scope is the control-plane tenancy boundary: a tenant key reaches its tenant’s namespaces (narrowed by -n), cross-tenant requests are 403 before any cluster call, fork and restore never leave their namespace, and quotas are enforced under a per-tenant Lease. Node scope (--node) plus k7 nodes dedicate (label + NoSchedule taint) is the placement boundary that keeps a tenant off another tenant’s k7d daemon. Cluster-scoped reads such as GET /api/v1/nodes/storage need operator/admin (CWE-862 fix, reported by @skeletonsec).
OIDC (k7 install --oidc-*): humans send Authorization: Bearer <JWT>; k7-api validates statelessly against the issuer’s JWKS (RS256/ES256, cache expires, failed refresh is 503), maps groups/users/subjects to roles through k7-role-bindings, and keeps no sessions. deny-subjects is the immediate stop; a JWT otherwise lives until exp.
Audit log: one JSON line per mutating request and per 401/403 on k7-api stdout (k7.audit) with principal, role, tenant, route, status, latency and client IP; never tokens, groups, bodies or exec text. Emission failure fails the request closed.
Registry SSRF guard: control-plane OCI inspection rejects registry hosts that are not on the allowlist (default registry-1.docker.io, ghcr.io, quay.io, public.ecr.aws; extend with K7_REGISTRY_ALLOWLIST) or that resolve to loopback / private / link-local / reserved addresses. Redirects are disabled.
Remote install (POST /api/v1/install) is off by default (404) and, when enabled with --api-remote-install in development mode, takes only node names from a cluster-wide admin: the installer image and config are fixed at deploy time, and the API never runs Ansible. Up to 0.4.0 any valid key could post playbook content.

The API runs as a K3s Deployment (2 replicas, uid 1000, all capabilities dropped) with its own ServiceAccount and a scoped ClusterRole — no admin kubeconfig is ever mounted. k7-agent is not privileged: uid 0 with the k7d group (gid 770), all capabilities dropped, read-only root filesystem, RuntimeDefault seccomp, no ServiceAccount token, and only the containerd socket directory, /run/k7d, /var/lib/k7d (ro) and /run/k7 (ro) mounted.

Production and development modes#

k7 install --api-mode (chart api.mode) is required and never defaulted. production refuses to start while the key store holds a key without a tenant or cluster role (pre-role, migrated unscoped_legacy, or namespace-only) and names each one; returns 403 to such keys on every route; requires CPU and memory limits on every sandbox (filled from the tenant’s defaultSize); refuses --api-insecure-http and remote install; and makes metering explicit (--metering-sample-interval). development keeps unscoped keys working as cluster admin. Every response carries X-K7-Mode.

Supply chain#

Tags build ghcr.io/katakate/k7-api and ghcr.io/katakate/k7-node-installer, scan them with Trivy, sign them with cosign keyless, and attach an SPDX SBOM and SLSA provenance; both digests go into the release’s images.lock.yaml, which is what k7 install deploys (imagePullPolicy: IfNotPresent, signature verified). The .deb and wheel ship with sha256sums.txt and cosign bundles. Every install download (K3s script, CNI, Firecracker, Kata, k7d, Cilium CLI, Hubble, Longhorn, Caddy by digest) is checked against a pinned sha256; a mismatch is fatal. Building k7-api:local from a checkout (--api-image local) is the unsigned opt-out and there is no silent fallback between the two.

Operational guidance#

Always lock down egress for untrusted workloads. Prefer FQDN allowlists (Cilium) — much less error-prone than tracking CIDR ranges that change.
Don’t whitelist public DNS resolvers (1.1.1.1, 8.8.8.8) — that re-enables DNS exfiltration.
Rotate API keys with k7 revoke-api-key + k7 generate-api-key --role …. Treat them as credentials. Give machines tenant keys (--role user --tenant T, --expires-days) and humans OIDC; keep one break-glass admin key offline; run --api-mode production once no key is flagged unscoped_legacy.
Keep TLS on and distribute the CA (default install) — or use --api-hostname for a publicly trusted certificate. Add --api-allow-cidr when you know your client CIDRs. Do not fall back to --api-insecure-http on a public node.
Be deliberate with ingress. Open only the ports you need, scope them with sandbox: / namespace:, and treat --expose-port + cidr:0.0.0.0/0 as publishing to the internet.
Pin image tags — :latest is convenient but non-reproducible. K7 itself pins every image and Kubernetes manifest version it ships; do the same in your sandbox configs.
Dedicate nodes to tenants that must not share a k7d daemon (k7 nodes dedicate), and keep tenant boxes in [k7_agents]: root on a [k7_servers] node is cluster-admin, root on an agent is that node only (k7-agent has no ServiceAccount token).
Set quotas per tenant and ship the audit log and usage events (Observability and metering).

Known limitations#

Pre-1.0 software under active security review. Avoid for hyper-sensitive workloads until 1.0.
No API rate limiting yet — put a proxy with rate limits in front for public exposure. TLS and --api-allow-cidr are not a “secure API” claim.
Platform isolation, FQDN egress and --api-allow-cidr are Cilium-only — --cni flannel clusters (and the chart’s networkPolicy.provider: kubernetes) keep per-sandbox CIDR policies but not the cluster-wide deny or domain allowlists.
Quota enforcement is at the K7 API; kubectl apply by a cluster admin bypasses it (single-namespace tenants get a ResourceQuota backstop). diskIO limits are accepted by the Tenant CRD but no backend enforces them yet; k7d / k7d-fc cannot enforce network caps or disk quotas — such creates are refused (LimitNotEnforceable) rather than run unlimited.
Host compromise is cluster-wide on servers. --node / dedicate isolate k7d placement, not a root shell on [k7_servers].
An OIDC token is valid until it expires; deny-subjects is the only revocation. Entra ID group overage (> ~200 groups) is not followed — bind by user / subject or use app roles.
No independent security audit yet.
AppArmor profiles for the sandbox container are on the roadmap (the k7d Ubuntu guest ships its own).
TEE (Trusted Execution Environment) support is on the roadmap.
Tetragon (runtime enforcement) was evaluated and rejected for now; Hubble observability is available opt-in.

See SECURITY.md for the full disclosure policy and reporting channel (security@katakate.org).