Skip to main content

This page covers sandbox security posture and how to configure it via the API.

Non-root execution#

  • pod_non_root (boolean, default false): Run the Pod as non-root (UID/GID/FSGroup 65532). Applies pod-wide filesystem ownership.
  • container_non_root (boolean, default false): Run the main container as non-root (UID 65532) and disallow privilege escalation.

Guidance:

  • Enable both flags for consistent non-root behavior and fewer permission surprises when writing to volumes.
  • Some package managers (e.g., Alpine apk add) require root. To run apk add inside the container, you have options:
    • Use before_script with a base image that already includes needed tools, or
    • Temporarily run the main container with root by leaving container_non_root disabled for setup, or
    • Build a custom image with dependencies pre-installed (recommended for production reproducibility).

Example (non-root):

Example (install packages first as root, then lock down egress):

Linux capabilities#

Default policy: drop ALL capabilities. Add back only what you need. If you specify cap_drop explicitly, you override the default; to keep drop ALL and add back minimal caps, leave cap_drop unset and only use cap_add.

  • cap_drop (string[]): Capabilities to drop. If omitted, ALL is dropped by default.
  • cap_add (string[]): Capabilities to add back.
  • allow_privilege_escalation: always set to false.
  • Seccomp profile: RuntimeDefault.

Examples:

Minimal add-back while still dropping ALL by default:

Override drop policy (not recommended unless you know why):

Common capability requirements#

When you encounter permission errors in before_script or during container execution, you may need to add specific capabilities. Here are common use cases:

Package managers (apt-get, yum, dnf):

  • SETUID/SETGID: Required for package managers to drop privileges during installation
  • CHOWN: Needed for changing file ownership during package installation
  • DAC_OVERRIDE: Allows bypassing file permission checks (needed for installing packages)

Alpine package manager (apk):

File operations requiring ownership changes:

Network operations (raw sockets, packet capture):

Adding NET_RAW or NET_ADMIN significantly reduces isolation. Only use when absolutely necessary for network debugging or specialized tools.

Best practice: pre-build custom images with dependencies installed rather than installing packages at runtime. This improves security, reproducibility, and startup time.

Network isolation and egress lockdown#

Platform isolation (always on, Cilium)#

A cluster-wide deny-only policy (k7-sandbox-platform-deny) stops every sandbox — in every egress mode, including open egress — from reaching the nodes, the Kubernetes API server, cloud metadata / link-local, and the kube-system / longhorn-system pods except CoreDNS. Nothing to configure; not available on --cni flannel.

Ingress isolation (Default: deny all, opt-in per sandbox)#

All inter-VM communication is blocked by default to prevent sandbox-to-sandbox access.

Key points:

  • Ingress blocking: VM sandboxes cannot communicate with each other unless a port is opened
  • Administrative access preserved: kubectl exec and k7 shell still work normally (they use the Kubernetes API, not pod networking)
  • System services allowed: Traffic from kube-system namespace is permitted for cluster functionality

Open ports with ingress_ports, scope the callers with ingress_from, and publish outside the cluster with expose_ports:

  • ingress_ports without ingress_from = reachable from other sandboxes in the same namespace only. Reaching a sandbox from anywhere requires "cidr:0.0.0.0/0", spelled out.
  • Every entry in expose_ports must also be in ingress_ports; the NodePort Service uses externalTrafficPolicy: Local so the pod sees the real client IP, and only answers on the node running the sandbox. The allocated NodePorts come back as node_ports on list / get.
  • ingress_from without ingress_ports, an unknown source prefix, or an expose_ports entry missing from ingress_ports is a 400.
Warning

On Cilium a cidr: peer is not evaluated for pod-to-pod traffic, so a rule whose sources are all cidr: opens the port to every sandbox in the cluster. Use cidr: for external clients (expose_ports) and sandbox: / namespace: to scope in-cluster access. See Networking.

Egress lockdown and whitelisting#

Use egress_whitelist to control outbound traffic. The policy is applied after the container becomes Ready so before_script runs with open egress.

Behavior:

  • Omit egress_whitelist: egress open (external internet allowed).
  • []: full egress block (no DNS resolution; no outbound IPs).
  • ["CIDR", ...]: allow only listed CIDR blocks; DNS is blocked.
  • ["api.example.com", "*.pypi.org", ...] (Cilium only): allow only those names. CoreDNS is reachable through Cilium’s DNS proxy, which answers only the allowlisted names; every other query gets NXDOMAIN, so the resolver cannot be used as an exfiltration channel.
  • network_limits (egressBps / ingressBps, bits/s) caps bandwidth per VM on kfd and kql; see Tenants.

Examples:

Full isolation (no inter-VM communication, no external access):

Partial isolation (no inter-VM communication, but external internet allowed):

Whitelist specific external services (avoid public DNS resolvers):

Info

Network Policy Details:

  • Ingress: Blocked by default (inter-VM isolation) - system services and kubectl exec still work
  • DNS: With [] or a CIDR-only allowlist, DNS resolution is blocked (no CoreDNS exception). With an FQDN allowlist, only the listed names resolve; the rest get NXDOMAIN
  • Administrative access: kubectl exec, k7 shell, and API operations bypass network policies
Warning

Do not whitelist public DNS resolver IPs (e.g., 1.1.1.1, 8.8.8.8). A CIDR allowlist has no L7/port filter, so those IPs re-enable outbound DNS (UDP/TCP 53) and DNS-over-HTTPS (443) and can be used for exfiltration. Prefer your own egress proxy IP, or use FQDN allowlists. Cilium is the default CNI, so egress_whitelist: ["api.openai.com", "*.huggingface.co"] works without extra setup. See Networking.

Mitigations when DNS is blocked#

  • Use an FQDN allowlist (Cilium): names resolve, everything else is NXDOMAIN
  • Use IP/CIDR whitelisting only (no domains post-lockdown)
  • Pre-resolve/fetch in before_script (runs before lockdown with open egress)

Roles, tenants and key scope#

Every credential has a role. read-only and user are granted per tenant — a Tenant claims namespaces, so a tenant key reaches every namespace of its tenant (narrowed further by the key’s optional -n list). operator and admin are cluster-wide. The API enforces this on every namespace-bearing endpoint:

  • The tenant’s namespaces only — another namespace returns 403 Forbidden before any cluster call. Fork and restore never leave their namespace.
  • All-namespaces list / GC (namespace omitted or all_namespaces=true) is denied for tenant keys.
  • Cluster-scoped reads (GET /api/v1/nodes/storage, tenants) need operator or admin; keys, role bindings and remote install need admin.
  • A node-scoped key (--node) may only create / restore / fork onto its listed nodes. Pair it with k7 nodes dedicate to keep other tenants off that node’s k7d daemon.
  • Creates, forks, restores and snapshots are checked against the tenant’s quotas (sandboxes, vcpu, memory, disk, snapshotStorage); over-quota is 403 QuotaExceeded.
  • In development mode a user key without a tenant is an unscoped key with full cross-namespace access; production mode refuses such keys.

Transport#

The API is served over HTTPS by default (Caddy sidecar, cluster CA at /etc/k7/tls/ca.crt unless --api-hostname or an operator certificate was used at install). Operators can additionally restrict the NodePort to known client CIDRs with k7 install --api-allow-cidr (Cilium only). Neither is rate limiting. See Security model.

Audit log#

Every mutating request (POST/PUT/PATCH/DELETE) and every 401/403 produces one JSON line on the k7-api pod’s stdout (k7.audit logger): request id, principal type (key / oidc), subject, role, tenant, key id and name, route template, namespace, sandbox, status, latency and client IP. Tokens, emails, groups, bodies and exec command text are never logged (exec is sha256 + length). An audit line that cannot be emitted fails the request closed (500). Ship stdout with your log collector.

docker: true#

Turning on Docker changes what bounds the guest: on k7d / k7d-fc the sandbox container stays unprivileged but docker.sock is guest root; on kfd / kql a privileged docker-vehicle container runs in the same Kata VM. The isolation boundary is the VM either way — use k7d-fc when tenants do not trust each other. See Docker in a sandbox.

Image registry allowlist (SSRF)#

When the control plane inspects a sandbox image (entrypoint/cmd resolution), the registry host is resolved before any fetch. Hosts must be on the allowlist (registry-1.docker.io, ghcr.io, quay.io, public.ecr.aws, plus K7_REGISTRY_ALLOWLIST) and every A/AAAA must be a public unicast address. Loopback, private, link-local, metadata, and reserved ranges are rejected. Redirects are not followed.