Observability and metering

Since 0.5.0, k7-api and k7-agent export Prometheus metrics, log JSON lines, and measure what every tenant and sandbox uses. K7 ships Grafana dashboards, alert rules with a runbook per alert, and k7 install --monitoring can install a monitoring stack for a single-box install. Nothing is priced: rating and invoices belong to whatever consumes the usage events.

Metrics#

Both processes serve /metrics (and nothing else) as plain HTTP on port 9464. Only the pods you name may reach that port:

On Cilium this is the CiliumNetworkPolicy k7-metrics (host and other nodes are refused too); other CNIs get NetworkPolicy objects k7-api-metrics / k7-agent-metrics.

k7-api, each replica: k7_http_requests_total / k7_http_request_duration_seconds (route template, method, status), k7_operation_duration_seconds / k7_operation_errors_total (op, backend, outcome / error_code), k7_auth_failures_total (reason: missing, invalid, expired, forbidden, oidc_invalid), k7_key_store_last_sync_timestamp_seconds, k7_kube_api_errors_total, k7_quota_rejections_total (tenant, dimension), k7_sandboxes (tenant, backend, state; leader only, so sum() is the cluster count), k7_api_info (version, mode), k7_tls_cert_expiry_timestamp_seconds.

k7-agent, each node: k7d_rpc_duration_seconds / k7d_rpc_errors_total (op), k7d_daemon_up, k7_storage_pool_bytes (pool, kind), k7_storage_report_age_seconds, k7_vm_count (backend, state), plus the usage counters below.

No series carries a sandbox name, uid or pod (a unit test enforces this): 2 replicas, 20 nodes and 500 tenants come to at most ~17k series. Per-sandbox detail is in k7 top and the usage events.

Logs and request ids#

k7-api, k7-agent and the snapshot-gc Job write one JSON object per line on stderr (K7_LOG_FORMAT / K7_LOG_LEVEL, set by the manifests; chart observability.logLevel): ts, level, logger, msg, plus request_id, tenant, namespace, sandbox, backend, op, duration_ms, error_code. Tokens and JWTs are replaced with [REDACTED]; no level logs request bodies or exec output.

k7-api accepts a well-formed X-Request-ID (8–64 chars of [A-Za-z0-9-]) or mints a UUIDv7, returns it in the response and sends it to the k7-agent that runs the VM op, so one id follows a request end to end. Per request you get one access line, an audit line for mutating requests (k7.audit, see Security & networking) and a k7.op summary per lifecycle op. A line is 400–600 bytes; K3s keeps 10 MiB × 5 files per container, so ship them with your collector or raise the kubelet log limits.

Dashboards, alerts, runbooks#

Sources live in src/k7/deploy/observability/: dashboards API overview, Sandbox lifecycle, Nodes and storage, Usage by tenant; alert rules in rules/k7.rules.yaml with thresholds as k7:threshold:* recording rules (promtool test rules covers one firing and one quiet case per alert). Alerts: K7APIDown, K7APIHighErrorRate, K7OperationSlow, K7KeyStoreStale, K7AgentDown, K7dDaemonDown, K7StoragePoolFilling, K7StorageReportStale, K7SnapshotGCFailing, K7UsageSpoolOverflow, K7UsageSinkStopped, K7TLSCertExpiringSoon. Every alert links to docs/runbooks/<alert>.md, which starts with the first three commands to run.

Chart values: observability.serviceMonitor (Service k7-api-metrics, ServiceMonitor and PodMonitor), observability.prometheusRule, observability.grafanaDashboards, observability.monitorLabels, observability.thresholds.*.

k7 install --monitoring#

Opt-in and meant for single-box installs (existing clusters keep their own monitoring and use the chart values). It installs on the first master the pinned, checksummed kube-prometheus-stack 92.3.0 in namespace monitoring with 7 days of retention, the K7 ServiceMonitor/PodMonitor, the PrometheusRule k7, the dashboard ConfigMaps, a /metrics policy admitting that Prometheus, and sets metering.prometheusUrl so k7 usage works. --metrics-scraper cannot be combined with it; --no-api is refused.

kubectl -n monitoring get secret kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echokubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Re-running k7 install without --monitoring leaves the stack but reverts the /metrics policy to --metrics-scraper (by default nobody), so its K7 targets go down. Removal steps are in docs/OBSERVABILITY.md.

Metering#

• Reserved, not used. vCPU and memory are the container limits stamped on the Deployment at create / fork / restore. Actual CPU and memory stay in k7 top. Size class is <cpu>c-<memGi>g, or unsized for a sandbox without both limits.
• States: running accrues everything; paused (a k7d VM frozen in place) accrues seconds, GiB·s and disk but 0 vCPU·s; stopped (kql/kfd scaled to 0) accrues disk only.
• Node sampler: every sampleInterval seconds (5–60; 15 in development, explicit in production) k7-agent lists the node’s ready sandbox pods with crictl and their eth0 txBytes. Windows are closed on the monotonic clock; k7d pause/resume transitions are recorded and split windows exactly; a pod deleted between samples is charged to the midpoint and flagged approximate. Time no sampler saw becomes a gap event (metering-start, agent-restart, leader-change) — nothing is extrapolated, and no window is emitted twice.
• Storage sampler: the k7-api replica holding Lease k7-metering-leader runs every storageInterval seconds (default 60) over PVCs, ephemeral claims, the 10 GB kfd devmapper root, and every VolumeSnapshot. Not metered yet because their sources are not in K7 yet: k7d writable disks (reported in /health as disk_unavailable, never as 0) and k7d-native snapshots.
• Events are CloudEvents 1.0 JSON (io.katakate.k7.sandbox.usage.v1, .sandbox.usage.gap.v1, .egress.usage.v1, .disk.usage.v1, .snapshot.usage.v1) with a deterministic id = sha256(cluster-id, uid, meter, window_start), so a replay after a crash deduplicates downstream. The JSON Schemas (src/k7/metering/schemas/, v1 frozen) are the contract. cluster-id lives in ConfigMap k7-cluster-id, written once at install — keep it in your backups.

Spool and delivery#

Events are appended to /var/lib/k7/usage-spool/ on each node as fsynced JSON-lines segments; each sink keeps its own offset, so delivery is at least once. The spool is capped (--metering-spool-size, 1 GiB): past it the oldest segment is dropped, k7_usage_events_dropped_total grows, k7-agent /health reports degraded and K7UsageSpoolOverflow fires. k7 nodes storage shows each node’s spool fill. Sandboxes are never blocked by metering unless you ask: --metering-require-delivery makes k7-api steer creates off nodes whose spool is above --metering-spool-high-water (80 %) and refuse them with 503 UsageSpoolFull when no node qualifies.

Sinks#

Prometheus is always on: k7_usage_sandbox_seconds_total, k7_usage_vcpu_seconds_total, k7_usage_memory_gib_seconds_total, k7_usage_egress_bytes_total (tenant × namespace × backend × state) from k7-agent, and the gauges k7_usage_disk_bytes / k7_usage_snapshot_bytes from the k7-api leader. Opt-in sinks are a YAML list (--metering-sinks-file, chart metering.sinks):

- name: openmeter  type: http  url: http://openmeter.openmeter.svc:8888  preset: openmeter          # POST <url>/api/v1/events, drops null data keys  mode: batch                # application/cloudevents-batch+json  batchSize: 100  headersSecretRef: {name: k7-metering-headers, key: openmeter}- name: logs  type: stdout               # one event per line on the k7.usage logger

http backs off on transport errors, 429 and 5xx while the spool keeps the events; any other 4xx stops that sink (k7_usage_sink_stopped, alert K7UsageSinkStopped) until you fix the cause and rollout restart ds/k7-agent. Auth headers come from Secret k7-metering-headers in the system namespace, read on every request.

Configuration#

k7 install and the chart render the same document into ConfigMap k7-metering; every key is validated at startup and a change rolls both pods.

Showback#

k7 usage --from 7d --group-by tenant,statek7 usage --tenant acme --from 2026-01-01T00:00:00Z --to 2026-02-01T00:00:00Z --json

k7 usage and GET /api/v1/usage read Prometheus at metering.prometheusUrl (400 naming the key when unset). K7 keeps no usage database. Counters are summed exactly from raw samples, so the result matches the events to within one scrape interval per series inside Prometheus retention (7 days with --monitoring); disk and snapshot GiB·s integrate the gauges at scrape resolution. A tenant key sees only its own tenant. For anything billable, consume the events.

Reference: docs/OBSERVABILITY.md, docs/METERING.md. Not yet: tracing (request ids are the correlation mechanism), rate limiting (k7_rate_limited_total stays at zero).