Skip to content

architecture.md

Latest commit

 

History

History
210 lines (161 loc) · 14.8 KB

architecture.md

File metadata and controls

210 lines (161 loc) · 14.8 KB
Raw

Architecture

Devboxes is a namespaced Kubernetes application with three independently consumable parts: the controller and dashboard image, the workspace image, and the Rust CLI.

Request and tenant resolution path

Automation may authenticate with the master or a configured principal bearer token. Interactive devbox login uses the native-app Authorization Code flow with PKCE S256: the CLI binds a numeric loopback callback, opens the external browser, validates state, receives a short-lived one-time code, and exchanges it for a scoped CLI bearer token. The browser exchanges the submitted operator or principal token for a signed, HTTP-only SameSite session plus a separate CSRF token. The controller never sends the submitted access token back to either client.

Authorization codes live only in a bounded in-memory store. The controller retains a SHA-256 digest, client ID, exact loopback redirect URI, PKCE challenge, subject, and expiry. Codes expire after about two minutes, are consumed atomically once, and are pruned during store operations. Approval and denial require the browser session's form CSRF token.

CLI tokens are HMAC-signed JWTs with fixed issuer, audience, type, scope, subject, issued time, expiry, and token ID claims. The default signing key is derived from the master token with explicit domain separation; an optional dedicated signing key can be supplied from the existing controller Secret. Rotating the effective signing key revokes all issued CLI tokens. There are no refresh tokens.

Multi-tenancy adds an identity and policy resolution step before application work:

master, principal, CLI, or browser credential
                     |
                     v
              live identity lookup
                     |
          X-Devboxes-Tenant or identity default
                     |
                     v
      membership + role authorization (one tenant)
                     |
                     v
 tenant namespace + effective policy + scoped operation

Principal bearer tokens are read from an existing Kubernetes Secret. Browser sessions and CLI tokens contain a stable principal subject and a credential epoch, not an immutable role snapshot. Every request resolves current memberships, so a configuration rollout can remove access without waiting for a CLI token to expire. Unknown and unauthorized tenant IDs both return the same forbidden response.

The master token resolves to an installation operator with access to every configured tenant. When tenancy is disabled, it resolves to a synthetic default tenant backed by the release namespace, preserving the original API and resource behavior.

The controller translates lifecycle operations into native Kubernetes resources in the selected tenant namespace:

  • One Deployment per devbox for disposable compute.
  • One Service per devbox for SSH through LoadBalancer or NodePort.
  • One PersistentVolumeClaim per devbox, mounted at /home/dev.
  • When Insights is enabled, one scoped ingest Secret per devbox and one central Insights PersistentVolumeClaim for the controller.

Resource names are deterministic (devbox-NAME) inside a namespace. Labels and annotations carry controller ownership, tenant, creator, creation time, expiry, preset, repository, retained storage size, and any resolved GPU or custom image allocation.

Namespace isolation

The Helm release keeps one controller ServiceAccount in the release namespace. With tenancy enabled, Helm binds that account to an identical namespaced Role in each tenant namespace. It does not receive a ClusterRole. Each tenant gets a separate tokenless workspace ServiceAccount; workspaces cannot use it to call the Kubernetes API.

Optional ResourceQuota objects provide an aggregate namespace ceiling. Tenant TTL, presets, GPU profiles, custom images, workspace Secret, ServiceAccount, and StorageClass are resolved in application policy before Kubernetes admission. For per-person isolation, an operator creates one tenant per person; a team tenant deliberately shares its inventory and quota.

The chart's optional NetworkPolicy selects only managed tenant workspace pods, permits the internal SSH target port TCP 2222, and leaves egress open. The SSH Service may expose port 22. This blocks other inbound pod ports without breaking SSH, repository access, package registries, provider APIs, or Insights forwarding. NetworkPolicy enforcement depends on the cluster CNI.

GPU resolution path

GPU acceleration adds an operator-owned policy layer without changing resource ownership:

Helm GPU profiles
       |
       v
validated controller settings
       |
       +---- authenticated capability catalog ----> CLI and dashboard
       |
user selects profile name
       |
       v
resolved pinned snapshot
       |
       v
Deployment pod template ----> scheduler ----> device plugin or DRA-bridged resource

The create API accepts only an optional profile name. The controller resolves that name before any Kubernetes write, then applies the trusted image override, RuntimeClass, supplemental groups, node selector, tolerations, and extended resource count. It writes the extended resource to both requests and limits on the main container. Insights and other sidecars do not receive GPU resources. The Insights sidecar also remains on the installation's release workspace image, preserving its pinned privacy sanitizer when the interactive container uses a specialized GPU image.

The resolved profile is stored as a bounded Deployment annotation. Existing boxes therefore retain their allocation across stop, start, TTL expiry, and template reconciliation even if the Helm catalog later changes. Capability discovery publishes only profile names, labels, descriptions, resources, and counts. Scheduling details and images remain operator policy.

Custom image resolution path

Custom images use the same operator-owned catalog pattern while preserving the Devboxes workspace contract:

Helm custom image profiles
       |
       v
validated controller settings
       |
       +---- authenticated capability catalog ----> CLI and dashboard
       |
user selects profile name or exact approved reference
       |
       v
resolved pinned snapshot
       |
       +---- sidecar profile ----> credential-free custom-image container
       |
       +---- workspace profile --> verified Devboxes-compatible main container

The controller resolves a selector before it creates a PVC, Deployment, or SSH Service. A default sidecar profile runs a compatible non-root service image in the same pod network namespace as the prepared workspace. It has bounded configured resources and optional high pod-local ports, but no Devboxes Secret mount, home PVC mount, Kubernetes API token, public Service, extra capability, command override, or scheduling policy. runAsNonRoot is enforced, privilege escalation is disabled, and all capabilities are dropped. The SSH workspace remains the only interactive and credential-bearing container.

A workspace profile deliberately replaces the main interactive image and is restricted to a compatible Devboxes-derived image. The controller prevents it from competing with a GPU profile that independently selects workspaceImage. The complete resolved profile is stored in a Deployment annotation, so stop, start, TTL expiry, and Insights template reconciliation retain the original image policy even if Helm values later change. Capability discovery exposes only profile labels, modes, descriptions, and declared ports.

Persistence model

Disconnecting SSH leaves the pod and tmux session running. Stopping scales the Deployment to zero, which ends processes but leaves the PVC. Deleting removes the Deployment and Service while retaining the PVC by default. Purging explicitly deletes the PVC.

The SSH host key lives on the persistent volume. Recreating a previously deleted devbox with the same name therefore retains both files and host identity. A purge intentionally creates a new identity.

TTL expiry is equivalent to stop: it scales compute to zero and never deletes data. Starting a stopped devbox renews its original TTL from the new start time.

When Insights is enabled, persistence has two additional layers. A bounded SQLite outbox lives on each workspace home PVC, so accepted local metric batches survive workspace and controller outages. The controller stores sanitized, deduplicated points, Git aggregates, collector health, and time rollups in a separate central SQLite PVC. Ordinary workspace deletion and home purge do not remove central history. Insights history has its own explicit purge operation.

A UUID instance identity is stored on both the Deployment and home PVC. Retaining and reusing the PVC preserves that identity. Purging the PVC and recreating the box creates a new identity. This separates a box name from the lifetime of the storage that produced its data.

Every central Insights record also carries a tenant ID. Reads, rollups, collector status, exports, and purge operations require that tenant scope. A SQLite export is produced through the online backup API and then deletes all rows owned by other tenants before it leaves the controller. Legacy rows are assigned to the configured default tenant during the database migration.

Readiness

A devbox becomes ready only when its workspace pod reports Ready and its SSH Service has a usable endpoint. LoadBalancer uses the first published IP or hostname, with an optional configured fallback host. NodePort combines the configured node host with the port allocated by Kubernetes.

The workspace entrypoint refuses to start without SSH_AUTHORIZED_KEYS. It prepares the persistent home, host key, optional credentials, and optional repository clone before starting OpenSSH. The controller's TCP readiness probe therefore represents a usable SSH daemon, not merely a scheduled pod.

Security boundaries

  • With tenancy disabled, controller RBAC is a Role scoped to the release namespace.
  • With tenancy enabled, the controller has one RoleBinding in each explicitly configured tenant namespace, never a ClusterRole.
  • Workspace service accounts have no RBAC binding and do not mount Kubernetes API tokens.
  • Principal credentials are loaded from a read-only Secret mount, compared without exposing token values, and constrained by live tenant memberships and application roles.
  • Tenant clients cannot select a Kubernetes namespace, Secret, ServiceAccount, StorageClass, quota, preset outside their allowlist, GPU scheduling policy, or custom-image policy.
  • GPU clients can choose only a configured profile name. They cannot inject images, device resources, RuntimeClasses, supplemental groups, selectors, tolerations, privileged mode, host paths, or device paths.
  • Custom-image clients can choose only a configured profile name or exact configured image reference. They cannot inject a registry, command, Service, volume, port mapping, resource request, capability, host path, ServiceAccount, or scheduling field. Sidecar containers drop all Linux capabilities and cannot mount Devboxes credentials or persistent storage.
  • Workspace Secrets are mounted read-only with mode 0440, scoped to the workspace group, and are not embedded in either image.
  • Each Insights ingest credential is HMAC-signed, write-only, scoped to one box and UUID instance, and stored in a dedicated namespaced Secret. It is never a controller, browser, or CLI credential.
  • The controller runs as a non-root user with a read-only root filesystem and all Linux capabilities dropped.
  • The workspace runs as root during initialization, then exposes only the unprivileged dev SSH user. Password login and root login are disabled.
  • The trusted dev user has passwordless sudo. The pod adds only AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID, and SYS_CHROOT; AUDIT_WRITE lets OpenSSH allocate audited PTYs, while SYS_ADMIN and privileged mode are not used.
  • SSH host checking uses a stable alias scoped to the Devboxes installation, tenant, and box name, preventing collisions across installations or same-named cross-tenant boxes.
  • CLI callbacks accept only exact HTTP loopback URIs with numeric loopback hosts, explicit non-privileged ports, and /callback; login return targets accept only the internal authorization route.

The master controller token is an operator credential. Anyone holding it can create, stop, delete, or purge every devbox in every tenant. A scoped CLI token expires, cannot mint browser sessions or new CLI tokens, and retains the live identity of the principal that authorized it. viewer is read-only, member can manage ordinary lifecycle state, and admin can also purge retained home or Insights data through the application. SSH remains separate: a viewer whose key is present in the tenant workspace Secret can still change files inside a running workspace.

Scheduling and storage

Presets specify requests and memory limits while intentionally leaving CPU burstable:

Preset CPU request Memory request Memory limit PVC request
small 250m 512Mi 4Gi 20Gi
medium 750m 2Gi 8Gi 30Gi
large 2 4Gi 16Gi 50Gi

A retained PVC is expanded when a larger preset is requested, subject to the StorageClass supporting expansion. PVCs are never shrunk.

GPU profiles add one vendor-qualified extended resource with the same integer value in requests and limits. Kubernetes extended resources are not overcommitted unless the installed device plugin intentionally advertises shared units. Optional selectors and tolerations direct boxes to operator-prepared GPU pools; an optional RuntimeClass activates a non-default vendor runtime.

The controller does not guess capacity or reserve a device before creation. Kubernetes scheduling is the source of truth. A box stays starting when capacity or constraints cannot be satisfied, and the controller exposes the PodScheduled=False reason to the CLI and dashboard. Device plugins remain responsible for allocation and device injection.

Availability

With Insights disabled, the controller is stateless apart from signed sessions and the short-lived in-memory authorization-code store. Restarting it cancels pending CLI approvals but does not affect issued tokens or devbox state.

With Insights enabled, the controller also owns a stateful SQLite database. The chart requires one controller replica and uses a Recreate strategy so one pod owns the database volume at a time. Kubernetes remains the source of truth for workspace lifecycle, while the Insights database is the source of truth for retained telemetry and aggregate activity.

Insights does not restart active legacy workspaces during an upgrade. They expose restart_required until a normal stop and start installs the sidecar and its current scoped credential. Stopped workspaces are reconciled in place, and every start reconciles the template before compute is scaled up.