Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,49 @@ smolvm machine run --ssh-agent --net --image alpine -- sh -c "apk add -q openssh
smolvm machine exec --name myvm -- git clone git@github.com:org/private-repo.git
```

**Run guest GUI apps on your host desktop over vsock.** `--waypipe` bridges a guest [waypipe](https://gitlab.freedesktop.org/mstoeckl/waypipe) vsock port to a host Unix socket in the VM data dir — no X11, no SSH server, no TCP port forward. The guest agent runs `waypipe server` as a daemon inside the workload container and exports `WAYLAND_DISPLAY` automatically, and on the host smolvm starts the matching `waypipe client` next to your Wayland compositor for you (Linux hosts). So you just run your GUI app — both ends are wired up. One daemon serves every app (like a normal Wayland display), and it starts on first launch — no per-app `waypipe server` wrapper.

```bash
smolvm machine create --name gui --net --waypipe --image ubuntu:24.04
smolvm machine start --name gui

# Guest: run any GUI app. WAYLAND_DISPLAY is set, the guest daemon is started,
# and the host waypipe client is running against your compositor.
smolvm machine exec --name gui -- weston-terminal
```

The host client is started automatically on Linux when `$WAYLAND_DISPLAY` is set and `waypipe` is on the host `PATH`; it lives as long as the VM and is killed with it. If either is missing, smolvm skips it and you can run one by hand:

```bash
waypipe -s "$(smolvm machine data-dir --name gui)/waypipe.sock" client &
```

`--waypipe` takes an optional value selecting which `waypipe` binary the guest daemon runs:

- `--waypipe` or `--waypipe=host` (default) — share the **host** waypipe binary into the guest, so the guest server and your host client are the exact same binary (no wire-version drift) and the image needs no waypipe installed. Requires the host glibc to be compatible with the guest image's (usually true for a recent image).
- `--waypipe=container` — use the image's **own** `waypipe` (install it yourself, e.g. `apt-get install -y waypipe`). The daemon starts on the first launch after waypipe is present, with no restart.
- `--waypipe=/path/to/waypipe` — share that specific host binary.

Requires a `--vsock`-capable waypipe (>= 0.9) on the host, and (for `container`) in the guest. `--waypipe` needs an `--image`: the guest daemon runs inside the workload container (the agent's own rootfs is musl and cannot exec a glibc waypipe), which a bare VM does not have — so `--waypipe` without an image is rejected up front.

One waypipe daemon in the guest serves every app (like the X11 display socket), and it starts lazily — the first launch after you install waypipe brings it up with no VM restart. The bridge uses the same vsock mechanism as `--cuda`, so no networking is required for forwarding itself (`--net` is only needed to install waypipe in the guest).

If the daemon can't start, smolvm says so rather than failing silently: `machine exec -- cmd` prints the reason on stderr (without disturbing the command's own output), and an interactive `machine exec -it` shell prints it at the top of the session. Either way the message says whether waypipe isn't installed yet (with how to install it — the daemon then comes up on the next command) or is present but failed to start (e.g. a host/guest glibc mismatch in `host` mode — try `--waypipe=container`).

**Or bridge the raw X11 socket, no waypipe.** `--x11` resolves the host `$DISPLAY` at launch and bridges a guest vsock port straight to that X server's Unix socket. X was designed for network transparency, so guest X clients talk to your host X server directly. The guest agent sets up the display socket and exports `DISPLAY=:10` for you — no `socat`, just run an X client. Needs a running host X server (a native X session, or an Xwayland/`Xephyr` on a Wayland host).

```bash
smolvm machine create --name xgui --net --x11 --image ubuntu:24.04
smolvm machine start --name xgui # start with $DISPLAY set

xhost +local: # allow the bridged connections

# DISPLAY=:10 is already set in the VM — just run an X client.
smolvm machine exec --name xgui -- sh -c 'apt-get install -y x11-apps && xeyes'
```

The X11 bridge is a plain byte pipe (guest connects out to host CID 2, port 7002), so it cannot pass `SCM_RIGHTS` ancillary fds — MIT-SHM and DRI3 fall back to wire-image transport (correct, just slower). For per-window Wayland integration and correct fd/GPU handling, prefer `--waypipe`.

**Declare environments with a Smolfile** — reproducible VM config in a simple TOML file.

```toml
Expand Down Expand Up @@ -145,6 +188,8 @@ smolvm strengthens the guest/host boundary by giving each workload a separate VM
* The `smolvm` CLI and VMM processes run with the permissions of the invoking host user. That user account, the host OS, the hypervisor backend, libkrun, and smolvm are in the trusted computing base.
* Host directories passed with `--volume` are intentionally exposed to the guest with the requested access. Do not mount secrets or sensitive paths into an untrusted workload.
* `--ssh-agent` does not copy private key material into the guest, but it grants the guest access to the forwarded agent socket and therefore the ability to request signatures while the VM is running.
* `--waypipe` opens a vsock channel from the guest to a host Unix socket that a `waypipe client` reads next to your Wayland compositor. A guest with this enabled can drive that client; only enable it for workloads whose GUI you intend to display.
* `--x11` bridges a guest vsock port straight to your host X server socket, giving the guest a direct connection to that X server. X11 has weak client isolation, so treat a guest with `--x11` as having access to the whole X server (input, other windows, clipboard); only enable it for trusted GUI workloads, and rely on X access control (`xhost`) deliberately.
* Networking is disabled by default. Enabling `--net`, port forwarding, or host services expands the workload's reachable surface.
* In standalone local use, smolvm's state and control endpoints are scoped to the invoking user's environment. For hostile local co-tenants, add host-level account separation and OS confinement around the VMM process. This section does not describe the separate smolmachines cloud control plane or its tenant-isolation guarantees.
* Release archives publish SHA-256 checksums and the installer rejects a mismatch when the checksum file is available. Releases are not currently signed or accompanied by provenance attestations, and the installer permits installation when the checksum file cannot be downloaded.
Expand Down
118 changes: 106 additions & 12 deletions crates/smolvm-agent/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ mod storage;
#[cfg(target_os = "linux")]
mod timesync;
mod vsock;
mod waypipe;
mod x11;

// ============================================================================
// Configuration Constants
Expand Down Expand Up @@ -420,6 +422,26 @@ fn main() {
// `--mount-socket`). No-op when none are configured.
publish_socket::start_all();

// Start the raw X11 socket bridge if enabled by host: the guest binds a
// local display socket (:10) and relays each connection out to the host X
// server over vsock, so guest X clients render on the host X server. Set
// DISPLAY so all child processes (and the agent's own workloads) find it.
if x11::is_enabled() {
info!("X11 socket bridge enabled, starting guest bridge");
x11::start();
std::env::set_var("DISPLAY", x11::GUEST_DISPLAY);
}

// Waypipe Wayland forwarding runs its daemon INSIDE the workload container
// (the agent rootfs is musl and cannot exec a glibc waypipe). The only
// boot-time work is mounting the host-shared waypipe binary (host/path mode)
// so it can be bind-mounted into the container; the daemon itself is started
// once the keep-alive container is up (see `handle_run_detached`).
if waypipe::is_enabled() {
info!("waypipe Wayland forwarding enabled");
waypipe::mount_shared_binary_at_boot();
}

// Mount the Rosetta 2 runtime and register the binfmt_misc handler if the
// host attached it. Must run after pivot_root (the wrapper lives in the
// rootfs) and after /proc is mounted (binfmt_misc registration).
Expand Down Expand Up @@ -3024,7 +3046,7 @@ fn handle_interactive_run(
};

// Spawn the command with crun
let (mut child, pty_master) = match spawn_interactive_command(
let (mut child, pty_master, waypipe_warning) = match spawn_interactive_command(
&prepared.rootfs_path,
&launch,
&mounts,
Expand All @@ -3046,6 +3068,20 @@ fn handle_interactive_run(
// Send Started response
send_response(stream, &AgentResponse::Started)?;

// If the waypipe daemon could not start, print the reason to the user's
// terminal before the session begins so forwarding never fails silently.
// Sent as a Stdout frame (the interactive protocol has no separate stderr
// channel); the message already carries the `smolvm:` prefix. CRLF, not LF:
// a raw TTY does no newline translation, so a bare LF would stair-step.
// No-op unless forwarding is enabled and did not come up. (Non-interactive
// exec surfaces the same text on the command stderr.)
if let Some(warning) = waypipe_warning {
let line = format!("\r\n{warning}\r\n");
let _ = send_response(stream, &AgentResponse::Stdout {
data: line.into_bytes(),
});
}

// Run the appropriate interactive I/O loop
let exit_code = match pty_master {
#[cfg(target_os = "linux")]
Expand Down Expand Up @@ -3213,6 +3249,8 @@ fn write_oci_bundle(
storage::add_storage_fallback(&mut spec, mounts, unprivileged);

ssh_agent::inject_into_container(&mut spec);
x11::inject_into_container(&mut spec);
waypipe::inject_into_container(&mut spec);
rosetta::inject_into_container(&mut spec);
cuda::inject_into_container(&mut spec, rootfs_path);
spec.write_to(bundle_path)
Expand Down Expand Up @@ -3430,6 +3468,7 @@ fn handle_run_detached(
container_id = %container_id,
"detached container started via create+start"
);

send_response(
stream,
&AgentResponse::Completed {
Expand Down Expand Up @@ -3556,8 +3595,19 @@ fn spawn_exec_in_container(

// An exec joining a running container inherits the same image-resolved env /
// workdir as the container's main process.
//
// The container's config.json got SSH_AUTH_SOCK / DISPLAY via the spec
// injection in `write_oci_bundle`, but `crun exec` builds a fresh process env
// from what we pass here, NOT the container's - so those vars have to be
// re-injected onto the exec env or an interactive `-it` join (this path)
// silently loses them. Mirrors the injection `handle_run` does for the
// non-interactive keep-alive exec path (#542).
let mut env: Vec<(String, String)> = launch.env.clone();
ssh_agent::inject_into_env(&mut env);
x11::inject_into_env(&mut env);
waypipe::inject_into_env(&mut env);
let command: &[String] = &launch.command;
let env: &[(String, String)] = &launch.env;
let env: &[(String, String)] = &env;
let workdir: Option<&str> = launch.workdir.as_deref();

info!(
Expand Down Expand Up @@ -3787,6 +3837,10 @@ fn ensure_main_container(
Ok(container_id)
}

/// Spawn the interactive command, returning the child, its PTY master (when a
/// TTY was requested), and an optional one-line warning to print to the user's
/// terminal before the session starts (currently the waypipe daemon-start
/// outcome, `None` unless forwarding is enabled and did not come up).
#[cfg(target_os = "linux")]
#[allow(clippy::too_many_arguments)]
fn spawn_interactive_command(
Expand All @@ -3796,7 +3850,7 @@ fn spawn_interactive_command(
tty: bool,
persistent_overlay_id: Option<&str>,
unprivileged: bool,
) -> Result<(Child, Option<pty::PtyMaster>), Box<dyn std::error::Error>> {
) -> Result<(Child, Option<pty::PtyMaster>, Option<String>), Box<dyn std::error::Error>> {
use std::path::Path;

if launch.command.is_empty() {
Expand All @@ -3818,7 +3872,14 @@ fn spawn_interactive_command(

// If a main workload container is running for this overlay, join it.
if let Some(cid) = resolve_main_container(persistent_overlay_id) {
return spawn_exec_in_container(&cid, launch, tty);
// Ensure the waypipe daemon is up in this container (idempotent; no-op
// unless forwarding is enabled). Covers the case where the container
// persisted from an earlier exec but the daemon has not been started.
// The outcome's warning (if any) is returned so the caller can print it
// to the user's terminal before the session starts.
let warning = waypipe::start_daemon_in_container(&cid).user_warning();
let (child, pty) = spawn_exec_in_container(&cid, launch, tty)?;
return Ok((child, pty, warning));
}

// On a persistent machine with no main container yet, establish a long-lived
Expand All @@ -3832,7 +3893,14 @@ fn spawn_interactive_command(
// so exec never breaks outright.
if let Some(overlay_id) = persistent_overlay_id {
match ensure_main_container(rootfs, overlay_id, mounts, unprivileged, launch) {
Ok(cid) => return spawn_exec_in_container(&cid, launch, tty),
Ok(cid) => {
// Start the waypipe daemon in the freshly-established keep-alive
// container (idempotent; no-op unless forwarding is enabled).
// The warning (if any) is returned to the caller to print.
let warning = waypipe::start_daemon_in_container(&cid).user_warning();
let (child, pty) = spawn_exec_in_container(&cid, launch, tty)?;
return Ok((child, pty, warning));
}
Err(e) => {
warn!(error = %e, "keep-alive main container setup failed; running in a fresh container")
}
Expand Down Expand Up @@ -3876,8 +3944,11 @@ fn spawn_interactive_command(
);

// The single-container `Run` path keeps cgroups disabled (its VM is the
// limit); per-container cgroups are a pod-only concern.
spawn_crun_run(&bundle_path, &container_id, tty, false)
// limit); per-container cgroups are a pod-only concern. The fresh-container
// path does not start the waypipe daemon (only the persistent keep-alive
// paths do), so there is no warning to carry here.
let (child, pty) = spawn_crun_run(&bundle_path, &container_id, tty, false)?;
Ok((child, pty, None))
}

/// Launch a container with `crun run` and hand back the child plus the PTY
Expand Down Expand Up @@ -3990,7 +4061,7 @@ fn spawn_interactive_command(
_tty: bool,
_persistent_overlay_id: Option<&str>,
unprivileged: bool,
) -> Result<(Child, Option<()>), Box<dyn std::error::Error>> {
) -> Result<(Child, Option<()>, Option<String>), Box<dyn std::error::Error>> {
use std::path::Path;

let command: &[String] = &launch.command;
Expand Down Expand Up @@ -4031,6 +4102,8 @@ fn spawn_interactive_command(

// Forward SSH agent into the container if enabled at boot.
ssh_agent::inject_into_container(&mut spec);
x11::inject_into_container(&mut spec);
waypipe::inject_into_container(&mut spec);
rosetta::inject_into_container(&mut spec);
cuda::inject_into_container(&mut spec, rootfs_path);

Expand All @@ -4044,7 +4117,7 @@ fn spawn_interactive_command(
.capture_output()
.spawn()?;

Ok((child, None))
Ok((child, None, None))
}

/// Run the interactive I/O loop using poll() for efficient I/O multiplexing.
Expand Down Expand Up @@ -5042,6 +5115,13 @@ fn run_in_keepalive_container(
}
};

// Ensure the waypipe daemon is up in the keep-alive container before the
// workload runs (idempotent; no-op unless forwarding is enabled). If it
// could not come up - waypipe not installed yet, or present but broken - the
// reason is surfaced on this exec's stderr below so forwarding never fails
// silently. Only a running daemon is silent.
let waypipe_warning = waypipe::start_daemon_in_container(&cid).user_warning();

// The workload runs via `crun exec --user`, which requires a NUMERIC uid[:gid]
// — a username (the image's `config.User`, e.g. `nobody`/`node`, or the
// request user) is rejected with "invalid USERSPEC specified". Resolve it
Expand Down Expand Up @@ -5087,11 +5167,23 @@ fn run_in_keepalive_container(
},
)?;

// Prepend the waypipe warning (if any) to the command's stderr so the user
// sees why forwarding did not come up, without corrupting stdout.
let prepend_waypipe_warning = |stderr: Vec<u8>| -> Vec<u8> {
if let Some(warning) = &waypipe_warning {
let mut prefixed = format!("{warning}\n").into_bytes();
prefixed.extend_from_slice(&stderr);
prefixed
} else {
stderr
}
};

Ok(match result {
crate::process::WaitResult::Completed { exit_code, output } => AgentResponse::Completed {
exit_code,
stdout: output.stdout,
stderr: output.stderr,
stderr: prepend_waypipe_warning(output.stderr),
},
crate::process::WaitResult::TimedOut { output, timeout_ms } => {
let mut stderr = output.stderr;
Expand All @@ -5101,7 +5193,7 @@ fn run_in_keepalive_container(
AgentResponse::Completed {
exit_code: crate::process::TIMEOUT_EXIT_CODE,
stdout: output.stdout,
stderr,
stderr: prepend_waypipe_warning(stderr),
}
}
crate::process::WaitResult::ClientDisconnected { output } => {
Expand All @@ -5110,7 +5202,7 @@ fn run_in_keepalive_container(
AgentResponse::Completed {
exit_code: 137,
stdout: output.stdout,
stderr,
stderr: prepend_waypipe_warning(stderr),
}
}
})
Expand All @@ -5137,6 +5229,8 @@ fn handle_run(
// is off; harmless on the fresh-container path.
let mut env = env.to_vec();
ssh_agent::inject_into_env(&mut env);
x11::inject_into_env(&mut env);
waypipe::inject_into_env(&mut env);
let env = &env[..];

// Honor the image's default USER when the request doesn't pin one, so every
Expand Down
2 changes: 2 additions & 0 deletions crates/smolvm-agent/src/pod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -963,6 +963,8 @@ fn write_pod_bundle(

// Same injections as Run's bundle build (write_oci_bundle).
crate::ssh_agent::inject_into_container(&mut spec);
crate::x11::inject_into_container(&mut spec);
crate::waypipe::inject_into_container(&mut spec);
crate::rosetta::inject_into_container(&mut spec);
crate::cuda::inject_into_container(&mut spec, &pod.rootfs);

Expand Down
4 changes: 4 additions & 0 deletions crates/smolvm-agent/src/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2796,6 +2796,8 @@ pub fn run_command(

// Forward SSH agent into the container if enabled at boot.
crate::ssh_agent::inject_into_container(&mut spec);
crate::x11::inject_into_container(&mut spec);
crate::waypipe::inject_into_container(&mut spec);
crate::cuda::inject_into_container(&mut spec, Path::new(&prepared.rootfs_path));

// Write config.json to bundle
Expand Down Expand Up @@ -2887,6 +2889,8 @@ pub fn spawn_in_overlay(
add_storage_fallback(&mut spec, mounts, unprivileged);

crate::ssh_agent::inject_into_container(&mut spec);
crate::x11::inject_into_container(&mut spec);
crate::waypipe::inject_into_container(&mut spec);
crate::cuda::inject_into_container(&mut spec, Path::new(&prepared.rootfs_path));
spec.add_gpu_devices_if_available();

Expand Down
Loading