cuda: stage shims for conda layouts and via SMOLVM_CUDA_STAGE_EXTRA_DIRS

[NickyHeC]

Refs #598. Complements #600. Safe to merge alone; full no-LD_PRELOAD conda torch still wants #638.

Problem

find_rpath_pinned_libs only matches pip-wheel paths (…/site-packages/nvidia/*/lib/<soname>). Conda (/opt/conda/lib, soname→versioned symlink) and vendored CUDA dirs miss staging → real cudart/cublas load.

Change (crates/smolvm-agent/src/cuda.rs, scripts/build-agent-rootfs.sh)

• Treat /opt/conda/lib + /opt/conda/pkgs/*/lib as staged layouts.
• Resolve soname symlinks lexically under the rootfs (avoid canonicalize escaping through host path).
• SMOLVM_CUDA_STAGE_EXTRA_DIRS (colon-separated guest paths) escape hatch.
• Standalone safety: packaging stamps runtime-capabilities with conda-overmount-v1 only after nm verifies the Runtime exports conda torch needs. Without that stamp, conda/extra-dir sonames are left untouched so the old LD_PRELOAD fall-through still works. Pip-wheel overmounts remain unconditional.

Unit: finds_conda_symlinked_libs, finds_extra_dir_libs, incomplete_runtime_preserves_conda_ld_preload_fallback, capable_runtime_stages_conda_without_ld_preload.

Standalone vs combined behavior

Config	Conda discovery	Conda overmount	Torch import
This PR alone (incomplete shim, no stamp)	yes	skipped (real cudart kept; LD_PRELOAD fall-through preserved)	falls back to pre-#602 behavior
This PR + capable shim (#638 exports stamped)	yes	yes	no-LD_PRELOAD path

Proof — GPU (Lambda A10)

Image: pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime. Agent from this PR’s discovery commits. No LD_PRELOAD.

Tested commit set:

• agent tip at the time: f1b81c94 (conda/extra-dirs discovery)
• cudart stubs: 79d9ed52 + 8d7e0d66 (equivalent to current cudart: stub Runtime exports required by conda libtorch_cuda #638 731a60e6 + 07b04d2f)

Staging (discovery alone, pre-gate): conda soname overmounted with ~901KB shim.
Torch import (discovery alone): FAIL — undefined symbol: cudaDeviceSetLimit (motivates the capability gate above).
Torch e2e (discovery + #638 stubs): cuda: True → matmul → tiny-Llama backward ok.

The capability gate itself is unit-tested; it was not re-run on A10.

[ghost]

Ready to review this PR? Stage has broken it down into 3 individual chapters for you:

	Title
1	Define layout detection and symlink resolution logic
2	Expand library scanning to include conda and extra dirs
3	Verify conda symlinks and extra dir scanning

_{Chapters generated by Stage for commit f1b81c9 on Jul 15, 2026 6:07am UTC.}

[BinSquare]

Looks to be in the right direction - let's continue push for end to end validation

[NickyHeC]

GPU e2e — conda staging validated; torch import blocked by cudart shim surface

Rebased onto current main (tip f1b81c94) and exercised on Lambda A10 (Ubuntu 22.04, x86_64) with agent built from this branch. Packaged CUDA shims from #601 in the same rootfs. pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime, no LD_PRELOAD.

Staging discovery — PASS

Inside the guest after machine run --cuda:

/opt/smolvm-cuda/libcudart-shim.so   901240 (uid 1000)
/opt/conda/lib/libcudart.so.12 -> libcudart.so.12.4.127
/opt/conda/lib/libcudart.so.12.4.127 901240 (uid 1000)   ← bind-mounted shim, not the ~100MB real cudart

So /opt/conda/lib/ RPATH layout is found and overmounted as intended. SMOLVM_CUDA_STAGE_EXTRA_DIRS=/opt/conda/lib also exercised (smoke); staging already covered that path via the conda rules.

Torch e2e without LD_PRELOAD — FAIL (shim completeness, not discovery)

ImportError: .../libtorch_cuda.so: undefined symbol: cudaDeviceSetLimit, version libcudart.so.12

cudaDeviceSetLimit is not exported by smolvm-cudart-shim today. Once staging replaces the real libcudart.so.12* with the shim, torch's versioned dependency cannot be satisfied.

Notable side effect: classic LD_PRELOAD=/opt/smolvm-cuda/libcudart-shim.so no longer rescues this conda image either, because staging has already replaced the underlying soname — preload can no longer fall through to a full cudart for missing symbols. Pre-#602, staging missed conda paths so LD_PRELOAD worked with the real library remaining underneath.

Control — pip portal-cuda still OK

Same agent + shims, portal-cuda.tar (pip NVIDIA wheels):

cuda: True

Takeaway for merge gate

#602 piece	Result
Conda path discovery + overmount	PASS
SMOLVM_CUDA_STAGE_EXTRA_DIRS	accepted / smoke OK
conda torch init / backward() sans LD_PRELOAD	blocked on cudart shim API (cudaDeviceSetLimit at minimum)

Happy to land discovery as-is if the follow-up is a shim stub (or selective interposition) for missing Runtime API symbols — otherwise conda “drop LD_PRELOAD” stays blocked after this change lands, and the old LD_PRELOAD escape hatch regresses for staged conda layouts.

Host note (orthogonal): stock v1.6.0 libkrun.so needs GLIBC_2.39; this 22.04 box used v1.5.2 host libs to boot.

[NickyHeC]

Follow-up: conda torch e2e now PASS (with cudart stub PR)

After the staging discovery above failed torch import on cudaDeviceSetLimit, we audited libtorch_cuda.so (conda pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime) against the shim. Missing UND symbols (8 total after adding SetLimit):

• cudaDeviceSetLimit / cudaDeviceGetLimit
• cudaGraphAddKernelNode / AddHostNode / AddEventRecordNode / AddEventWaitNode
• cudaGraphRetainUserObject / cudaUserObjectCreate / cudaStreamUpdateCaptureDependencies

Shipped as draft #638 (nickyhec/cudart-stub-device-set-limit).

Retest on Lambda A10 — #602 agent + #638 cudart shim, no LD_PRELOAD

cuda: True
matmul_ok torch.Size([4, 4])
backward ok   # tiny-random-LlamaForCausalLM

So: #602 staging is sufficient for conda once the shim exports the Runtime API surface conda torch links against. Landing #602 alone without #638 (or equivalent) still regresses the old LD_PRELOAD fallthrough for incomplete shims.

[NickyHeC]

Evidence update (description rewritten).

Staging alone: conda /opt/conda/lib/libcudart.so.12* overmounted (~901KB shim) — PASS.

Torch without LD_PRELOAD: needs Runtime exports from #638 (bind-mount removes fallthrough). Combined #602+#638: cuda: True, matmul, tiny Llama backward ok.

Land with #638; this PR alone regresses the old LD_PRELOAD escape hatch for staged sonames.

[NickyHeC]

Following up on the end-to-end validation ask — that's now complete, so marking this ready for review.

e2e status (A10, Ubuntu 22.04): conda pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime, no LD_PRELOAD, agent built from this branch + shims from the released rootfs → cuda: True → matmul → tiny-Llama backward ok. The one caveat still holds: this PR needs #638 to work standalone — the bind-mount staging removes the old LD_PRELOAD fall-through, and conda libtorch_cuda.so needs the Runtime exports stubbed there. Combined #602 + #638 is the passing configuration.

Suggested landing: take the two together (I've marked both ready). Happy to reorder/rebase or fold the extra-dirs escape hatch however you'd prefer — just let me know.

Separately, while stress-testing the remoting path at scale I filed #667 (intermittent cold-start guest wedge on the first large fp32 load) — unrelated to this PR, noted only so it's on your radar.