bug(sdk): deploy_distributed source= hardcodes /workspace, fails on uid=1000 pods

[epappas]

Summary

Discovered during the Phase 5b end-to-end smoke (after PR #447 + the basilica-backend platform fixes #430/#431/#432 went live). The Phase 5b SDK's client.deploy_distributed(name=..., source="...") path constructs a bash one-liner that writes to /workspace/__basilica_source.py. The default base image is pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime where /workspace is the WORKDIR and root-owned. The operator-rendered pod runs as runAsUser: 1000 with no writable volume mounted at /workspace. Result: every worker rank's container hits cannot create /workspace/__basilica_source.py: Permission denied and CrashLoopBackoff.

Reproduction

import basilica
from basilica import BasilicaClient, ProviderFilter, WorldSize

client = BasilicaClient()
t = client.deploy_distributed(
    name="repro-source-bug",
    source="""\
import os, time
print(f"[rank {os.environ.get('RANK','?')}] hello", flush=True)
time.sleep(3600)
""",
    world_size=WorldSize(min=2, target=2, max=3),
    gpu_count=1,
    gpu_models=["A100"],
    provider_filter=ProviderFilter(include=["verda"]),
    bench="on-start",
)

$ kubectl logs <ud>-0
Defaulted container "<ud>" out of: <ud>, node-env (init)
--: 1: cannot create /workspace/__basilica_source.py: Permission denied

Root cause

crates/basilica-sdk-python/python/basilica/__init__.py:1220:

distributed_command = (
    f"{pip_install}"
    f"echo {src_b64} | base64 -d > /workspace/__basilica_source.py && "
    f"exec torchrun ..."
    f"/workspace/__basilica_source.py"
)

The pod template (rendered by basilica-operator's controllers/distributed.rs) has:

• runAsUser: 1000
• runAsNonRoot: true
• readOnlyRootFilesystem: false but /workspace itself is owned by root in the pytorch base image
• Only basilica-env (emptyDir at /etc/basilica) and the projected token volume are writable
• No emptyDir at /workspace

Fix shape (suggested)

Pick whichever is least invasive:

1. Write to /tmp/__basilica_source.py instead of /workspace/ — /tmp is writable by uid 1000 in any base image. Update the torchrun argument to match. One-line change.
2. Mount an emptyDir at /workspace from the operator side — bigger change, requires basilica-operator coordination.
3. Ship source via a ConfigMap volume mounted readonly at /workspace — cleaner long-term but more complex.

Workaround (verified working)

Use command= instead of source=:

import base64
src = """..."""
src_b64 = base64.b64encode(src.encode()).decode()
t = client.deploy_distributed(
    name="...",
    command=["bash", "-c",
        f"echo {src_b64} | base64 -d > /tmp/script.py && "
        f"exec torchrun --rdzv-backend=etcd-v2 --rdzv-endpoint=\"$BASILICA_RDZV_ENDPOINT\" "
        f"--rdzv-id=\"$BASILICA_RDZV_ID\" --nnodes=\"$BASILICA_WORLD_MIN\":\"$BASILICA_WORLD_MAX\" "
        f"--nproc-per-node=\"$BASILICA_GPUS_PER_POD\" --max-restarts=3 /tmp/script.py"],
    ...
)

This was verified end-to-end on prod K3s; both ranks reached phase=Running, worldSize.ready=2/2.

Scope

• ✅ Fix __init__.py:1220 to write to a uid-1000-writable path (likely /tmp/).
• ❌ Out of scope: changing pod securityContext (would weaken tenant isolation).
• ❌ Out of scope: changing pytorch base image default.

Refs

• Phase 5b SDK + CLI PR: one-covenant/basilica#447 (merged 2026-05-03)
• Phase 5b platform precursor: one-covenant/basilica-backend#421
• DNS-1035 cr_name fix: one-covenant/basilica-backend#430
• Phase 5b end-to-end smoke transcript (this issue's discovery): logs available on request

Priority

P1 — every researcher invoking client.deploy_distributed(source=...) against the default pytorch image hits this. The decorator path likely shares the same code path (Callable → SourcePackager.from_function → same source-shipping bash), so @basilica.distributed is also broken in the default config.