diff --git a/skills/use-basilica/SKILL.md b/skills/use-basilica/SKILL.md index 964c8ab..6304f6b 100644 --- a/skills/use-basilica/SKILL.md +++ b/skills/use-basilica/SKILL.md @@ -37,7 +37,7 @@ positional GPU filters, `--compute`, `--gpu-count`, `--spot`, `--region`, concurrent orchestration, programmatic deploys, usage history, and distributed PyTorch/NCCL training. - Prefer direct rentals when the workload needs SSH, custom system setup, - persistent rented hosts, manual model warmup, or very large models. + persistent rented hosts, or manual host/process control. - Prefer serverless deploys when the user wants a public HTTP service, inference endpoint, hosted app URL, or short-lived demo. - Use the CLI for deposit-address creation and deposit history. The SDK exposes @@ -231,8 +231,11 @@ print(rental.ssh_command) ## Serverless Deployments Use CLI deploys for quick services, hosted URLs, container images, and -inference-style endpoints. Add `--ttl` unless the user explicitly wants the -deployment to persist. +inference-style endpoints. + +For source files, containers, persistent storage, GPU apps, custom Docker +images, WebSockets, public metadata, async orchestration, and progress +monitoring patterns, read `references/serverless-deployments.md`. ```bash basilica deploy my_api.py --name my-api --port 8000 --pip fastapi uvicorn --ttl 600 @@ -338,9 +341,11 @@ deployment = client.deploy_vllm( print(f"{deployment.url}/v1/chat/completions") ``` -For very large models, rentals may be a better first choice when the workload -needs manual control, custom setup, or warmup longer than deployment health -checks tolerate. +For standard vLLM or SGLang inference, start with `deploy_vllm()` or +`deploy_sglang()`; the templates handle common GPU detection, model caching, +health checks, and OpenAI-compatible endpoints. For larger or slower-loading +models, read `references/large-model-deployments.md` before writing custom +deployment code. ## OpenClaw And Tau diff --git a/skills/use-basilica/references/large-model-deployments.md b/skills/use-basilica/references/large-model-deployments.md new file mode 100644 index 0000000..e052a9e --- /dev/null +++ b/skills/use-basilica/references/large-model-deployments.md @@ -0,0 +1,270 @@ +# Large Model Deployments + +Use this reference when a user wants to deploy a large vLLM or SGLang model, or +when an inference deployment is failing during download, startup, health checks, +or model loading. + +## Scope + +This reference is for hosted HTTP inference deployment flows. + +## Deployment Ladder + +Start with the highest-level surface that gives enough control: + +1. Use `client.deploy_vllm()` or `client.deploy_sglang()` for ordinary inference + servers. These helpers cover common GPU sizing, model caching, health checks, + and OpenAI-compatible endpoints. +2. Use `client.deploy(...)` or `@basilica.deployment` when the server needs + custom source, explicit images, custom env vars, explicit probe settings, + retries, or a longer timeout. +3. Use `CreateDeploymentRequest` when the model needs precise resources, + low-level command/args, multi-GPU tensor parallelism, custom vLLM/SGLang + images, or explicit `HealthCheckConfig` that the helper surface cannot + express clearly. + +Keep cost-bearing behavior explicit. Use `ttl_seconds=` for experiments and +show cleanup with `deployment.delete()` or `basilica deploy delete `. + +## Large Model Knobs + +Set these deliberately for large or slow-loading models: + +- GPU shape: `gpu_count`, GPU model, and per-GPU VRAM floor such as + `min_gpu_memory_gb`. +- Container resources: CPU and system RAM such as `cpu="32"` and + `memory="512Gi"` for 70B/1T-class model servers. +- Runtime image: use a vLLM/SGLang image that supports the model architecture; + use a custom image when stable upstream images do not include required model + classes or parsers. +- Server args: tensor parallelism, context length, `trust_remote_code`, + tool-call parser, reasoning parser, dtype, GPU memory utilization, and any + model-specific flags from the model vendor. +- Startup budget: set deployment `timeout` plus startup/readiness/liveness + probes to cover download, shard loading, CUDA graph capture, and first health + response. +- Download reliability: set Hugging Face timeout/cache env vars or pre-download + with retry logic when model files are large or flaky. +- Observability: tell the user to follow `basilica deploy logs --follow` + during startup; a deployment can be healthy eventually even if the initial + client wait times out. + +## Template Helper Example + +Use this for common vLLM or SGLang models before reaching for low-level APIs. + +```python +import basilica + +client = basilica.BasilicaClient() + +deployment = client.deploy_vllm( + model="Qwen/Qwen2.5-0.5B-Instruct", + name="qwen-0-5b-vllm", + gpu_count=1, + memory="16Gi", + ttl_seconds=3600, +) + +print(deployment.url) +print(f"{deployment.url}/v1/chat/completions") +deployment.delete() +``` + +For SGLang, use the SGLang helper and tune SGLang-specific options: + +```python +import basilica + +client = basilica.BasilicaClient() + +deployment = client.deploy_sglang( + model="Qwen/Qwen2.5-3B-Instruct", + name="qwen-3b-sglang", + gpu_count=1, + context_length=8192, + mem_fraction_static=0.85, + trust_remote_code=True, + ttl_seconds=3600, +) + +print(deployment.url) +print(f"{deployment.url}/v1/chat/completions") +deployment.delete() +``` + +## Custom Health Checks For Slow Startup + +Large models can be killed before they finish loading if the startup window is +too short. Configure a startup probe first; liveness and readiness should not +drive restarts until startup has had enough time. + +```python +from basilica import BasilicaClient, HealthCheckConfig, ProbeConfig + +PORT = 8000 + +def startup_health_check(startup_minutes: int) -> HealthCheckConfig: + initial_delay = 480 + period = 120 + timeout = 120 + failures = max(1, (startup_minutes * 60 - initial_delay) // period) + + return HealthCheckConfig( + startup=ProbeConfig( + path="/health", + port=PORT, + initial_delay_seconds=initial_delay, + period_seconds=period, + timeout_seconds=timeout, + failure_threshold=failures, + ), + liveness=ProbeConfig( + path="/health", + port=PORT, + initial_delay_seconds=initial_delay, + period_seconds=period, + timeout_seconds=timeout, + failure_threshold=5, + ), + readiness=ProbeConfig( + path="/health", + port=PORT, + initial_delay_seconds=initial_delay, + period_seconds=period, + timeout_seconds=timeout, + failure_threshold=5, + ), + ) + +client = BasilicaClient() +health_check = startup_health_check(startup_minutes=45) + +deployment = client.deploy( + name="sglang-large-model", + source="server.py", + image="lmsysorg/sglang:latest", + port=PORT, + health_check=health_check, + timeout=46 * 60, + ttl_seconds=3600, + gpu_count=1, + gpu_models=["A100"], + min_gpu_memory_gb=80, + cpu="2", + memory="64Gi", + env={ + "HF_HUB_DISABLE_SYMLINKS_WARNING": "1", + "HF_HUB_DISABLE_XET": "1", + }, +) + +print(f"logs: basilica deploy logs {deployment.name} --follow") +``` + +## Low-Level Multi-GPU vLLM Example + +Use this shape for 70B/1T-class models that need exact tensor parallelism, +high-VRAM GPUs, custom images, long startup budgets, or model-specific parsers. + +```python +import basilica +from basilica import ( + BasilicaClient, + HealthCheckConfig, + ProbeConfig, +) + +client = BasilicaClient() +model = "moonshotai/Kimi-K2-Instruct" + +args = [ + "serve", + model, + "--host", + "0.0.0.0", + "--port", + "8000", + "--tensor-parallel-size", + "8", + "--trust-remote-code", + "--tool-call-parser", + "kimi_k2", + "--enable-auto-tool-choice", + "--max-model-len", + "32768", + "--gpu-memory-utilization", + "0.95", +] + +health_check = HealthCheckConfig( + liveness=ProbeConfig( + path="/health", + port=8000, + initial_delay_seconds=5400, + period_seconds=30, + timeout_seconds=10, + failure_threshold=3, + ), + readiness=ProbeConfig( + path="/health", + port=8000, + initial_delay_seconds=5400, + period_seconds=10, + timeout_seconds=5, + failure_threshold=3, + ), +) + +response = client.create_deployment( + instance_name="kimi-k2-instruct", + image="vllm/vllm-openai:latest", + replicas=1, + port=8000, + command=["vllm"], + args=args, + env={"HF_HUB_DOWNLOAD_TIMEOUT": "3600"}, + cpu="32", + memory="512Gi", + gpu_count=8, + gpu_models=["H200"], + min_gpu_memory_gb=80, + ttl_seconds=7200, + public=True, + health_check=health_check, +) + +deployment = client.get(response.instance_name) + +try: + deployment.wait_until_ready(timeout=2400, silent=False) +except ( + basilica.exceptions.DeploymentTimeout, + basilica.exceptions.DeploymentFailed, +): + print(f"Still loading. Follow logs: basilica deploy logs {deployment.name} --follow") + +print(f"{deployment.url}/v1/chat/completions") +``` + +For newer model architectures not available in the stable runtime image, keep +the same deployment shape but switch to a custom image that contains the needed +vLLM/SGLang build, parser, or model class. + +## Failure Handling + +When startup fails, inspect in this order: + +1. `basilica deploy status --show-phases` +2. `basilica deploy logs --tail 100` +3. `basilica deploy logs --follow` +4. Increase startup probe window and client timeout if logs show active + download/loading rather than a crash. +5. Change image or server args if logs show missing model architecture, parser, + CUDA/runtime incompatibility, or unsupported model flags. +6. Increase GPU count, GPU memory floor, CPU, or system RAM if logs show + out-of-memory, scheduler mismatch, or tensor-parallel placement failures. + +If the deployment is only still loading, do not delete it reflexively. Tell the +user how to monitor logs and how to delete it if they do not want to keep paying +for the experiment. diff --git a/skills/use-basilica/references/serverless-deployments.md b/skills/use-basilica/references/serverless-deployments.md new file mode 100644 index 0000000..640b35d --- /dev/null +++ b/skills/use-basilica/references/serverless-deployments.md @@ -0,0 +1,569 @@ +# Serverless Deployments + +Use this reference when a user wants to deploy an HTTP service, web app, +container, GPU app, stateful demo, WebSocket app, public-metadata deployment, or +many short-lived deployments on Basilica. + +## Contents + +- Surface Selection +- CLI Patterns +- SDK Basic HTTP Service +- SDK FastAPI File Deployment +- SDK Container Deployment +- SDK Decorator Deployment +- Storage And Volumes +- GPU App Deployment +- WebSockets +- Public Metadata +- Custom Commands +- Progress And Async Orchestration +- Troubleshooting + +## Surface Selection + +Use the highest-level surface that gives enough control: + +1. Use `basilica deploy ...` for interactive CLI workflows and quick demos. +2. Use `client.deploy(...)` for scripts, CI, notebooks, and source/file/container + deployments where the agent should wait for readiness. +3. Use `@basilica.deployment` when the app is naturally expressed as a Python + function and should be deployable by calling that function. +4. Use `client.create_deployment(...)` for lower-level deployment features such + as custom commands, WebSockets, public metadata, custom health checks, + custom images, or when the script should create first and wait separately. +5. Use `client.deploy_async(...)` and async cleanup when launching many + deployments concurrently. + +Always make cost-bearing behavior explicit. Prefer `--ttl` or `ttl_seconds=` +for experiments, and show cleanup with `basilica deploy delete ` or +`deployment.delete()`. + +## CLI Patterns + +Deploy a Python file with dependencies: + +```bash +basilica deploy my_api.py \ + --name my-api \ + --port 8000 \ + --pip fastapi uvicorn \ + --ttl 600 +``` + +Deploy a non-root container image: + +```bash +basilica deploy nginxinc/nginx-unprivileged:alpine \ + --name nginx-demo \ + --port 8080 \ + --cpu 250m \ + --memory 256Mi \ + --ttl 300 +``` + +Deploy with GPU resources: + +```bash +basilica deploy inference.py \ + --name gpu-model \ + --gpu 1 \ + --gpu-model H100 \ + --gpu-memory-gb 80 \ + --memory 32Gi \ + --pip torch \ + --ttl 3600 +``` + +Deploy with persistent storage mounted at `/data`: + +```bash +basilica deploy hello.py \ + --name stateful-app \ + --storage \ + --storage-path /data \ + --ttl 3600 +``` + +Deploy with custom health checks: + +```bash +basilica deploy my_api.py \ + --name health-api \ + --port 8000 \ + --pip fastapi uvicorn \ + --health-path /health \ + --health-initial-delay 10 \ + --health-period 30 \ + --ttl 600 +``` + +Deploy with WebSocket support: + +```bash +basilica deploy ws_app.py \ + --name ws-app \ + --port 8000 \ + --websocket \ + --ws-idle-timeout 3600 \ + --ttl 3600 +``` + +Enroll deployment metadata for public validator verification: + +```bash +basilica deploy hashicorp/http-echo:latest \ + --name metadata-demo \ + --port 5678 \ + --public-metadata \ + --ttl 600 + +basilica deploy enroll-metadata metadata-demo +basilica deploy metadata metadata-demo --json +``` + +Manage deployments: + +```bash +basilica deploy ls --json +basilica deploy status my-api --show-phases +basilica deploy logs my-api --tail 100 +basilica deploy logs my-api --follow +basilica deploy scale my-api --replicas 3 +basilica deploy restart my-api +basilica deploy delete my-api --yes +``` + +## SDK Basic HTTP Service + +Use inline source for small demos and prototypes: + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.deploy( + name="hello", + source=""" +from http.server import HTTPServer, BaseHTTPRequestHandler + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + self.send_response(200) + self.end_headers() + self.wfile.write(b"Hello from Basilica!") + +HTTPServer(("", 8000), Handler).serve_forever() +""", + port=8000, + ttl_seconds=600, +) + +print(deployment.url) +deployment.delete() +``` + +## SDK FastAPI File Deployment + +Use a source file for single-file apps. The SDK reads and packages the file. + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.deploy( + name="file-api", + source="app_file.py", + port=8000, + pip_packages=["fastapi", "uvicorn"], + ttl_seconds=600, + timeout=180, +) + +print(f"docs: {deployment.url}/docs") +print(f"health: {deployment.url}/health") +deployment.delete() +``` + +The app file should bind to `0.0.0.0` on the deployed port: + +```python +from fastapi import FastAPI + +app = FastAPI() + +@app.get("/health") +def health(): + return {"status": "healthy"} + +if __name__ == "__main__": + import uvicorn + uvicorn.run(app, host="0.0.0.0", port=8000) +``` + +## SDK Container Deployment + +Use `image=` when the app is already packaged. Basilica runs containers as a +non-root user, so choose images that work without root privileges. + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.deploy( + name="nginx-demo", + image="nginxinc/nginx-unprivileged:alpine", + port=8080, + replicas=1, + env={"NGINX_HOST": "localhost"}, + cpu="250m", + memory="256Mi", + ttl_seconds=600, + timeout=120, +) + +print(deployment.url) +deployment.delete() +``` + +For multi-file projects, build and push a custom image first: + +```dockerfile +FROM python:3.11-slim + +RUN useradd -m -u 1000 appuser +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY --chown=appuser:appuser app/ ./app/ +USER appuser + +EXPOSE 8000 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +```bash +docker build -t ghcr.io/yourusername/my-api:latest . +docker push ghcr.io/yourusername/my-api:latest +``` + +```python +deployment = client.deploy( + name="custom-api", + image="ghcr.io/yourusername/my-api:latest", + port=8000, + ttl_seconds=3600, + timeout=180, +) +``` + +## SDK Decorator Deployment + +Use `@basilica.deployment` for Python functions that start their own HTTP +server. + +```python +import basilica + +@basilica.deployment( + name="decorator-api", + port=8000, + pip_packages=["fastapi", "uvicorn"], + ttl_seconds=600, +) +def serve(): + from fastapi import FastAPI + import uvicorn + + app = FastAPI() + + @app.get("/") + def root(): + return {"message": "Hello from decorator FastAPI!"} + + uvicorn.run(app, host="0.0.0.0", port=8000) + +deployment = serve() +print(deployment.url) +deployment.delete() +``` + +## Storage And Volumes + +Use `storage=True` for high-level deploys that need persistent data at `/data`. +Use `Volume.from_name(..., create_if_missing=True)` with decorator deployments. + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.deploy( + name="counter", + source=""" +from http.server import HTTPServer, BaseHTTPRequestHandler +from pathlib import Path + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + f = Path("/data/count") + n = int(f.read_text()) + 1 if f.exists() else 1 + f.write_text(str(n)) + self.send_response(200) + self.end_headers() + self.wfile.write(f"Visit #{n}".encode()) + +HTTPServer(("", 8000), Handler).serve_forever() +""", + port=8000, + storage=True, + ttl_seconds=600, +) +``` + +```python +import basilica + +cache = basilica.Volume.from_name("counter-cache", create_if_missing=True) + +@basilica.deployment( + name="decorator-counter", + port=8000, + volumes={"/data": cache}, + ttl_seconds=600, +) +def serve(): + ... +``` + +## GPU App Deployment + +Use the CUDA-capable image plus explicit GPU and memory requirements. + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.deploy( + name="gpu-test", + source=""" +import json +import torch +from http.server import HTTPServer, BaseHTTPRequestHandler + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + info = { + "cuda_available": torch.cuda.is_available(), + "device_count": torch.cuda.device_count(), + "device_name": torch.cuda.get_device_name(0) if torch.cuda.is_available() else None, + } + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(json.dumps(info).encode()) + +HTTPServer(("", 8000), Handler).serve_forever() +""", + image="pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime", + port=8000, + gpu_count=1, + min_gpu_memory_gb=16, + memory="8Gi", + ttl_seconds=600, + timeout=300, +) + +print(deployment.url) +deployment.delete() +``` + +Use placement preferences when needed: + +```python +deployment = client.deploy( + name="flavour-hello", + source="app.py", + port=8000, + gpu_count=1, + gpu_models=["H100"], + interconnect="SXM", + ttl_seconds=600, +) +``` + +## WebSockets + +Use `WebSocketConfig` when the gateway must allow long-lived bidirectional +connections. + +```python +from basilica import BasilicaClient, WebSocketConfig + +client = BasilicaClient() + +deployment = client.create_deployment( + instance_name="ws-demo", + image="hashicorp/http-echo:latest", + replicas=1, + port=5678, + websocket=WebSocketConfig(enabled=True, idle_timeout_seconds=3600), + ttl_seconds=600, +) + +print(deployment.url) +client.delete_deployment(deployment.instance_name) +``` + +## Public Metadata + +Use public metadata only when non-sensitive deployment metadata should be +publicly queryable for validator verification. + +```python +from basilica import BasilicaClient + +client = BasilicaClient() + +deployment = client.create_deployment( + instance_name="metadata-demo", + image="hashicorp/http-echo:latest", + replicas=1, + port=5678, + public_metadata=True, + ttl_seconds=600, +) + +status = client.get_enrollment_status(deployment.instance_name) +metadata = client.get_public_deployment_metadata(deployment.instance_name) + +print(status.public_metadata) +print(metadata.state) + +client.enroll_metadata(deployment.instance_name, enabled=False) +client.enroll_metadata(deployment.instance_name, enabled=True) +client.delete_deployment(deployment.instance_name) +``` + +## Custom Commands + +Use `create_deployment()` when the app needs a specific container command. +This is useful for frameworks that need a custom runner. + +```python +import base64 +from pathlib import Path +from basilica import BasilicaClient + +client = BasilicaClient() + +app_source = Path("streamlit_app.py").read_text() +app_b64 = base64.b64encode(app_source.encode()).decode() + +script = ( + "pip install -q streamlit && " + f'echo "{app_b64}" | base64 -d > /tmp/app.py && ' + "python3 -m streamlit run /tmp/app.py " + "--server.port=8501 --server.address=0.0.0.0 --server.headless=true" +) + +response = client.create_deployment( + instance_name="streamlit-demo", + image="python:3.11-slim", + port=8501, + command=["bash", "-c", script], + cpu="500m", + memory="512Mi", + ttl_seconds=3600, +) + +deployment = client.get(response.instance_name) +deployment.wait_until_ready(timeout=300) +print(deployment.url) +deployment.delete() +``` + +## Progress And Async Orchestration + +Use progress callbacks when building a UI, logging deployment stages, or +debugging startup phases. + +```python +from basilica import BasilicaClient, DeploymentStatus + +def on_progress(status: DeploymentStatus) -> None: + phase = status.phase or "unknown" + replicas = f"{status.replicas_ready}/{status.replicas_desired}" + print(f"{phase} replicas={replicas}") + +client = BasilicaClient() + +response = client.create_deployment( + instance_name="progress-demo", + image="python:3.11-slim", + command=[ + "python", + "-c", + "from http.server import HTTPServer, BaseHTTPRequestHandler; " + "HTTPServer(('', 8000), type('H', (BaseHTTPRequestHandler,), " + "{'do_GET': lambda s: (s.send_response(200), s.end_headers(), " + "s.wfile.write(b'Progress demo!'))})).serve_forever()", + ], + port=8000, + ttl_seconds=300, +) + +deployment = client.get(response.instance_name) +deployment.wait_until_ready(timeout=120, poll_interval=3, on_progress=on_progress) +``` + +Use async APIs for many short-lived deployments and cleanup all successful +deployments. + +```python +import asyncio +from basilica import BasilicaClient + +async def deploy_one(client: BasilicaClient, index: int): + return await client.deploy_async( + name=f"async-{index:02d}", + source="app.py", + env={"APP_ID": f"{index:02d}"}, + port=8000, + ttl_seconds=180, + timeout=180, + ) + +async def main(): + client = BasilicaClient() + deployments = await asyncio.gather( + *(deploy_one(client, i) for i in range(1, 6)), + return_exceptions=True, + ) + + ready = [d for d in deployments if not isinstance(d, Exception)] + await asyncio.gather(*(d.delete_async() for d in ready), return_exceptions=True) + +asyncio.run(main()) +``` + +## Troubleshooting + +- If `client.deploy()` times out, check `basilica deploy status + --show-phases` and `basilica deploy logs --tail 100`. +- If the public URL returns 502/503, verify the process binds to `0.0.0.0` and + the app port matches the deployment `port`. +- If a container fails immediately, verify it can run as UID 1000 and does not + need root-only paths or privileged ports. +- If a Python-file deploy fails during startup, verify required packages are in + `pip_packages` or `--pip`. +- If storage paths are missing, verify the mount path and wait for storage sync + before treating it as an app bug. +- If GPU is unavailable, lower requirements or use `basilica ls` to inspect + available capacity before retrying.