diff --git a/AGENTS.md b/AGENTS.md index 7c5a24f..3b21e7a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,7 @@ templates/demo_llm_dark.html ← LLM demo template, dark-signal theme (technic templates/demo_vlm_crisp.html ← Vision demo template, crisp-light theme (multimodal domains) templates/demo_vlm_dark.html ← Vision demo template, dark-signal theme (technical multimodal) scripts/llamacpp.py ← llama.cpp unified CLI: install, quantize, bench, ppl, serve, chat, deploy +scripts/litertlm.py ← LiteRT-LM unified CLI: install, bundle, serve, chat, deploy templates/chat_ui.html ← Gaslamp Chat WebUI for local GGUF inference via llama-server ``` @@ -52,7 +53,7 @@ templates/chat_ui.html ← Gaslamp Chat WebUI for local GGUF inference v | 5. Eval | Run eval against base and fine-tuned model — both batch and `--compare` mode | | 5.5. Demo | Ask user if they want a shareable demo; read `sub-skills/demo_builder.md`; write `demos//index.html` | | 6. Export | Convert to GGUF / merge / push to HF Hub per user's deploy target | -| 6.5. Deploy | Optional: `llamacpp.py deploy` → quantize + bench + serve + chat UI (requires llama.cpp) | +| 6.5. Deploy | Optional: `llamacpp.py deploy` or `litertlm.py deploy` → serve + chat UI (requires llama.cpp or LiteRT-LM) | | 7. Reflect | Run `reflect.py --extract`, classify candidates into lessons/recipes, pipe to `reflect.py --write` → updates `~/.gaslamp/` | Everything is scoped to the dated project directory. Nothing touches the repo root. diff --git a/README.md b/README.md index 2f60e0b..ac45407 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ **The self-evolving fine-tuning agent.** It talks like a colleague, learns your setup's quirks over time, and orchestrates the full lifecycle: from data formatting and model selection to training, validation, and deployment. -Runs on NVIDIA GPUs via [Unsloth](https://github.com/unslothai/unsloth), natively on Apple Silicon via [mlx-tune](https://github.com/ml-explore/mlx-lm), and on free cloud GPUs via [colab-mcp](https://github.com/googlecolab/colab-mcp). Part of the [Gaslamp](https://gaslamp.dev/) AI development platform — [docs](https://gaslamp.dev/unsloth). +Runs on NVIDIA GPUs via [Unsloth](https://github.com/unslothai/unsloth), natively on Apple Silicon via [mlx-tune](https://github.com/ml-explore/mlx-lm), and on free cloud GPUs via [google-colab-cli](https://github.com/googlecolab/google-colab-cli). Part of the [Gaslamp](https://gaslamp.dev/) AI development platform — [docs](https://gaslamp.dev/unsloth). --- @@ -128,7 +128,7 @@ Eight phases, each scoped to an isolated dated project directory that never touc | **5. Evaluation** | Batch tests, interactive REPL, base vs fine-tuned comparison | `logs/eval.log` | | **5.5. Demo** | Generates a shareable static HTML page — base vs fine-tuned side-by-side | `demos//index.html` | | **6. Export** | GGUF, merged 16-bit, or Hub push | `outputs/` | -| **6.5. Local Deploy** | Optional: quantize → bench → serve + Gaslamp Chat WebUI (requires llama.cpp) | `outputs/*.gguf` | +| **6.5. Local Deploy** | Optional: quantize → bench → serve + Gaslamp Chat WebUI (requires llama.cpp or LiteRT-LM) | `outputs/*.gguf` | | **7. Reflect** | Synthesizes lessons, gotchas, and recipes into `~/.gaslamp/` for future projects | `~/.gaslamp/` | ``` @@ -151,7 +151,7 @@ customer_faq_sft_2026_03_17/ | NVIDIA T4 (16 GB) | `unsloth` | 7B QLoRA, small-scale GRPO | | NVIDIA A100 (80 GB) | `unsloth` | 70B QLoRA, 14B LoRA 16-bit | | Apple M1 / M2 / M3 / M4 | `mlx-tune` / `mlx-vlm` / `trl` | SFT/DPO: 7B on 10 GB, 13B on 24 GB; Vision SFT via `mlx-vlm`; GRPO: 1–7B via TRL + PyTorch MPS | -| Google Colab (T4/L4/A100) | `unsloth` via `colab-mcp` | Free cloud GPU, opt-in | +| Google Colab (T4/L4/A100) | `unsloth` via `google-colab-cli` | Free cloud GPU, opt-in | Unsloth is ~2× faster than standard HuggingFace training, uses up to 80% less VRAM, and produces exact gradients. @@ -218,20 +218,33 @@ python scripts/llamacpp.py chat --model model-q4_k_m.gguf Requires [llama.cpp](https://github.com/ggml-org/llama.cpp) — installed automatically via `llamacpp.py install`. +For on-device TFLite models, [LiteRT-LM](https://github.com/nicholasgasior/litert-lm) is also supported: + +```bash +python scripts/litertlm.py install # install litert-lm-api & litert-lm-builder +python scripts/litertlm.py deploy --tflite model.tflite --tokenizer tokenizer.model --output model.litertlm +python scripts/litertlm.py serve --model model.litertlm +``` + +LiteRT-LM is a deploy target for models already converted to TFLite with a SentencePiece tokenizer. Standard LoRA adapter, GGUF, and merged safetensors exports should use the matching deploy path instead. + --- ## Google Colab Training Apple Silicon users who need larger models or CUDA-only features can offload training to a free Colab GPU: -1. Install `colab-mcp` in Claude Code: +1. Install the official Colab CLI: + ```bash + uv tool install google-colab-cli + ``` +2. Provision a named GPU runtime: `colab new -s unsloth-buddy --gpu T4` +3. Run setup and training on the remote VM: ```bash - uv python install 3.13 - claude mcp add colab-mcp -- uvx --from git+https://github.com/googlecolab/colab-mcp --python 3.13 colab-mcp + colab exec -s unsloth-buddy -f scripts/setup_colab.py + colab exec -s unsloth-buddy -f train.py ``` -2. Open a Colab notebook, connect to a T4/L4 GPU runtime -3. The agent connects, installs Unsloth, starts training in a background thread, and polls metrics every 30s -4. Download adapters from the Colab file browser when done +4. Download adapters: `colab download -s unsloth-buddy /content/outputs/ outputs/` Local mlx-tune remains the default — Colab is opt-in for when you need more power. @@ -274,6 +287,7 @@ This works through a local `~/.gaslamp/` memory directory (never committed to yo ## Changelog +- **2026-06-28** — Added **LiteRT-LM support** (Phase 6.5): `scripts/litertlm.py` provides 5 subcommands (`install`, `bundle`, `serve`, `chat`, `deploy`) for on-device TFLite LLM inference. Migrated Google Colab integration from `colab-mcp` to the official [google-colab-cli](https://github.com/googlecolab/google-colab-cli). - **2026-04-14** — **Self-evolving memory** (Phase 7 + global inject): after each project, the agent synthesizes lessons, model-specific gotchas, and reusable scenario recipes into `~/.gaslamp/`. Every new project injects a frozen snapshot at startup and silently applies past knowledge. Implements the Frozen Snapshot pattern from agent memory research. See `ref/self_evolve_plan.md`. - **2026-04-12** — Added **llama.cpp local deploy** (Phase 6.5): after GGUF export, if llama.cpp is installed, the agent offers a one-command pipeline — quantize → benchmark → serve + open the Gaslamp Chat WebUI (`templates/chat_ui.html`). `scripts/llamacpp.py` provides 7 subcommands (`install`, `quantize`, `bench`, `ppl`, `serve`, `chat`, `deploy`); auto-selects GPU offload on Apple Silicon (Metal) and NVIDIA. `scripts/detect_system.py` now detects llama.cpp binaries and prints an install hint if missing. - **2026-04-10** — Added native **Vision SFT for Apple Silicon**: Integrated [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) to support multimodal fine-tuning (e.g. Gemma 4 Vision, Qwen2.5-VL) on M-series chips. Added `scripts/unsloth_mlx_vision_example.py` training template and `mlx_eval_vision_template.py` for comparative vision evaluation. Demo Builder now supports wide-format VLM layouts (`vlm-crisp`, `vlm-dark`) and relative PNG asset packaging for offline-portable multimodal dashboards. diff --git a/SKILL.md b/SKILL.md index e6e73e0..7fb4695 100644 --- a/SKILL.md +++ b/SKILL.md @@ -36,7 +36,7 @@ All scripts and templates are installed alongside this skill. Do NOT `ls` to dis | `scripts/gaslamp_callback.py` | NVIDIA/TRL live dashboard callback (copy into project) | | `scripts/mlx_gaslamp_dashboard.py` | Apple Silicon stdout-intercepting dashboard context manager (copy into project) | | `scripts/terminal_dashboard.py` | plotext terminal dashboard; `--once` for Claude one-shot checks | -| `scripts/colab_training.py` | Colab cell generators: `SETUP_CELL`, `VERIFY_CELL`, `get_training_cell()`, `POLL_CELL`, `FINAL_CELL` | +| `scripts/colab_training.py` | Colab training helpers: code templates (`SETUP_CELL`, `VERIFY_CELL`, `get_training_cell()`) and `INSTALL_INSTRUCTIONS` for `google-colab-cli` | | `scripts/setup_colab.py` | Colab environment setup utilities | | `scripts/unsloth_mlx_sft_example.py` | **Apple Silicon SFT training template** — copy as `train.py` | | `scripts/unsloth_mlx_vision_example.py` | **Apple Silicon vision training template** — copy as `train.py` | @@ -57,7 +57,8 @@ All scripts and templates are installed alongside this skill. Do NOT `ls` to dis | `templates/demo_vlm_crisp.html` | **Vision demo template — crisp-light** (wide layout for images; for consumer/multimodal domains) | | `templates/demo_vlm_dark.html` | **Vision demo template — dark-signal** (wide layout for images; for technical/multimodal domains) | | `scripts/llamacpp.py` | **llama.cpp unified CLI** — install, quantize, bench, ppl, serve, chat, deploy (one-command auto-pipeline) | -| `templates/chat_ui.html` | **Gaslamp Chat WebUI** — dark glassmorphism chat interface for local GGUF inference via llama-server | +| `scripts/litertlm.py` | **LiteRT-LM unified CLI** — install, bundle, serve, chat, deploy (local TensorFlow Lite-based LLM inference) | +| `templates/chat_ui.html` | **Gaslamp Chat WebUI** — dark glassmorphism chat interface for GGUF/LiteRT local inference | ## The 7-Phase End-to-End Lifecycle (+Deploy) @@ -166,97 +167,55 @@ Follow the matching path below. --- -#### Path A: Google Colab (via colab-mcp) +#### Path A: Google Colab (via google-colab-cli) -Colab gives free GPU access with no local installation. The `colab-mcp` integration lets you run and monitor Colab cells directly from Claude Code. +Colab gives free GPU/TPU access with no local installation. The official `google-colab-cli` integration lets you provision, run, and monitor training on Colab runtimes directly from your local terminal. -**Step A1 — Install colab-mcp (first time only)** - -**First, check whether `execute_code` is available as an MCP tool in the current session.** -- If `execute_code` IS available → skip to Step A2. -- If `execute_code` is NOT available → colab-mcp is not installed. Run the install flow below. - -**Install for Claude Code (CLI):** +**Step A1 — Install Colab CLI** +Install the official Colab CLI tool globally or in your environment: ```bash -# 1. (If needed) Install Python 3.13 -uv python install 3.13 - -# 2. Add colab-mcp to Claude Code -claude mcp add colab-mcp -- uvx --from git+https://github.com/googlecolab/colab-mcp --python 3.13 colab-mcp - -# 3. Verify it was added -claude mcp list -``` - -Open `~/.claude.json`, find the `colab-mcp` entry under your project's `mcpServers`, and ensure it matches: -```json -"colab-mcp": { - "command": "uvx", - "args": ["--from", "git+https://github.com/googlecolab/colab-mcp", - "--python", "3.13", "colab-mcp"], - "timeout": 30000 -} +uv tool install google-colab-cli ``` -> Note: colab-mcp requires Python ≥ 3.13. `uvx --python 3.13` runs it in an isolated env, keeping your training venv (Python ≤ 3.12 for mlx-tune) untouched. Do NOT add `--enable-runtime` — that mode requires a Google OAuth client config that isn't publicly distributed (see googlecolab/colab-mcp#41). - -**3. Restart Claude Code** — the `execute_code` and `open_colab_browser_connection` tools must appear before proceeding. - -> Note: colab-mcp connects to a live Colab runtime. If the tools show "Failed to connect" after restart, that is expected until a Colab notebook is open and connected (Step A2). - -**Step A2 — Connect to a Colab runtime** - -1. Tell the user to open a new notebook at https://colab.research.google.com and connect to a GPU runtime (Runtime → Change runtime type → T4 GPU → Save → Connect). -2. Call the MCP tool `open_colab_browser_connection`. A browser window opens; the user clicks the auth link. The tool returns `true` when connected. - -**Step A3 — Setup: install Unsloth and verify GPU** - -Add a code cell with `scripts/colab_training.py::SETUP_CELL` content via `add_code_cell`, then run it with `run_code_cell`. +**Step A2 — Provision a GPU Runtime** -Parse the output — it prints a JSON line then `SETUP_OK`. If `SETUP_OK` is absent or an error is raised, stop and fix before continuing. - -**Step A4 — Verify: smoke-test all packages** - -Add a code cell with `scripts/colab_training.py::VERIFY_CELL` content and run it. - -The output is a JSON dict with versions and VRAM. Check: -- `vram_gb >= 6` (T4 = 15 GB, L4 = 22 GB — should pass) -- All package versions are present -- Output ends with `VERIFY_OK` - -Show the user the GPU name and VRAM, then proceed. - -**Step A5 — Generate and start training** +Provision a remote T4 GPU runtime: +```bash +colab new -s unsloth-buddy --gpu T4 +``` +You can also choose other runtimes like `A100` or `L4` if your account has credits. -Call `scripts/colab_training.py::get_training_cell(...)` with the parameters from the Phase 1 interview. Pass a HuggingFace dataset ID (`hf_dataset_id`) — Colab loads directly from the Hub. +**Step A3 — Setup environment on VM** -Add the returned code as a cell via `add_code_cell` and run it. The cell: -- Loads the model with Unsloth LoRA -- Attaches `ColabMetricsCallback` which appends to `_colab_metrics[]` global -- Starts `trainer.train()` in a background daemon thread -- Prints `TRAINING_STARTED: ` immediately and returns +Upload and run the Colab setup script on the remote VM: +```bash +colab exec -s unsloth-buddy -f scripts/setup_colab.py +``` +This automatically: +1. Detects the assigned GPU (T4/L4/A100) +2. Installs Unsloth and required packages on the VM +3. Prints a structured JSON status to verify readiness -Parse the `TRAINING_STARTED:` line to confirm training began. +**Step A4 — Verify GPU & Packages** -**Step A6 — Monitor training loop** +Verify setup output. Ensure the printed JSON status is `"status": "ready"`. -Every 30 seconds, update the poll cell with `scripts/colab_training.py::POLL_CELL` content (or add once and re-run it) via `run_code_cell`. +**Step A5 — Generate training script and run** -The output is a line beginning `POLL: ` with: -```json -{"done": false, "n_logs": 12, "latest_step": 60, "latest_loss": 1.42, "recent": [...], "error": null} +Assemble your training script as `train.py` inside the dated project directory, containing the SFTTrainer or GRPO configuration. +Execute the script synchronously inside the remote session: +```bash +colab exec -s unsloth-buddy -f train.py ``` +Because the training runs synchronously on the remote VM, the CLI logs/progress are streamed directly to your terminal. -Report progress to the user each poll. Stop looping when `done: true`. If `error` is non-null, report it and stop. - -**Step A7 — Fetch final results** - -Add a code cell with `scripts/colab_training.py::FINAL_CELL` content and run it. +**Step A6 — Retrieve results** -The output starts with `FINAL: ` containing `final_loss`, `total_steps`, and `adapter_files` (paths to `.safetensors` in `/content/outputs/`). - -Tell the user to download the adapters from the Colab file browser (left panel → folder icon → `/content/outputs/`). +Download the output adapters/checkpoints from the VM to your local project directory: +```bash +colab download -s unsloth-buddy /content/outputs/ outputs/ +``` Update `progress_log.md` and `memory.md` with final loss, GPU used, and adapter location. @@ -277,13 +236,13 @@ Only proceed when Stage 2 prints **"READY FOR TRAINING"**. **Apple Silicon users**: You have two training paths available: - **Local mlx-tune** (default) — best for models ≤8B, fast iteration, no internet needed. Use Path C. -- **Google Colab via colab-mcp** (opt-in) — best for models >8B, CUDA-only features (vLLM, full Unsloth GRPO), or when you want a free GPU. Use Path E. Requires `colab-mcp` configured in MCP settings. +- **Google Colab via Colab CLI** (opt-in) — best for models >8B, CUDA-only features (vLLM, full Unsloth GRPO), or when you want a free GPU. Use Path E. Requires `google-colab-cli` installed locally. Ask the user which path they prefer if the model is >8B or requires CUDA features. ### Phase 4: Code Generation & Execution -**If using Colab (Path A):** Phases A5–A7 above already cover training and monitoring. Skip to Phase 5 once `FINAL_CELL` returns successfully. +**If using Colab (Path A):** Steps A5–A6 above already cover training and retrieval. Skip to Phase 5 once `colab exec` finishes and adapters are downloaded. **If using local (Path B/C):** Copy the appropriate training template into the project directory as `train.py`, then customise the top-level config variables — do NOT generate from scratch: - **Apple Silicon — SFT/Vision (mlx-tune)**: @@ -363,7 +322,13 @@ Copy the eval template into the project and configure it: cp ./scripts/mlx_eval_template.py eval.py # Apple Silicon # or: cp ./scripts/eval_template.py eval.py # Linux/CUDA ``` -Edit the top-level config vars (MODEL_NAME, ADAPTER_PATH, STYLE) to match training, then run **both modes** in sequence: +Before running it, **read `eval.py` carefully and customise it for this project**: +- Set the top-level config vars (`MODEL_NAME`, `ADAPTER_PATH`, prompt style, generation settings) to match training. +- Replace the default `TEST_PROMPTS` / sample selection with 4–8 project-specific held-out test cases. These should come from the project brief, data strategy, validation rows, or realistic user workflows — not generic prompts. +- Include edge cases that verify the trained behavior: expected format, domain vocabulary, refusal/uncertainty behavior, short vs long inputs, and at least one case likely to expose a regression. +- For vision eval, map the dataset image/label columns and choose representative validation images before running. + +After the test cases are customised, run **both modes automatically** in sequence: ```bash # 1. Standard batch eval python eval.py 2>&1 | tee logs/eval.log @@ -374,7 +339,7 @@ python eval.py --compare 2>&1 | tee logs/eval_compare.log **Critical — Apple Silicon / mlx-tune:** `ADAPTER_PATH` in `eval.py` must be the full relative path to the adapters directory (e.g. `"outputs/adapters"`). Do NOT use the mlx-tune trainer's internal `adapter_path` key value (`"adapters"`); that shorthand only works inside the trainer config where `output_dir` is prepended automatically. `FastLanguageModel.from_pretrained(adapter_path=...)` expects the actual path. -Record the qualitative results in `memory.md`. +Record the exact test cases and qualitative results in `memory.md`. **→ After Phase 5: update `gaslamp.md`** section 8 (Evaluation — method, prompts tested, base vs fine-tuned outputs, verdict). Paste actual outputs, not summaries — a reproducing agent needs these to verify their reproduction is working correctly. @@ -404,11 +369,14 @@ Ask the user their deployment target. Run export commands from within the projec **→ After Phase 6: update `gaslamp.md`** section 10 (Export — format, why, output path, run command). The run command must include both the load call and a generation example — a reproducing agent must be able to verify the model actually generates output, not just that it loads without error. -### Phase 6.5: Local Deploy & Test (Optional — requires llama.cpp) +### Phase 6.5: Local Deploy & Test (Optional — requires llama.cpp or LiteRT-LM) +Depending on your model export choice (GGUF/llama.cpp or .litertlm), you can run a local served deploy: + +#### Option A: GGUF Deployment (via llama.cpp) If llama.cpp is installed (detected in Phase 3 via `detect_system.py`), offer the user a one-command deploy after GGUF export: -> *"GGUF export is ready. Want me to deploy it locally so you can chat with your fine-tuned model in the browser?"* +> *"GGUF export is ready. Want me to deploy it locally so you can chat with your fine-tuned model?"* If yes, run the auto-deploy pipeline: ```bash @@ -423,9 +391,7 @@ This single command: 3. **Starts** an OpenAI-compatible server (`llama-server`) on port 8081 4. **Opens** the Gaslamp Chat WebUI (`templates/chat_ui.html`) in the browser -The user is chatting with their fine-tuned model within ~60 seconds of saying "yes". - -Individual subcommands also available for advanced users: +Individual subcommands also available: ```bash python scripts/llamacpp.py install # install llama.cpp python scripts/llamacpp.py quantize --input model.gguf --types q4_k_m q8_0 @@ -435,7 +401,37 @@ python scripts/llamacpp.py serve --model model-q4_k_m.gguf --port 8081 python scripts/llamacpp.py chat --model model-q4_k_m.gguf ``` -If llama.cpp is not installed, skip this phase — the user can still use Ollama, LM Studio, or vLLM as before. +#### Option B: LiteRT-LM Deployment +If the target format is `.litertlm` (converted via Google AI Edge pipelines), offer the user a LiteRT-LM local server/deploy command. Only use this path when the project already has a TFLite model plus a SentencePiece tokenizer, or an existing `.litertlm` bundle. Standard LoRA adapters, GGUF files, and merged safetensors are not valid LiteRT-LM inputs. + +> *".litertlm bundle is ready. Want me to deploy it locally so you can chat with your fine-tuned model?"* + +If yes, run the auto-deploy pipeline: +```bash +python scripts/litertlm.py deploy \ + --tflite outputs/model.tflite \ + --tokenizer outputs/tokenizer.model \ + --output outputs/model.litertlm +``` +If the `.litertlm` bundle already exists, serve it directly: +```bash +python scripts/litertlm.py serve --model outputs/model.litertlm --port 8082 +``` + +This single command: +1. **Bundles** the TFLite model and SentencePiece tokenizer into a `.litertlm` package. +2. **Starts** a local OpenAI-compatible server wrapping `litert_lm.Engine` on port 8082. +3. **Opens** the WebUI (`templates/chat_ui.html?port=8082`) in your default browser. + +Individual subcommands also available: +```bash +python scripts/litertlm.py install # install litert-lm-api & litert-lm-builder +python scripts/litertlm.py bundle --tflite model.tflite --tokenizer tokenizer.model --output model.litertlm +python scripts/litertlm.py serve --model outputs/model.litertlm --port 8082 +python scripts/litertlm.py chat --model outputs/model.litertlm +``` + +If neither is installed, skip this phase — the model can still be served via Ollama or vLLM. **→ After Phase 6.5: update `gaslamp.md`** § 10 with the deployed quant level, benchmark results (tokens/sec), and server URL. @@ -577,47 +573,30 @@ docker run -d -p 8888:8888 -v $(pwd):/workspace/work --gpus all unsloth/unsloth ``` Tell them to access Jupyter Lab at `http://localhost:8888`. -**E. Google Colab via colab-mcp (Remote GPU for Mac Users)**: +**E. Google Colab via Colab CLI (Remote GPU for Mac Users)**: This path gives Apple Silicon users (or anyone without a local NVIDIA GPU) access to free Colab GPUs (T4/L4/A100) while keeping the local project structure intact. **Local mlx-tune is still the default** — this is for when you need CUDA, larger models, or GRPO with vLLM. **Prerequisites:** -1. Install `uv`: `pip install uv` -2. Configure colab-mcp in your MCP settings (`.gemini/settings.json` or equivalent): -```json -{ - "mcpServers": { - "colab-mcp": { - "command": "uvx", - "args": ["git+https://github.com/googlecolab/colab-mcp"], - "timeout": 30000 - } - } -} -``` -3. Google account with Colab access +1. Install `google-colab-cli` via `uv tool install google-colab-cli` +2. Google account with Colab access **Setup steps:** -1. Use colab-mcp's `execute_code` tool to run `scripts/setup_colab.py` on the Colab VM: -```python -# The agent reads setup_colab.py and sends it via execute_code -from scripts.colab_training import generate_setup_code -code = generate_setup_code() -# → execute via colab-mcp execute_code tool -``` -2. Verify the JSON output shows `"status": "ready"` and a GPU is detected. -3. Upload your dataset and training script (see Phase 4 Colab workflow). -4. Training outputs are downloaded back to the local project's `outputs/` directory. +1. Spin up a new VM session: `colab new -s unsloth-buddy --gpu T4` +2. Execute setup script: `colab exec -s unsloth-buddy -f scripts/setup_colab.py` +3. Confirm VM shows ready GPU and Unsloth package installations. +4. Train script execution from the dated project directory: `colab exec -s unsloth-buddy -f train.py` +5. Sync adapters back: `colab download -s unsloth-buddy /content/outputs/ outputs/` **When to suggest this path:** - User is on Apple Silicon and needs a model >8B parameters - User needs CUDA-exclusive features (vLLM fast inference, FP8 quantization) - User wants GRPO with vLLM generation (requires CUDA) -- User's local machine doesn't have enough RAM for the desired model +- Agent needs automated workflow on remote cloud GPU without local CUDA setups **Helper scripts:** - `scripts/setup_colab.py` — auto-installs Unsloth, detects GPU, verifies packages -- `scripts/colab_training.py` — code generators for upload, train, download, and metrics polling +- `scripts/colab_training.py` — code templates for remote training VM execution --- diff --git a/scripts/colab_training.py b/scripts/colab_training.py index 9641500..260b8dc 100644 --- a/scripts/colab_training.py +++ b/scripts/colab_training.py @@ -1,16 +1,14 @@ """ -colab_training.py — Code templates for training on Google Colab via colab-mcp. - -Each constant / function returns a Python code string to pass to the -`execute_code` MCP tool. The training cell runs the trainer in a background -thread so execute_code returns immediately; use POLL_CELL to track progress. - -Typical call sequence: - 1. execute_code(SETUP_CELL) # install unsloth, check GPU - 2. execute_code(VERIFY_CELL) # smoke-test imports + VRAM - 3. execute_code(get_training_cell(...)) # start training in background - 4. loop: execute_code(POLL_CELL) # monitor until done: true - 5. execute_code(FINAL_CELL) # fetch final metrics + adapter path +colab_training.py — Code templates for training on Google Colab via google-colab-cli. + +Each constant / function returns a Python code string or template to execute on +the remote Colab runtime VM using the `colab exec` command. + +Typical CLI workflow: + 1. colab new -s unsloth-buddy --gpu T4 + 2. colab exec -s unsloth-buddy -f scripts/setup_colab.py + 3. colab exec -s unsloth-buddy -f train.py + 4. colab download -s unsloth-buddy /content/outputs/ outputs/ """ # ── Step 1: Install & GPU check ─────────────────────────────────────────────── @@ -202,37 +200,22 @@ def _run(): print("FINAL: " + json.dumps(summary)) """ -# ── Colab MCP installation instructions (printed to user) ──────────────────── +# ── Google Colab CLI installation instructions (printed to user) ───────────── INSTALL_INSTRUCTIONS = """ -To use Google Colab for training, install colab-mcp in Claude Code: - -1. Install Python 3.13 (colab-mcp requires it; keeps your training venv intact): - uv python install 3.13 - -2. Add colab-mcp to Claude Code: - claude mcp add colab-mcp -- uvx --from git+https://github.com/googlecolab/colab-mcp --python 3.13 colab-mcp - -3. Open ~/.claude.json, find the colab-mcp entry under your project's - mcpServers, and make sure it looks like: - "colab-mcp": { - "command": "uvx", - "args": ["--from", "git+https://github.com/googlecolab/colab-mcp", - "--python", "3.13", "colab-mcp"], - "timeout": 30000 - } +To use Google Colab for remote training, install the official Colab CLI: - Note: do NOT add --enable-runtime — proxy mode is correct and - --enable-runtime requires a Google OAuth config not publicly available. +1. Install the CLI using uv: + uv tool install google-colab-cli -3. Restart Claude Code. +2. Provision a new remote GPU runtime: + colab new -s unsloth-buddy --gpu T4 -4. Open a new Colab notebook at https://colab.research.google.com - and connect to a GPU runtime (Runtime → Change runtime type → T4 GPU). +3. Run the remote environment setup: + colab exec -s unsloth-buddy -f scripts/setup_colab.py -5. Confirm the MCP tools are available — you should see: - - execute_code - - open_colab_browser_connection +4. Execute your training script from the dated project directory: + colab exec -s unsloth-buddy -f train.py - Note: "Failed to connect" before opening a Colab notebook is normal. - The tools become active once a runtime is connected (Step 4). +5. Download output model assets back to your local machine: + colab download -s unsloth-buddy /content/outputs/ outputs/ """ diff --git a/scripts/detect_env.py b/scripts/detect_env.py index d2cea17..be2b355 100644 --- a/scripts/detect_env.py +++ b/scripts/detect_env.py @@ -132,6 +132,10 @@ def detect_env_type(): "safetensors": check_pkg("safetensors"), "huggingface_hub": check_pkg("huggingface_hub"), } +optional_deploy_packages = { + "litert_lm": check_pkg("litert_lm"), + "litert_lm_builder": check_pkg("litert_lm_builder"), +} # ── HF cache & disk ─────────────────────────────────────────────────────────── hf_cache = Path(os.environ.get("HF_HOME", Path.home() / ".cache" / "huggingface")) / "hub" @@ -210,6 +214,13 @@ def detect_env_type(): flag = "" if ver else f" ← {install_hint.format(pkg=pkg)}" print(f" {pkg:<20}: {status}{flag}") +print(f"\nOptional deploy packages:") +for pkg, ver in optional_deploy_packages.items(): + status = ver or "not installed" + pkg_install = "litert-lm-api" if pkg == "litert_lm" else "litert-lm-builder" + flag = "" if ver else f" ← optional: {install_hint.format(pkg=pkg_install)}" + print(f" {pkg:<20}: {status}{flag}") + print(f"\nHF cache : {'OK' if hf_cache_ok else 'MISSING'} ({hf_cache})") print(f"Disk free : {disk_free_gb} GB") @@ -234,6 +245,7 @@ def detect_env_type(): "backend": backend, "versions": {"unsloth": unsloth_ver, "mlx_tune": mlx_tune_ver, "mlx": mlx_ver, "torch": torch_ver, **packages}, + "optional_deploy_versions": optional_deploy_packages, "cuda_available": cuda_avail, "mps_available": mps_avail, "hf_cache": str(hf_cache), "hf_cache_ok": hf_cache_ok, "disk_free_gb": disk_free_gb, diff --git a/scripts/detect_system.py b/scripts/detect_system.py index 4f05bd4..d5c3a0d 100644 --- a/scripts/detect_system.py +++ b/scripts/detect_system.py @@ -115,6 +115,19 @@ def check_python(cmd): if _ver_out: llamacpp_version = _ver_out.splitlines()[0].strip() +# ── LiteRT-LM detection ────────────────────────────────────────────────────── +litertlm_bins = {} +for _bin in ["litert-lm", "litert-lm-peek", "litert-lm-builder"]: + _path = shutil.which(_bin) + if _path: + litertlm_bins[_bin] = _path +litertlm_installed = len(litertlm_bins) > 0 +litertlm_version = "" +if "litert-lm" in litertlm_bins: + _ver_out = run(f"{litertlm_bins['litert-lm']} --version 2>&1") + if _ver_out: + litertlm_version = _ver_out.splitlines()[0].strip() + # ── Decision logic ──────────────────────────────────────────────────────────── if is_apple_silicon: install_path = "C" # mlx-tune @@ -148,6 +161,10 @@ def check_python(cmd): print(f"llama.cpp : {llamacpp_version or 'installed'} ({len(llamacpp_bins)} binaries)") else: print(f"llama.cpp : not installed (optional — run: python {__file__.replace('detect_system.py', 'llamacpp.py')} install)") +if litertlm_installed: + print(f"LiteRT-LM : {litertlm_version or 'installed'} ({len(litertlm_bins)} binaries)") +else: + print(f"LiteRT-LM : not installed (optional — run: python {__file__.replace('detect_system.py', 'litertlm.py')} install)") print(f"\n→ Recommended install path : {install_path}") print(f"→ Recommended Python : {recommended_python}") @@ -185,6 +202,11 @@ def check_python(cmd): "version": llamacpp_version, "binaries": llamacpp_bins, }, + "litert_lm": { + "installed": litertlm_installed, + "version": litertlm_version, + "binaries": litertlm_bins, + }, "install_path": install_path, "recommended_python": recommended_python, } diff --git a/scripts/litertlm.py b/scripts/litertlm.py new file mode 100644 index 0000000..9d110a5 --- /dev/null +++ b/scripts/litertlm.py @@ -0,0 +1,571 @@ +#!/usr/bin/env python3 +""" +litertlm.py — Unified LiteRT-LM CLI for unsloth-buddy. + +Subcommands: + install Install/upgrade litert-lm-api and litert-lm-builder + bundle Bundle a TFLite model, tokenizer, and metadata into a .litertlm file + serve Start a local OpenAI-compatible API server + open chat WebUI + chat Interactive terminal chat session with a .litertlm model + deploy Auto-pipeline: bundle → serve + +Usage: + python scripts/litertlm.py install + python scripts/litertlm.py bundle --tflite model.tflite --tokenizer tokenizer.model --output model.litertlm + python scripts/litertlm.py serve --model model.litertlm --port 8082 +""" + +import argparse +import json +import os +import shutil +import signal +import subprocess +import sys +import textwrap +import time +import webbrowser +from http.server import HTTPServer, BaseHTTPRequestHandler +from typing import Optional + +# ─── Constants ────────────────────────────────────────────────────── +LITERT_BINS = ["litert-lm", "litert-lm-peek", "litert-lm-builder"] + +# ─── Utility ──────────────────────────────────────────────────────── + +def _find_bin(name: str) -> Optional[str]: + """Locate a LiteRT-LM binary on PATH.""" + return shutil.which(name) + + +def _require_bin(name: str) -> str: + """Return path to binary or exit with helpful error.""" + path = _find_bin(name) + if not path: + print(f"❌ '{name}' not found on PATH.") + print(f" Run: python {__file__} install") + sys.exit(1) + return path + + +def _run(cmd: list[str], capture=False, check=True, **kwargs): + """Run a subprocess, printing the command first.""" + print(f" ▸ {' '.join(cmd)}") + if capture: + return subprocess.run(cmd, capture_output=True, text=True, check=check, **kwargs) + return subprocess.run(cmd, check=check, **kwargs) + + +def _file_size_mb(path: str) -> float: + return os.path.getsize(path) / (1024 * 1024) + + +def _detect_litert() -> dict: + """Detect installed LiteRT-LM binaries.""" + info = {"installed": False, "version": None, "binaries": {}} + for name in LITERT_BINS: + path = _find_bin(name) + if path: + info["binaries"][name] = path + info["installed"] = True + # Try to get version + cli = info["binaries"].get("litert-lm") + if cli: + try: + r = subprocess.run([cli, "--version"], capture_output=True, text=True, timeout=5) + # litert-lm output might contain version number + lines = (r.stdout + r.stderr).splitlines() + if lines: + info["version"] = lines[0].strip() + except Exception: + pass + return info + + +# ─── Local HTTP Server for serving .litertlm ──────────────────────── + +class OpenAICompatibleHandler(BaseHTTPRequestHandler): + def log_message(self, format, *args): + # Mute standard http logs to avoid output spam + pass + + def send_cors_headers(self): + self.send_header('Access-Control-Allow-Origin', '*') + self.send_header('Access-Control-Allow-Methods', 'POST, GET, OPTIONS') + self.send_header('Access-Control-Allow-Headers', 'Content-Type, Authorization') + + def do_OPTIONS(self): + self.send_response(200) + self.send_cors_headers() + self.end_headers() + + def do_GET(self): + if self.path == "/v1/models": + self.send_response(200) + self.send_header('Content-type', 'application/json') + self.send_cors_headers() + self.end_headers() + model_id = os.path.basename(self.server.model_path) if hasattr(self.server, 'model_path') else "litert-lm" + resp = { + "object": "list", + "data": [{ + "id": model_id, + "object": "model", + "created": int(time.time()), + "owned_by": "litert-lm" + }] + } + self.wfile.write(json.dumps(resp).encode('utf-8')) + elif self.path == "/health": + self.send_response(200) + self.send_header('Content-type', 'application/json') + self.send_cors_headers() + self.end_headers() + self.wfile.write(json.dumps({"status": "ready"}).encode('utf-8')) + else: + self.send_response(404) + self.send_cors_headers() + self.end_headers() + + def do_POST(self): + if self.path == "/v1/chat/completions": + content_length = int(self.headers['Content-Length']) + post_data = self.rfile.read(content_length) + req = {} + try: + req = json.loads(post_data.decode('utf-8')) + except Exception as e: + self.send_response(400) + self.send_cors_headers() + self.end_headers() + self.wfile.write(json.dumps({"error": f"Invalid JSON: {e}"}).encode('utf-8')) + return + + messages = req.get("messages", []) + stream = req.get("stream", False) + user_msg = "" + for m in messages: + if m.get("role") == "user": + user_msg = m.get("content", "") + + # Run inference + if not hasattr(self.server, 'engine') or self.server.engine is None: + self.send_response(500) + self.send_header('Content-Type', 'application/json') + self.send_cors_headers() + self.end_headers() + self.wfile.write(json.dumps({"error": "LiteRT-LM engine is not initialized."}).encode('utf-8')) + return + + try: + if stream: + self.send_response(200) + self.send_header('Content-Type', 'text/event-stream') + self.send_header('Cache-Control', 'no-cache') + self.send_header('Connection', 'keep-alive') + self.send_cors_headers() + self.end_headers() + + # Create a new conversation per request to avoid context blending + with self.server.engine.create_conversation() as conv: + for chunk in conv.send_message_async(user_msg): + text = chunk.get("content", [{}])[0].get("text", "") + event_data = { + "choices": [{ + "delta": {"content": text}, + "index": 0, + "finish_reason": None + }] + } + self.wfile.write(f"data: {json.dumps(event_data)}\n\n".encode('utf-8')) + self.wfile.flush() + + # Send done signal + done_event = { + "choices": [{ + "delta": {}, + "index": 0, + "finish_reason": "stop" + }] + } + self.wfile.write(f"data: {json.dumps(done_event)}\n\ndata: [DONE]\n\n".encode('utf-8')) + self.wfile.flush() + else: + # Non-streamed response + response_text = "" + with self.server.engine.create_conversation() as conv: + for chunk in conv.send_message_async(user_msg): + response_text += chunk.get("content", [{}])[0].get("text", "") + + resp = { + "choices": [{ + "message": { + "role": "assistant", + "content": response_text + }, + "finish_reason": "stop", + "index": 0 + }], + "object": "chat.completion", + "model": os.path.basename(self.server.model_path) + } + self.send_response(200) + self.send_header('Content-Type', 'application/json') + self.send_cors_headers() + self.end_headers() + self.wfile.write(json.dumps(resp).encode('utf-8')) + except Exception as e: + # Handle error + if not stream: + self.send_response(500) + self.send_header('Content-Type', 'application/json') + self.send_cors_headers() + self.end_headers() + self.wfile.write(json.dumps({"error": f"Inference engine failure: {e}"}).encode('utf-8')) + else: + # In a stream, write error chunk and end stream + err_event = { + "choices": [{ + "delta": {"content": f"\n[Backend Error: {e}]"}, + "index": 0, + "finish_reason": "stop" + }] + } + try: + self.wfile.write(f"data: {json.dumps(err_event)}\n\ndata: [DONE]\n\n".encode('utf-8')) + self.wfile.flush() + except Exception: + pass + else: + self.send_response(404) + self.send_cors_headers() + self.end_headers() + + +# ─── Subcommands ──────────────────────────────────────────────────── + +def cmd_install(args): + """Install/upgrade LiteRT-LM framework packages.""" + print("⏳ Checking/Installing LiteRT-LM packages...") + packages = ["litert-lm-api", "litert-lm-builder"] + + # Check if uv is available + if shutil.which("uv"): + print("⚡ Found uv package manager. Using 'uv pip install'...") + cmd = ["uv", "pip", "install", "--upgrade"] + packages + else: + print("📦 Using standard 'pip install'...") + cmd = [sys.executable, "-m", "pip", "install", "--upgrade"] + packages + + try: + subprocess.run(cmd, check=True) + print("✅ Successfully installed LiteRT-LM packages!") + except subprocess.CalledProcessError as e: + print(f"❌ Package installation failed: {e}") + sys.exit(1) + + +def cmd_bundle(args): + """Bundle TFLite flatbuffer, tokenizer, and metadata into a .litertlm file.""" + builder_bin = _require_bin("litert-lm-builder") + + if not os.path.isfile(args.tflite): + print(f"❌ TFLite model file not found: {args.tflite}") + sys.exit(1) + if not os.path.isfile(args.tokenizer): + print(f"❌ Tokenizer file not found: {args.tokenizer}") + sys.exit(1) + + out_dir = os.path.dirname(args.output) + if out_dir: + os.makedirs(out_dir, exist_ok=True) + + print(f"📦 Bundling model assets into: {args.output}") + print(f" - TFLite Model : {args.tflite} ({_file_size_mb(args.tflite):.1f} MB)") + print(f" - Tokenizer : {args.tokenizer}") + + # Construct Builder Command + # litert-lm-builder system_metadata --str Authors "ODML Team" tflite_model --path --model_type sp_tokenizer --path output --path + cmd = [ + builder_bin, + "system_metadata", "--str", "Authors", args.author, + "tflite_model", "--path", args.tflite, "--model_type", args.model_type, + "sp_tokenizer", "--path", args.tokenizer, + "output", "--path", args.output + ] + + try: + _run(cmd, capture=False) + print(f"✅ Created .litertlm model: {args.output} ({_file_size_mb(args.output):.1f} MB)") + except subprocess.CalledProcessError as e: + print(f"❌ Bundling failed: {e}") + sys.exit(1) + + +def cmd_serve(args): + """Start local HTTP server wrapping litert_lm.Engine + open chat WebUI.""" + if not os.path.isfile(args.model): + print(f"❌ Model file not found: {args.model}") + sys.exit(1) + + print(f"🚀 Loading LiteRT-LM Engine with: {args.model}...") + + # Load package dynamically + try: + import litert_lm + except ImportError: + print("❌ 'litert_lm' package not found in Python path.") + print(f" Run: python {__file__} install") + sys.exit(1) + + # Select backend + backend_choice = args.backend.lower() + if backend_choice == "gpu": + backend = litert_lm.Backend.GPU() + elif backend_choice == "npu": + backend = litert_lm.Backend.NPU() + else: + backend = litert_lm.Backend.CPU() + + try: + litert_lm.set_min_log_severity(litert_lm.LogSeverity.ERROR) + except Exception: + pass + + try: + engine = litert_lm.Engine(args.model, backend=backend) + except Exception as e: + print(f"❌ Failed to initialize LiteRT-LM Engine: {e}") + sys.exit(1) + + # Spin up server + server = HTTPServer(('localhost', args.port), OpenAICompatibleHandler) + server.engine = engine + server.model_path = args.model + # Cache a single conversation just in case, though handler creates new ones + server.conversation = engine.create_conversation() + + print(f"✅ LiteRT-LM server running locally at http://localhost:{args.port}") + print(f" API: http://localhost:{args.port}/v1/chat/completions") + + # Find Chat WebUI + chat_ui_path = _find_chat_ui() + if chat_ui_path and not args.no_open: + url = f"file://{chat_ui_path}?port={args.port}" + print(f" WebUI: {url}") + webbrowser.open(url) + else: + print(f" WebUI: Open templates/chat_ui.html in browser and specify port {args.port}") + + def _sigint_handler(sig, frame): + print("\n🛑 Stopping server...") + try: + server.conversation.close() + except Exception: + pass + server.server_close() + sys.exit(0) + + signal.signal(signal.SIGINT, _sigint_handler) + + print("\n Press Ctrl+C to stop.") + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + try: + server.conversation.close() + except Exception: + pass + server.server_close() + + +def cmd_chat(args): + """Direct terminal chat session with .litertlm model.""" + if not os.path.isfile(args.model): + print(f"❌ Model file not found: {args.model}") + sys.exit(1) + + # Load package dynamically + try: + import litert_lm + except ImportError: + print("❌ 'litert_lm' package not found in Python path.") + print(f" Run: python {__file__} install") + sys.exit(1) + + # Select backend + backend_choice = args.backend.lower() + if backend_choice == "gpu": + backend = litert_lm.Backend.GPU() + elif backend_choice == "npu": + backend = litert_lm.Backend.NPU() + else: + backend = litert_lm.Backend.CPU() + + try: + litert_lm.set_min_log_severity(litert_lm.LogSeverity.ERROR) + except Exception: + pass + + print(f"⚡ Loading model into LiteRT-LM Engine...") + try: + with litert_lm.Engine(args.model, backend=backend) as engine: + with engine.create_conversation() as conv: + print("✅ Model loaded successfully!") + print("💬 Enter chat session. Press Ctrl+C or type 'exit' to quit.\n") + + while True: + try: + user_input = input("\033[92mUser > \033[0m").strip() + if not user_input: + continue + if user_input.lower() in ("exit", "quit"): + break + + print("\033[94mLiteRT-LM > \033[0m", end="", flush=True) + for chunk in conv.send_message_async(user_input): + text = chunk.get("content", [{}])[0].get("text", "") + print(text, end="", flush=True) + print("\n") + except KeyboardInterrupt: + print("\n👋 Exiting chat...") + break + except Exception as e: + print(f"\n❌ Inference error: {e}\n") + except Exception as e: + print(f"❌ Failed to run model: {e}") + sys.exit(1) + + +def cmd_deploy(args): + """Auto-pipeline: bundle → serve.""" + if args.tflite.endswith(".litertlm"): + if not os.path.isfile(args.tflite): + print(f"❌ LiteRT-LM model file not found: {args.tflite}") + sys.exit(1) + print("📌 Model argument is already a .litertlm file. Direct serving...") + serve_args = argparse.Namespace( + model=args.tflite, + port=args.port, + backend=args.backend, + no_open=args.no_open + ) + cmd_serve(serve_args) + return + + if not os.path.isfile(args.tflite): + print(f"❌ TFLite model file not found: {args.tflite}") + sys.exit(1) + if not args.tokenizer: + print("❌ --tokenizer is required when bundling a .tflite model.") + print(" To serve an existing bundle, run: python scripts/litertlm.py serve --model outputs/model.litertlm") + sys.exit(1) + + # 1. Bundle + print(f"\n{'━'*60}\n Step 1/2: Bundling TFLite assets\n{'━'*60}") + bundle_args = argparse.Namespace( + tflite=args.tflite, + tokenizer=args.tokenizer, + output=args.output, + author=args.author, + model_type=args.model_type + ) + cmd_bundle(bundle_args) + + # 2. Serve + print(f"\n{'━'*60}\n Step 2/2: Serving model\n{'━'*60}") + serve_args = argparse.Namespace( + model=args.output, + port=args.port, + backend=args.backend, + no_open=args.no_open + ) + cmd_serve(serve_args) + + +# ─── Chat UI Finder ───────────────────────────────────────────────── + +def _find_chat_ui() -> Optional[str]: + """Locate templates/chat_ui.html relative to this script or cwd.""" + candidates = [ + os.path.join(os.path.dirname(__file__), "..", "templates", "chat_ui.html"), + os.path.join(os.getcwd(), "templates", "chat_ui.html"), + os.path.join(os.path.dirname(__file__), "templates", "chat_ui.html"), + ] + for c in candidates: + if os.path.isfile(c): + return os.path.abspath(c) + return None + + +# ─── CLI Parser ───────────────────────────────────────────────────── + +def build_parser(): + p = argparse.ArgumentParser( + prog="litertlm", + description="Unified LiteRT-LM CLI for unsloth-buddy", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent("""\ + Quick start: + %(prog)s install + %(prog)s deploy --tflite model.tflite --tokenizer tokenizer.model --output model.litertlm + %(prog)s serve --model model.litertlm + """), + ) + sub = p.add_subparsers(dest="command", required=True) + + # install + sub.add_parser("install", help="Install LiteRT-LM components via pip") + + # bundle + sb = sub.add_parser("bundle", help="Package TFLite assets into a .litertlm file") + sb.add_argument("--tflite", required=True, help="Path to input .tflite model file") + sb.add_argument("--tokenizer", required=True, help="Path to SentencePiece tokenizer file (.model)") + sb.add_argument("--output", required=True, help="Path to output .litertlm file") + sb.add_argument("--author", default="ODML Team", help="Model author metadata") + sb.add_argument("--model-type", default="gemma", help="Model type metadata (e.g. gemma, gemma4)") + + # serve + ss = sub.add_parser("serve", help="Serve a .litertlm model over local OpenAI API") + ss.add_argument("--model", required=True, help="Path to .litertlm file") + ss.add_argument("--port", type=int, default=8082, help="Local HTTP server port") + ss.add_argument("--backend", default="cpu", choices=["cpu", "gpu", "npu"], help="Hardware acceleration backend") + ss.add_argument("--no-open", action="store_true", help="Do not auto-open chat UI browser") + + # chat + sc = sub.add_parser("chat", help="Start interactive terminal chat session") + sc.add_argument("--model", required=True, help="Path to .litertlm file") + sc.add_argument("--backend", default="cpu", choices=["cpu", "gpu", "npu"], help="Hardware backend") + + # deploy + sd = sub.add_parser("deploy", help="Unified bundle and serve pipeline") + sd.add_argument("--tflite", required=True, help="Path to .tflite model file (or serving .litertlm file)") + sd.add_argument("--tokenizer", help="Path to tokenizer file (required for bundling)") + sd.add_argument("--output", default="outputs/model.litertlm", help="Path to packaged .litertlm output") + sd.add_argument("--port", type=int, default=8082, help="Local HTTP server port") + sd.add_argument("--backend", default="cpu", choices=["cpu", "gpu", "npu"], help="Hardware backend") + sd.add_argument("--author", default="ODML Team", help="Model author metadata") + sd.add_argument("--model-type", default="gemma", help="Model type metadata") + sd.add_argument("--no-open", action="store_true", help="Do not auto-open WebUI browser") + + return p + + +def main(): + parser = build_parser() + args = parser.parse_args() + + dispatch = { + "install": cmd_install, + "bundle": cmd_bundle, + "serve": cmd_serve, + "chat": cmd_chat, + "deploy": cmd_deploy, + } + dispatch[args.command](args) + + +if __name__ == "__main__": + main() diff --git a/scripts/mlx_eval_template.py b/scripts/mlx_eval_template.py index 522159b..f31a95c 100644 --- a/scripts/mlx_eval_template.py +++ b/scripts/mlx_eval_template.py @@ -27,7 +27,9 @@ MAX_TOKENS = 200 TEMPERATURE = 0.7 # 0.0 = greedy / deterministic -# A few prompts representative of your training domain +# Replace these before running Phase 5. +# Use 4-8 held-out prompts that represent the actual task, audience, expected +# output format, and edge cases from project_brief.md / data_strategy.md. TEST_PROMPTS = [ "What is machine learning?", "Write a Python function that reverses a string.", diff --git a/scripts/mlx_eval_vision_template.py b/scripts/mlx_eval_vision_template.py index 5a5b0a6..75902cc 100644 --- a/scripts/mlx_eval_vision_template.py +++ b/scripts/mlx_eval_vision_template.py @@ -26,6 +26,10 @@ SYSTEM_PROMPT = "You are a helpful assistant." USER_PROMPT = "Describe this image." +# Before running Phase 5, edit DATASET_ID, SYSTEM_PROMPT, USER_PROMPT, and the +# image/label column mapping below. Use representative held-out validation +# images that test the project's real visual task and likely edge cases. + # ── 1. Fix Adapter JSON logic ─────────────────────────────────────────────── def patch_adapter_config(adapter_path): """ diff --git a/scripts/setup_colab.py b/scripts/setup_colab.py index 58df240..dfa03fb 100644 --- a/scripts/setup_colab.py +++ b/scripts/setup_colab.py @@ -1,15 +1,16 @@ """ setup_colab.py — Auto-setup Unsloth on a Google Colab VM. -Run this via colab-mcp's execute_code tool to: +Run this via google-colab-cli to: 1. Detect the assigned GPU (T4/L4/A100) 2. Install Unsloth 3. Verify all ML packages 4. Print a structured JSON status -Usage (from colab-mcp execute_code): - exec(open("setup_colab.py").read()) - # or paste directly into execute_code +Usage (using google-colab-cli): + colab exec -f scripts/setup_colab.py + # or: + colab run --gpu T4 scripts/setup_colab.py Output: JSON dict with keys: status: "ready" | "error" @@ -43,7 +44,7 @@ def detect_gpu(): import torch if torch.cuda.is_available(): name = torch.cuda.get_device_name(0) - vram = torch.cuda.get_device_properties(0).total_mem / (1024**3) + vram = torch.cuda.get_device_properties(0).total_memory / (1024**3) cuda_ver = torch.version.cuda or "unknown" return name, round(vram, 1), cuda_ver except ImportError: