Skip to content

[Train 3/3 — after #78] Schedule local SFT across the Grid - #79

Draft
deehw wants to merge 2 commits into
grid-train-goal-evalfrom
grid-train-jobs
Draft

[Train 3/3 — after #78] Schedule local SFT across the Grid#79
deehw wants to merge 2 commits into
grid-train-goal-evalfrom
grid-train-jobs

Conversation

@deehw

@deehw deehw commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What

  • add grid train submit-sft for a portable config + verified SFT dataset
  • reuse the durable task/Git plane; no second scheduler
  • advertise explicit train-mlx and train-torch worker capabilities
  • claim only on dependency-compatible nodes and return adapter artifacts in the result commit
  • recover a lost machine through the existing lease/reclaim path
  • give Train bounded per-job queue/run clocks instead of the ordinary one-hour agent-task limit
  • reject zero-exit false success unless a real non-empty adapter and matching SFT run record exist
  • add a portable SHA-256 result manifest and grid train verify-result

Honest scope

  • one trainer owns one job; this is node scheduling/failover, not cross-machine DDP
  • v1 restarts from immutable input after node loss rather than resuming optimizer state
  • training capacity is opt-in and absent from the default worker list
  • relay-first rollout; private relay companion PR required

Safety

  • shell-free, allowlisted job spec
  • isolated Python launch prevents a checkout from shadowing Grid's trainer package
  • config/data/result paths are confined to the task worktree
  • credentials require explicit passthrough
  • fetched results reject missing, added, symlinked, changed, or metadata-mismatched adapter files

Real acceptance

  • isolated self-hosted relay with production-like 120-second leases
  • real mlx-community/SmolLM2-135M-Instruct one-iteration SFT
  • task fbd1d906-5f9b-46d2-9e02-45ec9c59f843 completed on attempt 1
  • result commit 4ac1e59098076e820010b467139814939d20187b fetched into a clean client
  • 1,315,662-byte adapter independently verified from per-file SHA-256 hashes
  • controlled lease-loss tests prove attempt 2 keeps the identical immutable input commit on a different compatible provider

Validation

  • full public suite: 3,425 passed, 57 skipped, 7 deselected
  • focused Train suite: 7 passed
  • Ruff: clean
  • wheel built and inspected for the Train CLI and worker modules
  • private relay companion: final-base local + GitHub gates 326 passed each; shared tasks 129 passed; eligibility clocks 38 passed

Stack

Depends on #78, which depends on #77. Merge autonomous-ai/autonomous-grid-cli#22 before enabling training workers.

@deehw
deehw marked this pull request as draft September 3, 2026 00:53
@deehw deehw changed the title Grid Train: schedule local SFT across the Grid [Train 3/3 — after #78] Schedule local SFT across the Grid Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant