Design: warm-session daemon - #4
Closed
sshin23 wants to merge 1 commit into
Closed
Conversation
Motivation measured on the group's GV100 box: a cold GPU run costs 244-308 s of which ~60 s is stack load and the rest first-solve JIT, against 0.4-0.7 s for the same solve warm. The design adds a WARM tier to the existing Model(core) resolve ladder: the client records (pure Python, verified to pickle round-trip), the daemon replays and keeps live instances keyed by the digests the cache already computes. Includes the measurement scripts, two pre-work items surfaced by them (device set_parameters bug, recording's backend guard in the wrong layer), phase plan with acceptance tests, risks, and open questions for review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 18, 2026
Member
Author
|
Keeping the design doc on the working branch rather than merging it; implementation proceeds on #6. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A design document only — no implementation. Motivated by measured cold-start costs on the group's GPU box (244–308 s to a first GPU solution vs 0.4–0.7 s warm; breakdown and scripts included in
design/daemon-measurements/).The proposal: a persistent
examodelssession that user scripts dispatch to automatically, by shipping the recorded model (never the script) as a new WARM tier in the existingModel(core)resolve ladder between the compiled-library HIT and the boot-Julia MISS. The record already exists, already fingerprints, and verifiably pickles without Julia in the process.Review requests, in order of leverage:
Corerecords when a daemon is reachable; byte-identical behavior when not — the unmodified test suite is the acceptance test).--detach, memory budget default, per-device daemons).set_parametersbug (reproduced; mechanism traced tocopyto!into a device view from aPyArray) and moving the recorder's CPU-only guard into the cache consumer.🤖 Generated with Claude Code