You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 43ccd29
Browse filesBrowse the repository at this point in the historyBrowse files
Trim trailing whitespace in docs/SWITCH_KERNELS.md (#153)
Four lines of the SWITCH kernels write-up (13, 14, 39 and 111) end in
a single space. The trailing-whitespace hook rejects them, so the CI
Pre-commit job fails on main and on every PR into main that does not
fix them itself. The test-cpu job needs pre-commit, so the CPU tests
are skipped on those PRs too.
None of the four is a Markdown hard line break (that takes two
spaces), so the rendered page is unchanged.
Signed-off-by: AlonMalach <alonmalach@gmail.com>
Existing Punica technology, as implemented in vLLM, exhibits the following properties:
42
42
@@ -108,7 +108,7 @@ adapter. The figure below compares each system **as it currently implemented**:
108
108
109
109

110
110
111
-
Of note is that the overhead at idle that the SWITCH implementation in Granite Switch and vLLM exhibit is unrelated to the SWITCH kernels themselves, and rather
111
+
Of note is that the overhead at idle that the SWITCH implementation in Granite Switch and vLLM exhibit is unrelated to the SWITCH kernels themselves, and rather
112
112
related to the fact that SWITCH is operating as a single computational graph, whereas vLLM does not run an adapter specific graph when no adapters are present. It would be relatively
113
113
straightforward to implement SWITCH with two graphs, in which case the performance in the no adapter case would match vLLM/Punica exactly. As indicated earlier, we are currently focused in the setting where the model discovers which adapters need to be applied to which tokens through its switch layer, and currently there is no support in vLLM for adapting the computational graph once the model is already invoked.
0 commit comments