Skip to content

Commit 43ccd29

Browse files
authored
Trim trailing whitespace in docs/SWITCH_KERNELS.md (#153)
Four lines of the SWITCH kernels write-up (13, 14, 39 and 111) end in a single space. The trailing-whitespace hook rejects them, so the CI Pre-commit job fails on main and on every PR into main that does not fix them itself. The test-cpu job needs pre-commit, so the CPU tests are skipped on those PRs too. None of the four is a Markdown hard line break (that takes two spaces), so the rendered page is unchanged. Signed-off-by: AlonMalach <alonmalach@gmail.com>
1 parent 7e72456 commit 43ccd29

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

‎docs/SWITCH_KERNELS.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ rank-32 ones, and **up to 5.9× faster** with one rank-512 adapter. At idle (no
1010
## Motivation and Availability
1111

1212
As part of our mission to create a modular model architecture where adapter libraries are used to enhance
13-
a model, we encountered the problem that existing infrastructure for serving adapters introduced
14-
unnecessary performance overheads. These overheads are noticeable even when a single adapter, regardless of
13+
a model, we encountered the problem that existing infrastructure for serving adapters introduced
14+
unnecessary performance overheads. These overheads are noticeable even when a single adapter, regardless of
1515
its size, is introduced, but become very pronounced when the adapters have diverse sizes, which we anticipate will be a common
1616
scenario when a developer is integrating library components from different contributors into a single checkpoint.
1717

@@ -36,7 +36,7 @@ expensive in our setting.
3636
## What SWITCH does differently
3737

3838
SWITCH is named after its design: **S**witched delta, **W**-ext fusion, **I**n-place,
39-
**T**iled, **C**onditional skip, **H**eterogeneous rank.
39+
**T**iled, **C**onditional skip, **H**eterogeneous rank.
4040

4141
Existing Punica technology, as implemented in vLLM, exhibits the following properties:
4242

@@ -108,7 +108,7 @@ adapter. The figure below compares each system **as it currently implemented**:
108108

109109
![Idle vs adapter-active throughput at 12 adapters loaded, against the base model with LoRA off](figures/fig3_dispatch_3b_b32.png)
110110

111-
Of note is that the overhead at idle that the SWITCH implementation in Granite Switch and vLLM exhibit is unrelated to the SWITCH kernels themselves, and rather
111+
Of note is that the overhead at idle that the SWITCH implementation in Granite Switch and vLLM exhibit is unrelated to the SWITCH kernels themselves, and rather
112112
related to the fact that SWITCH is operating as a single computational graph, whereas vLLM does not run an adapter specific graph when no adapters are present. It would be relatively
113113
straightforward to implement SWITCH with two graphs, in which case the performance in the no adapter case would match vLLM/Punica exactly. As indicated earlier, we are currently focused in the setting where the model discovers which adapters need to be applied to which tokens through its switch layer, and currently there is no support in vLLM for adapting the computational graph once the model is already invoked.
114114

0 commit comments

Comments
 (0)