Skip to content

[rhaiis] add custom dspark spec-decoding preset #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aas008 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[rhaiis] add custom dspark spec-decoding preset #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a47a9b6
[rhaiis] orchestration: add DeepSeek-V4-Pro-0813 spec-decoding presets
aas008 Aug 19, 2026
be9efff
[rhaiis] presets: collect all profiles and vLLM profiler traces
aas008 Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions projects/rhaiis/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,11 @@ python3 -m projects.rhaiis.orchestration.cli test \
# deepseek-r1, deepseek-v3, gpt-oss
# Workload presets: profile1, profile2, profile3, profile4
# Accelerator presets: nvidia, amd
#
# Spec decoding comparison (H200 Zeus): presets.d/spec-decoding.yaml
# deepseek-v4-baseline | deepseek-v4-mtp | deepseek-v4-dspark | deepseek-v4-dspark-dynamic
python3 -m projects.rhaiis.orchestration.cli test \
--preset deepseek-v4-dspark
```

Per-cluster presets (e.g. `presets.d/mehulvalidation.yaml`) set cluster-specific
Expand Down
34 changes: 34 additions & 0 deletions projects/rhaiis/orchestration/config.d/models.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,40 @@ deepseek-v4-pro:
engine_args:
tensor-parallel-size: 8

deepseek-v4-pro-0813: &deepseek-v4-pro-0813

@Harshith-umesh Harshith-umesh Aug 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do the &, * < symbols here mean?

@aas008 aas008 Aug 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YAML anchors and aliases. I learnt its a way to write the shared model block once and reuse it. I used them here to keep the file succinct; & names the block, * copies it, and <<: merges it so the MTP/DSpark variants only add speculative-config.

@aas008 aas008 Aug 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In current architecture of this models.yaml, ever config would need a new entry. This will bloat up very very fast.

@kpouget kpouget Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will bloat up very very fast.

if you look how the presets work, something like:

common_values:
  abc: true

specific_values:
  extends: [common_values]
  efg: true

you can build a mechanism to avoid the config file size explosion. That may be more flexible than YAML anchors

or alternatively, if applicable, some default values that apply to all the models

default:
  engine_args:
    tensor-parallel-size: 8

name: DeepSeek-V4-Pro-0813
hf_model_id: deepseek-ai/DeepSeek-V4-Pro-0813
engine_args: &deepseek-v4-pro-0813-args
tensor-parallel-size: 8
enable-expert-parallel: true
kv-cache-dtype: fp8
max-model-len: 16384
max-num-seqs: 256
max-num-batched-tokens: 16384
gpu-memory-utilization: 0.9
compilation-config: '{"max_cudagraph_capture_size":4096}'

deepseek-v4-pro-0813-mtp:
<<: *deepseek-v4-pro-0813
name: DeepSeek-V4-Pro-0813-MTP
engine_args:
<<: *deepseek-v4-pro-0813-args
speculative-config: '{"method":"mtp"}'

deepseek-v4-pro-0813-dspark:
<<: *deepseek-v4-pro-0813
name: DeepSeek-V4-Pro-0813-DSpark
engine_args:
<<: *deepseek-v4-pro-0813-args
speculative-config: '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

deepseek-v4-pro-0813-dspark-dynamic:
<<: *deepseek-v4-pro-0813
name: DeepSeek-V4-Pro-0813-DSpark-Dynamic
engine_args:
<<: *deepseek-v4-pro-0813-args
speculative-config: '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","num_speculative_tokens_per_batch_size":[[1,64,7],[65,128,5],[129,512,0]]}'

deepseek-r1-0528:
name: DeepSeek-R1-0528
hf_model_id: deepseek-ai/DeepSeek-R1-0528
Expand Down
43 changes: 43 additions & 0 deletions projects/rhaiis/orchestration/presets.d/spec-decoding.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
__multiple: true

# DeepSeek-V4-Pro-0813 speculative decoding comparison on H200 Zeus (TP=8).
# Runs all four workload profiles so they land on the staging dashboard.
#
# Adaptive verification (enable_adaptive_verification) is omitted: DeepSeek-V4
# indexer/MLA backends only support the device-side trim + varlen full CUDA
# graphs on SM100 (B300). H200 (SM90) rejects it at startup.

deepseek-v4-spec-base:
rhaiis.accelerator: nvidia
rhaiis.cluster_tag: zeus
rhaiis.engines.vllm.args.gpu-memory-utilization: 0.9
tests.rhaiis.workload_keys:
- profile1
- profile2
- profile3
- profile4
rhaiis.profiler.enabled: true

@Harshith-umesh Harshith-umesh Aug 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need profiling for spec decoding?

@aas008 aas008 Aug 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a one time use preset @Harshith-umesh , should I add a tag to not merge it to main?

@kpouget kpouget Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can comment /hold in the main thread to prevent the merge and/or add Draft flag in the PR to convey that

@Harshith-umesh Harshith-umesh Aug 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes if it's a one time preset then you can add a do not merge tag to the PR

@aas008 aas008 Aug 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have permissions to add the label.


# 1. No speculative decoding
deepseek-v4-baseline:
extends:
- deepseek-v4-spec-base
tests.rhaiis.model_key: deepseek-v4-pro-0813

# 2. Native DeepSeek MTP
deepseek-v4-mtp:
extends:
- deepseek-v4-spec-base
tests.rhaiis.model_key: deepseek-v4-pro-0813-mtp

# 3. DSpark fixed-k (K=7)
deepseek-v4-dspark:
extends:
- deepseek-v4-spec-base
tests.rhaiis.model_key: deepseek-v4-pro-0813-dspark

# 4. DSpark + dynamic SD (K by concurrency bucket)
deepseek-v4-dspark-dynamic:
extends:
- deepseek-v4-spec-base
tests.rhaiis.model_key: deepseek-v4-pro-0813-dspark-dynamic
Loading