[FEAT] add grid-cardinality-and-auto-switch - #128
Conversation
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
📝 WalkthroughWalkthroughComputes the parameter-grid cardinality from a StudyConfig and integrates it into the CLI: the CLI compares requested trials against the grid size, forces grid sampler (or switches to random sampler) and adjusts trial/startup counts accordingly; validate output includes the grid cardinality. Changes
Sequence Diagram(s)sequenceDiagram
participant CLI
participant Loader as StudyConfig Loader
participant Grid as grid_cardinality.get_parameter_grid_cardinality
participant Optim as OptimizerRunner
CLI->>Loader: load StudyConfig (path or object)
CLI->>Grid: compute cardinality(config)
Grid-->>CLI: cardinality (int)
CLI->>CLI: compare requested_trials vs cardinality
alt requested >= cardinality
CLI->>CLI: set sampler = "grid", clamp n_trials, adjust n_startup_trials
else requested < cardinality and startup >= total_trials and sampler in (tpe,gp,botorch)
CLI->>CLI: set sampler = "random", clamp n_startup_trials
end
CLI->>Optim: run_optimization_sync(config, n_trials, sampler)
Optim-->>CLI: progress / results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
auto_tune_vllm/cli/main.py (1)
21-21: Consider importing from the package's public API.The function is re-exported in
auto_tune_vllm/utils/__init__.py. Importing from the package (from ..utils import get_parameter_grid_cardinality) would be more consistent with the established pattern.Suggested change
-from ..utils.grid_cardinality import get_parameter_grid_cardinality +from ..utils import get_parameter_grid_cardinality🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@auto_tune_vllm/cli/main.py` at line 21, The import in main.py currently pulls get_parameter_grid_cardinality from the internal module path; update the import to use the package's public API by importing get_parameter_grid_cardinality from ..utils (i.e., replace the current from ..utils.grid_cardinality import get_parameter_grid_cardinality with from ..utils import get_parameter_grid_cardinality) so it uses the re-export in utils/__init__.py and follows the project's import convention.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@auto_tune_vllm/cli/main.py`:
- Around line 373-387: Update the sampler selection logic in main.py around
total_trials/n_trials so it handles both auto-switch cases: (A) change the
current boundary check from ">" to ">=" when comparing total_trials to
cardinality and set config.optimization.sampler = "grid", adjust
config.optimization.n_trials, n_startup_trials, n_trials, and total_trials and
emit the console.print as done now (use symbols total_trials, n_trials,
cardinality, config.optimization.sampler, config.optimization.n_trials,
config.optimization.n_startup_trials, console.print,
get_parameter_grid_cardinality to find the block); (B) add the missing branch
that checks if n_trials <= config.optimization.n_startup_trials and in that case
set config.optimization.sampler = "random", set config.optimization.n_trials =
n_trials (or leave unchanged), ensure n_startup_trials is <= n_trials (e.g.,
reduce n_startup_trials to max(0, n_trials - 1)), update total_trials
accordingly, and emit a console.print explaining the auto-switch to random to
prevent TPE startup from consuming all trials. Ensure both branches are mutually
exclusive (check grid first then startup case).
In `@auto_tune_vllm/utils/grid_cardinality.py`:
- Around line 25-37: The float-range branch in grid_cardinality.py miscomputes
step counts due to floating-point truncation and mishandles step=None; update
the calculation in the RangeParameter float branch to compute n_steps robustly
(e.g., n_steps = int(round((max_val - min_val) / step)) + 1) and clamp to
_MAX_GRID_SIZE, and if step is None for a float RangeParameter treat the
cardinality as effectively large by returning _MAX_GRID_SIZE (instead of
defaulting step=1). Reference the RangeParameter float branch variables min_val,
max_val, step and the cap constant _MAX_GRID_SIZE when making the change.
- Around line 38-41: The function that computes grid cardinality (e.g.,
get_cardinality or similar) currently returns 0 for unhandled parameter types,
which will zero out total cardinality; change the default behavior to return 1
(neutral for multiplication) instead of 0 for unknown parameter types, and
optionally emit a warning or raise a ValueError if you prefer strict validation;
ensure the branch with BooleanParameter still returns 2 and update the default
return from 0 to 1 so unrecognized parameters do not silently zero the grid.
---
Nitpick comments:
In `@auto_tune_vllm/cli/main.py`:
- Line 21: The import in main.py currently pulls get_parameter_grid_cardinality
from the internal module path; update the import to use the package's public API
by importing get_parameter_grid_cardinality from ..utils (i.e., replace the
current from ..utils.grid_cardinality import get_parameter_grid_cardinality with
from ..utils import get_parameter_grid_cardinality) so it uses the re-export in
utils/__init__.py and follows the project's import convention.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 925ceea8-d974-4c8d-aa34-39d29bf442d7
📒 Files selected for processing (3)
auto_tune_vllm/cli/main.pyauto_tune_vllm/utils/__init__.pyauto_tune_vllm/utils/grid_cardinality.py
Signed-off-by: Vincent Gimenes <vincent.gimenes@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
auto_tune_vllm/cli/main.py (1)
389-405: Inconsistency: localn_trialsnot updated in Case 2.In Case 1 (line 382), the local
n_trialsvariable is reassigned tocardinality. In Case 2, onlyconfig.optimization.n_trialsis updated, but the localn_trialsvariable (which may beNone) is not reassigned. This value is later passed tocontroller.run_optimization(n_trials, ...)on line 430.If
run_optimizationtreatsNonedifferently from an explicit integer, the behavior may be inconsistent between the two cases. Consider updatingn_trialsfor symmetry:Proposed fix for consistency
config.optimization.sampler = "random" - config.optimization.n_trials = total_trials + n_trials = total_trials + config.optimization.n_trials = n_trials config.optimization.n_startup_trials = min(🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@auto_tune_vllm/cli/main.py` around lines 389 - 405, The local n_trials variable is not updated in the branch that auto-switches sampler to "random", causing inconsistent behavior when calling controller.run_optimization(n_trials, ...); update the local n_trials to match the new config by assigning n_trials = config.optimization.n_trials after you set config.optimization.n_trials so the value passed to controller.run_optimization is consistent with the config (refer to config.optimization.n_trials, config.optimization.sampler, and controller.run_optimization).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@auto_tune_vllm/cli/main.py`:
- Around line 373-405: The current logic treats a grid cardinality of 0 as
valid, causing total_trials >= cardinality to force grid mode with n_trials=0;
update the check around get_parameter_grid_cardinality(config) so that if
cardinality == 0 you raise or console.error a clear configuration error
(mentioning ListParameter/EnvironmentParameter empty options) and abort early
instead of switching to grid mode. Specifically, in the block using total_trials
and cardinality, add a guard: if cardinality == 0, log/raise an error describing
the empty options issue (or set a fallback behavior), and do not set
config.optimization.sampler = "grid" or overwrite n_trials; reference
get_parameter_grid_cardinality, total_trials, and config.optimization.n_trials
when implementing the guard.
---
Nitpick comments:
In `@auto_tune_vllm/cli/main.py`:
- Around line 389-405: The local n_trials variable is not updated in the branch
that auto-switches sampler to "random", causing inconsistent behavior when
calling controller.run_optimization(n_trials, ...); update the local n_trials to
match the new config by assigning n_trials = config.optimization.n_trials after
you set config.optimization.n_trials so the value passed to
controller.run_optimization is consistent with the config (refer to
config.optimization.n_trials, config.optimization.sampler, and
controller.run_optimization).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f7e32139-2fb7-4e71-9940-64de88363c63
📒 Files selected for processing (2)
auto_tune_vllm/cli/main.pyauto_tune_vllm/utils/grid_cardinality.py
🚧 Files skipped from review as they are similar to previous changes (1)
- auto_tune_vllm/utils/grid_cardinality.py
| total_trials = n_trials or config.optimization.n_trials | ||
| cardinality = get_parameter_grid_cardinality(config) | ||
| if total_trials >= cardinality: | ||
| requested = total_trials | ||
| config.optimization.sampler = "grid" | ||
| config.optimization.n_trials = cardinality | ||
| config.optimization.n_startup_trials = min( | ||
| config.optimization.n_startup_trials, max(0, cardinality - 1) | ||
| ) | ||
| n_trials = cardinality | ||
| total_trials = cardinality | ||
| console.print( | ||
| f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality " | ||
| f"({cardinality}). " | ||
| "Search set to grid mode with n_trials = cardinality.[/yellow]" | ||
| ) | ||
| elif ( | ||
| config.optimization.sampler.lower() in ("tpe", "gp", "botorch") | ||
| and total_trials <= config.optimization.n_startup_trials | ||
| ): | ||
| startup_before = config.optimization.n_startup_trials | ||
| prev_sampler = config.optimization.sampler | ||
| config.optimization.sampler = "random" | ||
| config.optimization.n_trials = total_trials | ||
| config.optimization.n_startup_trials = min( | ||
| startup_before, max(0, total_trials - 1) | ||
| ) | ||
| console.print( | ||
| f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': " | ||
| f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), " | ||
| "so startup sampling would consume the full trial budget. " | ||
| f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]" | ||
| ) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify what get_parameter_grid_cardinality returns for empty/disabled parameters
ast-grep --pattern $'def get_parameter_grid_cardinality($$$) {
$$$
}'Repository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Find where get_parameter_grid_cardinality is defined
rg -n "def get_parameter_grid_cardinality" --type pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 154
🏁 Script executed:
#!/bin/bash
# Also check imports in main.py to understand where this function comes from
head -50 auto_tune_vllm/cli/main.py | grep -E "^(from|import)" || true
rg "get_parameter_grid_cardinality" auto_tune_vllm/cli/main.py | head -5Repository: openshift-psap/auto-tuning-vllm
Length of output: 903
🏁 Script executed:
#!/bin/bash
# Search for any validation or checks around parameter grid cardinality
rg -B 5 -A 10 "get_parameter_grid_cardinality" auto_tune_vllm/cli/main.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 1894
🏁 Script executed:
#!/bin/bash
# Examine the get_parameter_grid_cardinality function
cat -n auto_tune_vllm/utils/grid_cardinality.py | head -100Repository: openshift-psap/auto-tuning-vllm
Length of output: 2461
🏁 Script executed:
#!/bin/bash
# Check the full file to understand the function
wc -l auto_tune_vllm/utils/grid_cardinality.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 118
🏁 Script executed:
#!/bin/bash
# Get the specific function and context
sed -n '40,80p' auto_tune_vllm/utils/grid_cardinality.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 807
🏁 Script executed:
#!/bin/bash
# Check what happens with parameters that have 0 options or values
# Look at the _count_parameter_values function and its return values
grep -A 5 "def _count_parameter_values" auto_tune_vllm/utils/grid_cardinality.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 391
🏁 Script executed:
#!/bin/bash
# Check if any parameter configuration can result in 0 values returned from _count_parameter_values
# Examine the function logic more carefully
sed -n '21,43p' auto_tune_vllm/utils/grid_cardinality.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 1004
🏁 Script executed:
#!/bin/bash
# Check if parameters can have empty options and what validation exists
# Look for ListParameter and EnvironmentParameter definitions
rg -A 10 "class ListParameter" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 393
🏁 Script executed:
#!/bin/bash
# Check for validation in the StudyConfig or parameter loading
rg -B 5 -A 5 "options" auto_tune_vllm/core/parameters.py | head -50Repository: openshift-psap/auto-tuning-vllm
Length of output: 828
🏁 Script executed:
#!/bin/bash
# Check if there's any validation when parameters are loaded
rg "from_file\|validate" auto_tune_vllm/core/config.py | head -20Repository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if there's validation preventing empty options in the parameter classes
rg -B 5 -A 15 "class ParameterConfig" auto_tune_vllm/core/parameters.py | head -60Repository: openshift-psap/auto-tuning-vllm
Length of output: 589
🏁 Script executed:
#!/bin/bash
# Look for field validators or constraints on options
rg "validator\|Field\|constraints" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if options can be empty by looking at test files or examples
fd "test" --type f --name "*parameter*" | head -5Repository: openshift-psap/auto-tuning-vllm
Length of output: 325
🏁 Script executed:
#!/bin/bash
# Look for test files that might show parameter configurations
find . -type f -name "test_*.py" -o -name "*_test.py" | grep -i param | head -5Repository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Search for example configs or test data
find . -type f -name "*.yaml" -o -name "*.yml" | head -10Repository: openshift-psap/auto-tuning-vllm
Length of output: 487
🏁 Script executed:
#!/bin/bash
# Check if there's a pydantic validator on the options field
rg -B 2 -A 2 "options.*=" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Look at an example config to understand realistic parameter configurations
cat examples/study_config_minimal.yamlRepository: openshift-psap/auto-tuning-vllm
Length of output: 1375
🏁 Script executed:
#!/bin/bash
# Check what happens if there are no enabled parameters
rg -B 3 -A 8 "for param_config in config.parameters" auto_tune_vllm/utils/grid_cardinality.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 368
🏁 Script executed:
#!/bin/bash
# Verify the actual scenario: what if a ListParameter has empty options?
# Look for any validation when loading configs
cat auto_tune_vllm/core/config.py | head -100Repository: openshift-psap/auto-tuning-vllm
Length of output: 3254
🏁 Script executed:
#!/bin/bash
# Check if there's any validation preventing empty options lists in parameter definitions
rg -C 10 "class ListParameter\|class EnvironmentParameter" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check the StudyConfig.from_file method for validation
rg -A 30 "def from_file" auto_tune_vllm/core/config.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 1170
🏁 Script executed:
#!/bin/bash
# Check for any validation in the ConfigValidator class
rg -A 50 "class ConfigValidator" auto_tune_vllm/core/config.py | head -80Repository: openshift-psap/auto-tuning-vllm
Length of output: 1860
🏁 Script executed:
#!/bin/bash
# Look for validation of parameters section in ConfigValidator
rg -B 5 -A 15 "def.*validate.*parameter\|validate.*options" auto_tune_vllm/core/config.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if pydantic models have field validators
grep -r "field_validator\|@validator" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Look at the full parameters.py file structure for validation
wc -l auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 111
🏁 Script executed:
#!/bin/bash
# Check if there's a __post_init__ or validation in ParameterConfig subclasses
rg -B 2 -A 5 "__post_init__\|model_validate\|validate" auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if parameters can actually have empty options by looking at actual schema usage
cat auto_tune_vllm/schemas/vllm_defaults/v0_10_0.yaml | head -100Repository: openshift-psap/auto-tuning-vllm
Length of output: 2088
🏁 Script executed:
#!/bin/bash
# Look at the full parameters.py to see if there's any validation
cat -n auto_tune_vllm/core/parameters.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 3003
🏁 Script executed:
#!/bin/bash
# Check test files to see if cardinality == 0 is tested or handled
find . -type f -name "test_*.py" -o -name "*_test.py" | xargs rg -l "cardinality\|grid" 2>/dev/null | head -5Repository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Look for any test that checks grid cardinality
rg "get_parameter_grid_cardinality" --type py | grep -i testRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if there's any validation when Optuna creates search space
rg -B 5 -A 15 "create_search_space\|_create_search_space" auto_tune_vllm/core/study_controller.py | head -50Repository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Check if empty options would cause errors earlier in the pipeline
rg "suggest_categorical" auto_tune_vllm/core/study_controller.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 57
🏁 Script executed:
#!/bin/bash
# Verify the actual context around line 375 in main.py to see if there's any prior validation
sed -n '360,380p' auto_tune_vllm/cli/main.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 803
🏁 Script executed:
#!/bin/bash
# Check if there's any validation in the ConfigValidator._validate_config method
rg -A 50 "def _validate_config" auto_tune_vllm/core/config.py | head -80Repository: openshift-psap/auto-tuning-vllm
Length of output: 2234
🏁 Script executed:
#!/bin/bash
# Verify if parameters are validated before being used
rg -B 5 -A 15 "parameters.*=" auto_tune_vllm/core/config.py | grep -A 10 "parameters:"Repository: openshift-psap/auto-tuning-vllm
Length of output: 1115
🏁 Script executed:
#!/bin/bash
# Check what happens if we have zero enabled parameters - does cardinality stay at 1?
# Verify the logic in grid_cardinality.py more carefully
cat -n auto_tune_vllm/utils/grid_cardinality.py | sed -n '46,61p'Repository: openshift-psap/auto-tuning-vllm
Length of output: 772
🏁 Script executed:
#!/bin/bash
# Check if Optuna would fail earlier with empty options
rg -B 5 -A 10 "suggest_categorical" auto_tune_vllm/ | head -30Repository: openshift-psap/auto-tuning-vllm
Length of output: 1848
🏁 Script executed:
#!/bin/bash
# Check _build_parameter_config to see if it validates non-empty options for ListParameter
rg -A 30 "_build_parameter_config" auto_tune_vllm/core/config.py | head -50Repository: openshift-psap/auto-tuning-vllm
Length of output: 2133
🏁 Script executed:
#!/bin/bash
# Get the rest of _build_parameter_config
rg -A 60 "def _build_parameter_config" auto_tune_vllm/core/config.py | tail -40Repository: openshift-psap/auto-tuning-vllm
Length of output: 1234
🏁 Script executed:
#!/bin/bash
# Also check _infer_parameter_type to understand how ListParameter is created
rg -B 5 -A 30 "_infer_parameter_type" auto_tune_vllm/core/config.pyRepository: openshift-psap/auto-tuning-vllm
Length of output: 2570
Guard against zero cardinality from empty parameter options.
If a ListParameter or EnvironmentParameter is configured with an empty options list, get_parameter_grid_cardinality() returns 0. The condition total_trials >= cardinality (line 375) then always evaluates to true, forcing grid mode with n_trials = 0, which runs no trials and produces no results. While rare, this edge case is possible because validation does not prevent empty options lists.
Add a guard to catch this configuration error early:
Proposed fix
total_trials = n_trials or config.optimization.n_trials
cardinality = get_parameter_grid_cardinality(config)
+if cardinality == 0:
+ console.print(
+ "[bold red]Error: Grid cardinality is 0 (empty parameter options or no enabled parameters).[/bold red]"
+ )
+ raise typer.Exit(1)
if total_trials >= cardinality:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| total_trials = n_trials or config.optimization.n_trials | |
| cardinality = get_parameter_grid_cardinality(config) | |
| if total_trials >= cardinality: | |
| requested = total_trials | |
| config.optimization.sampler = "grid" | |
| config.optimization.n_trials = cardinality | |
| config.optimization.n_startup_trials = min( | |
| config.optimization.n_startup_trials, max(0, cardinality - 1) | |
| ) | |
| n_trials = cardinality | |
| total_trials = cardinality | |
| console.print( | |
| f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality " | |
| f"({cardinality}). " | |
| "Search set to grid mode with n_trials = cardinality.[/yellow]" | |
| ) | |
| elif ( | |
| config.optimization.sampler.lower() in ("tpe", "gp", "botorch") | |
| and total_trials <= config.optimization.n_startup_trials | |
| ): | |
| startup_before = config.optimization.n_startup_trials | |
| prev_sampler = config.optimization.sampler | |
| config.optimization.sampler = "random" | |
| config.optimization.n_trials = total_trials | |
| config.optimization.n_startup_trials = min( | |
| startup_before, max(0, total_trials - 1) | |
| ) | |
| console.print( | |
| f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': " | |
| f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), " | |
| "so startup sampling would consume the full trial budget. " | |
| f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]" | |
| ) | |
| total_trials = n_trials or config.optimization.n_trials | |
| cardinality = get_parameter_grid_cardinality(config) | |
| if cardinality == 0: | |
| console.print( | |
| "[bold red]Error: Grid cardinality is 0 (empty parameter options or no enabled parameters).[/bold red]" | |
| ) | |
| raise typer.Exit(1) | |
| if total_trials >= cardinality: | |
| requested = total_trials | |
| config.optimization.sampler = "grid" | |
| config.optimization.n_trials = cardinality | |
| config.optimization.n_startup_trials = min( | |
| config.optimization.n_startup_trials, max(0, cardinality - 1) | |
| ) | |
| n_trials = cardinality | |
| total_trials = cardinality | |
| console.print( | |
| f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality " | |
| f"({cardinality}). " | |
| "Search set to grid mode with n_trials = cardinality.[/yellow]" | |
| ) | |
| elif ( | |
| config.optimization.sampler.lower() in ("tpe", "gp", "botorch") | |
| and total_trials <= config.optimization.n_startup_trials | |
| ): | |
| startup_before = config.optimization.n_startup_trials | |
| prev_sampler = config.optimization.sampler | |
| config.optimization.sampler = "random" | |
| config.optimization.n_trials = total_trials | |
| config.optimization.n_startup_trials = min( | |
| startup_before, max(0, total_trials - 1) | |
| ) | |
| console.print( | |
| f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': " | |
| f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), " | |
| "so startup sampling would consume the full trial budget. " | |
| f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@auto_tune_vllm/cli/main.py` around lines 373 - 405, The current logic treats
a grid cardinality of 0 as valid, causing total_trials >= cardinality to force
grid mode with n_trials=0; update the check around
get_parameter_grid_cardinality(config) so that if cardinality == 0 you raise or
console.error a clear configuration error (mentioning
ListParameter/EnvironmentParameter empty options) and abort early instead of
switching to grid mode. Specifically, in the block using total_trials and
cardinality, add a guard: if cardinality == 0, log/raise an error describing the
empty options issue (or set a fallback behavior), and do not set
config.optimization.sampler = "grid" or overwrite n_trials; reference
get_parameter_grid_cardinality, total_trials, and config.optimization.n_trials
when implementing the guard.
2 participants
Summary
This PR adds grid cardinality calculation and implements smart sampler selection logic to automatically switch between optimization strategies based on the study configuration.
Changes
New: Grid Cardinality Calculator (
auto_tune_vllm/utils/grid_cardinality.py)ListParameter,RangeParameter,BooleanParameterSmart Sampler Auto-Switching (
auto_tune_vllm/cli/main.py)Case 1:
n_trials >= grid_cardinalitysampler="grid"and setn_trials = cardinalityCase 2:
n_trials <= n_startup_trialssampler="random"and ignoren_startup_trialsCLI Enhancement
validatecommand outputExample Behavior
study_config.yaml with small search space
parameters:
tensor_parallel_size:
type: list
options: [1, 2] # 2 values
dtype:
type: list
options: [float16, bfloat16] # 2 values
→ Grid cardinality = 2 × 2 = 4
If user requests --trials 10 (> 4) → Warning: n_trials (10) exceeds grid cardinality (4). Search set to grid mode with n_trials = 4.
If user requests --trials 5 with n_startup_trials=10 → Warning: n_trials (5) <= n_startup_trials (10). Search set to random mode; n_startup_trials ignored.
Summary by CodeRabbit