Skip to content

[FEAT] add grid-cardinality-and-auto-switch #128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
VincentG1234 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[FEAT] add grid-cardinality-and-auto-switch #128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cd44b88
add grid-cardinality-and-auto-switch
VincentG1234 Mar 13, 2026
3e8d3e5
fix ruff
VincentG1234 Mar 21, 2026
213675f
apply rabbit ai feedbacks
VincentG1234 Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 40 additions & 3 deletions auto_tune_vllm/cli/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from ..core.study_controller import StudyController
from ..execution.backends import RayExecutionBackend
from ..logging.manager import CentralizedLogger, LogStreamer
from ..utils.grid_cardinality import get_parameter_grid_cardinality

# Setup rich console and app
console = Console()
Expand Down Expand Up @@ -369,7 +370,40 @@
create_db: bool = False,
):
"""Synchronous optimization runner with progress display."""
# Create study controller
total_trials = n_trials or config.optimization.n_trials
cardinality = get_parameter_grid_cardinality(config)
if total_trials >= cardinality:
requested = total_trials
config.optimization.sampler = "grid"
config.optimization.n_trials = cardinality
config.optimization.n_startup_trials = min(
config.optimization.n_startup_trials, max(0, cardinality - 1)
)
n_trials = cardinality
total_trials = cardinality
console.print(
f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality "
f"({cardinality}). "
"Search set to grid mode with n_trials = cardinality.[/yellow]"
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
elif (
config.optimization.sampler.lower() in ("tpe", "gp", "botorch")
and total_trials <= config.optimization.n_startup_trials
):
startup_before = config.optimization.n_startup_trials
prev_sampler = config.optimization.sampler
config.optimization.sampler = "random"
config.optimization.n_trials = total_trials
config.optimization.n_startup_trials = min(
startup_before, max(0, total_trials - 1)
)
console.print(
f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': "
f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), "
"so startup sampling would consume the full trial budget. "
f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]"
)
Comment on lines +373 to +405

@coderabbitai coderabbitai Bot Mar 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify what get_parameter_grid_cardinality returns for empty/disabled parameters
ast-grep --pattern $'def get_parameter_grid_cardinality($$$) {
  $$$
}'

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Find where get_parameter_grid_cardinality is defined
rg -n "def get_parameter_grid_cardinality" --type py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 154

🏁 Script executed:

#!/bin/bash
# Also check imports in main.py to understand where this function comes from
head -50 auto_tune_vllm/cli/main.py | grep -E "^(from|import)" || true
rg "get_parameter_grid_cardinality" auto_tune_vllm/cli/main.py | head -5

Repository: openshift-psap/auto-tuning-vllm

Length of output: 903

🏁 Script executed:

#!/bin/bash
# Search for any validation or checks around parameter grid cardinality
rg -B 5 -A 10 "get_parameter_grid_cardinality" auto_tune_vllm/cli/main.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1894

🏁 Script executed:

#!/bin/bash
# Examine the get_parameter_grid_cardinality function
cat -n auto_tune_vllm/utils/grid_cardinality.py | head -100

Repository: openshift-psap/auto-tuning-vllm

Length of output: 2461

🏁 Script executed:

#!/bin/bash
# Check the full file to understand the function
wc -l auto_tune_vllm/utils/grid_cardinality.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 118

🏁 Script executed:

#!/bin/bash
# Get the specific function and context
sed -n '40,80p' auto_tune_vllm/utils/grid_cardinality.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 807

🏁 Script executed:

#!/bin/bash
# Check what happens with parameters that have 0 options or values
# Look at the _count_parameter_values function and its return values
grep -A 5 "def _count_parameter_values" auto_tune_vllm/utils/grid_cardinality.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 391

🏁 Script executed:

#!/bin/bash
# Check if any parameter configuration can result in 0 values returned from _count_parameter_values
# Examine the function logic more carefully
sed -n '21,43p' auto_tune_vllm/utils/grid_cardinality.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1004

🏁 Script executed:

#!/bin/bash
# Check if parameters can have empty options and what validation exists
# Look for ListParameter and EnvironmentParameter definitions
rg -A 10 "class ListParameter" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 393

🏁 Script executed:

#!/bin/bash
# Check for validation in the StudyConfig or parameter loading
rg -B 5 -A 5 "options" auto_tune_vllm/core/parameters.py | head -50

Repository: openshift-psap/auto-tuning-vllm

Length of output: 828

🏁 Script executed:

#!/bin/bash
# Check if there's any validation when parameters are loaded
rg "from_file\|validate" auto_tune_vllm/core/config.py | head -20

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if there's validation preventing empty options in the parameter classes
rg -B 5 -A 15 "class ParameterConfig" auto_tune_vllm/core/parameters.py | head -60

Repository: openshift-psap/auto-tuning-vllm

Length of output: 589

🏁 Script executed:

#!/bin/bash
# Look for field validators or constraints on options
rg "validator\|Field\|constraints" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if options can be empty by looking at test files or examples
fd "test" --type f --name "*parameter*" | head -5

Repository: openshift-psap/auto-tuning-vllm

Length of output: 325

🏁 Script executed:

#!/bin/bash
# Look for test files that might show parameter configurations
find . -type f -name "test_*.py" -o -name "*_test.py" | grep -i param | head -5

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Search for example configs or test data
find . -type f -name "*.yaml" -o -name "*.yml" | head -10

Repository: openshift-psap/auto-tuning-vllm

Length of output: 487

🏁 Script executed:

#!/bin/bash
# Check if there's a pydantic validator on the options field
rg -B 2 -A 2 "options.*=" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Look at an example config to understand realistic parameter configurations
cat examples/study_config_minimal.yaml

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1375

🏁 Script executed:

#!/bin/bash
# Check what happens if there are no enabled parameters
rg -B 3 -A 8 "for param_config in config.parameters" auto_tune_vllm/utils/grid_cardinality.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 368

🏁 Script executed:

#!/bin/bash
# Verify the actual scenario: what if a ListParameter has empty options?
# Look for any validation when loading configs
cat auto_tune_vllm/core/config.py | head -100

Repository: openshift-psap/auto-tuning-vllm

Length of output: 3254

🏁 Script executed:

#!/bin/bash
# Check if there's any validation preventing empty options lists in parameter definitions
rg -C 10 "class ListParameter\|class EnvironmentParameter" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check the StudyConfig.from_file method for validation
rg -A 30 "def from_file" auto_tune_vllm/core/config.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1170

🏁 Script executed:

#!/bin/bash
# Check for any validation in the ConfigValidator class
rg -A 50 "class ConfigValidator" auto_tune_vllm/core/config.py | head -80

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1860

🏁 Script executed:

#!/bin/bash
# Look for validation of parameters section in ConfigValidator
rg -B 5 -A 15 "def.*validate.*parameter\|validate.*options" auto_tune_vllm/core/config.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if pydantic models have field validators
grep -r "field_validator\|@validator" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Look at the full parameters.py file structure for validation
wc -l auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 111

🏁 Script executed:

#!/bin/bash
# Check if there's a __post_init__ or validation in ParameterConfig subclasses
rg -B 2 -A 5 "__post_init__\|model_validate\|validate" auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if parameters can actually have empty options by looking at actual schema usage
cat auto_tune_vllm/schemas/vllm_defaults/v0_10_0.yaml | head -100

Repository: openshift-psap/auto-tuning-vllm

Length of output: 2088

🏁 Script executed:

#!/bin/bash
# Look at the full parameters.py to see if there's any validation
cat -n auto_tune_vllm/core/parameters.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 3003

🏁 Script executed:

#!/bin/bash
# Check test files to see if cardinality == 0 is tested or handled
find . -type f -name "test_*.py" -o -name "*_test.py" | xargs rg -l "cardinality\|grid" 2>/dev/null | head -5

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Look for any test that checks grid cardinality
rg "get_parameter_grid_cardinality" --type py | grep -i test

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if there's any validation when Optuna creates search space
rg -B 5 -A 15 "create_search_space\|_create_search_space" auto_tune_vllm/core/study_controller.py | head -50

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Check if empty options would cause errors earlier in the pipeline
rg "suggest_categorical" auto_tune_vllm/core/study_controller.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 57

🏁 Script executed:

#!/bin/bash
# Verify the actual context around line 375 in main.py to see if there's any prior validation
sed -n '360,380p' auto_tune_vllm/cli/main.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 803

🏁 Script executed:

#!/bin/bash
# Check if there's any validation in the ConfigValidator._validate_config method
rg -A 50 "def _validate_config" auto_tune_vllm/core/config.py | head -80

Repository: openshift-psap/auto-tuning-vllm

Length of output: 2234

🏁 Script executed:

#!/bin/bash
# Verify if parameters are validated before being used
rg -B 5 -A 15 "parameters.*=" auto_tune_vllm/core/config.py | grep -A 10 "parameters:"

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1115

🏁 Script executed:

#!/bin/bash
# Check what happens if we have zero enabled parameters - does cardinality stay at 1?
# Verify the logic in grid_cardinality.py more carefully
cat -n auto_tune_vllm/utils/grid_cardinality.py | sed -n '46,61p'

Repository: openshift-psap/auto-tuning-vllm

Length of output: 772

🏁 Script executed:

#!/bin/bash
# Check if Optuna would fail earlier with empty options
rg -B 5 -A 10 "suggest_categorical" auto_tune_vllm/ | head -30

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1848

🏁 Script executed:

#!/bin/bash
# Check _build_parameter_config to see if it validates non-empty options for ListParameter
rg -A 30 "_build_parameter_config" auto_tune_vllm/core/config.py | head -50

Repository: openshift-psap/auto-tuning-vllm

Length of output: 2133

🏁 Script executed:

#!/bin/bash
# Get the rest of _build_parameter_config
rg -A 60 "def _build_parameter_config" auto_tune_vllm/core/config.py | tail -40

Repository: openshift-psap/auto-tuning-vllm

Length of output: 1234

🏁 Script executed:

#!/bin/bash
# Also check _infer_parameter_type to understand how ListParameter is created
rg -B 5 -A 30 "_infer_parameter_type" auto_tune_vllm/core/config.py

Repository: openshift-psap/auto-tuning-vllm

Length of output: 2570

Guard against zero cardinality from empty parameter options.

If a ListParameter or EnvironmentParameter is configured with an empty options list, get_parameter_grid_cardinality() returns 0. The condition total_trials >= cardinality (line 375) then always evaluates to true, forcing grid mode with n_trials = 0, which runs no trials and produces no results. While rare, this edge case is possible because validation does not prevent empty options lists.

Add a guard to catch this configuration error early:

Proposed fix 
 total_trials = n_trials or config.optimization.n_trials
 cardinality = get_parameter_grid_cardinality(config)
+if cardinality == 0:
+    console.print(
+        "[bold red]Error: Grid cardinality is 0 (empty parameter options or no enabled parameters).[/bold red]"
+    )
+    raise typer.Exit(1)
 if total_trials >= cardinality:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
total_trials = n_trials or config.optimization.n_trials
cardinality = get_parameter_grid_cardinality(config)
if total_trials >= cardinality:
requested = total_trials
config.optimization.sampler = "grid"
config.optimization.n_trials = cardinality
config.optimization.n_startup_trials = min(
config.optimization.n_startup_trials, max(0, cardinality - 1)
)
n_trials = cardinality
total_trials = cardinality
console.print(
f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality "
f"({cardinality}). "
"Search set to grid mode with n_trials = cardinality.[/yellow]"
)
elif (
config.optimization.sampler.lower() in ("tpe", "gp", "botorch")
and total_trials <= config.optimization.n_startup_trials
):
startup_before = config.optimization.n_startup_trials
prev_sampler = config.optimization.sampler
config.optimization.sampler = "random"
config.optimization.n_trials = total_trials
config.optimization.n_startup_trials = min(
startup_before, max(0, total_trials - 1)
)
console.print(
f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': "
f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), "
"so startup sampling would consume the full trial budget. "
f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]"
)
total_trials = n_trials or config.optimization.n_trials
cardinality = get_parameter_grid_cardinality(config)
if cardinality == 0:
console.print(
"[bold red]Error: Grid cardinality is 0 (empty parameter options or no enabled parameters).[/bold red]"
)
raise typer.Exit(1)
if total_trials >= cardinality:
requested = total_trials
config.optimization.sampler = "grid"
config.optimization.n_trials = cardinality
config.optimization.n_startup_trials = min(
config.optimization.n_startup_trials, max(0, cardinality - 1)
)
n_trials = cardinality
total_trials = cardinality
console.print(
f"[yellow]n_trials ({requested}) meets or exceeds grid cardinality "
f"({cardinality}). "
"Search set to grid mode with n_trials = cardinality.[/yellow]"
)
elif (
config.optimization.sampler.lower() in ("tpe", "gp", "botorch")
and total_trials <= config.optimization.n_startup_trials
):
startup_before = config.optimization.n_startup_trials
prev_sampler = config.optimization.sampler
config.optimization.sampler = "random"
config.optimization.n_trials = total_trials
config.optimization.n_startup_trials = min(
startup_before, max(0, total_trials - 1)
)
console.print(
f"[yellow]Auto-switched sampler from '{prev_sampler}' to 'random': "
f"n_trials ({total_trials}) is <= n_startup_trials ({startup_before}), "
"so startup sampling would consume the full trial budget. "
f"n_startup_trials is now {config.optimization.n_startup_trials}.[/yellow]"
)
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@auto_tune_vllm/cli/main.py` around lines 373 - 405, The current logic treats
a grid cardinality of 0 as valid, causing total_trials >= cardinality to force
grid mode with n_trials=0; update the check around
get_parameter_grid_cardinality(config) so that if cardinality == 0 you raise or
console.error a clear configuration error (mentioning
ListParameter/EnvironmentParameter empty options) and abort early instead of
switching to grid mode. Specifically, in the block using total_trials and
cardinality, add a guard: if cardinality == 0, log/raise an error describing the
empty options issue (or set a fallback behavior), and do not set
config.optimization.sampler = "grid" or overwrite n_trials; reference
get_parameter_grid_cardinality, total_trials, and config.optimization.n_trials
when implementing the guard.

# Create study controller (uses config with possibly updated sampler/n_trials)
controller = StudyController.create_from_config(
backend, config, create_db=create_db
)
Expand All @@ -381,8 +415,6 @@
_display_log_viewing_instructions(config)
console.print() # Add blank line for better readability

total_trials = n_trials or config.optimization.n_trials

with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
Expand Down Expand Up @@ -443,7 +475,7 @@

if results.get("baseline_improvement") is not None:
improvement = results["baseline_improvement"]
if improvement > 0:

Check failure on line 478 in auto_tune_vllm/cli/main.py

View workflow job for this annotation

GitHub Actions / pyright 

Operator ">" not supported for types "int | float | str | list[float] | list[dict[str, int | float | str | list[float] | None]] | dict[str, str | int | float] | None" and "Literal[0]"
  Operator ">" not supported for types "str" and "Literal[0]"
  Operator ">" not supported for types "list[float]" and "Literal[0]"
  Operator ">" not supported for types "list[dict[str, int | float | str | list[float] | None]]" and "Literal[0]"
  Operator ">" not supported for types "dict[str, str | int | float]" and "Literal[0]"
  Operator ">" not supported for types "None" and "Literal[0]" (reportOperatorIssue)
improvement_text = f"+{improvement:.1f}%"
improvement_style = "green"
else:
Expand Down Expand Up @@ -1086,6 +1118,11 @@
"Optimization", f"{opt_summary} ({study_config.optimization.sampler})"
)
table.add_row("Trials", str(study_config.optimization.n_trials))
cardinality = get_parameter_grid_cardinality(study_config)
table.add_row(
"Possible combinations (grid cardinality)",
str(cardinality),
)
table.add_row("Model", study_config.benchmark.model)
table.add_row(
"Parameters",
Expand Down
2 changes: 2 additions & 0 deletions auto_tune_vllm/utils/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
Utilities for auto-tune-vllm package.
"""

from .grid_cardinality import get_parameter_grid_cardinality
from .version_manager import VLLMDefaultsVersion, VLLMVersionManager
from .vllm_cli_parser import ArgumentType, CLIArgument, VLLMCLIParser

Expand All @@ -11,4 +12,5 @@
"ArgumentType",
"VLLMVersionManager",
"VLLMDefaultsVersion",
"get_parameter_grid_cardinality",
]
61 changes: 61 additions & 0 deletions auto_tune_vllm/utils/grid_cardinality.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""
Compute the total cardinality of the parameter grid from a study config.
Uses the same logic as StudyController._create_search_space for consistency.
"""

from pathlib import Path
from typing import Union

from auto_tune_vllm.core.config import StudyConfig
from auto_tune_vllm.core.parameters import (
BooleanParameter,
EnvironmentParameter,
ListParameter,
ParameterConfig,
RangeParameter,
)

_MAX_GRID_SIZE = 10000


def _count_parameter_values(param: ParameterConfig) -> int:
"""Count distinct values for one parameter (mirrors _create_search_space)."""
if isinstance(param, (ListParameter, EnvironmentParameter)):
return len(param.options)
if isinstance(param, RangeParameter):
min_val = param.min_value
max_val = param.max_value
if param.data_type is float:
step = param.step
if step is None:
return _MAX_GRID_SIZE
n_steps = int(round((max_val - min_val) / step)) + 1
return min(n_steps, _MAX_GRID_SIZE)
step = param.step or 1
current = min_val
count = 0
while current <= max_val and count < _MAX_GRID_SIZE:
count += 1
current += step
return count
Comment thread
coderabbitai[bot] marked this conversation as resolved.
if isinstance(param, BooleanParameter):
return 2
raise ValueError(f"Unknown parameter type: {type(param)}")

Comment thread
coderabbitai[bot] marked this conversation as resolved.

def get_parameter_grid_cardinality(
config: Union[str, Path, StudyConfig],
vllm_version: str | None = None,
) -> int:
"""
Return the total number of points in the parameter grid (product of enabled params).

config: Path to YAML, or an already-loaded StudyConfig.
"""
if isinstance(config, (str, Path)):
config = StudyConfig.from_file(str(config), vllm_version=vllm_version)
total = 1
for param_config in config.parameters.values():
if param_config.enabled:
total *= _count_parameter_values(param_config)
return total
Loading