Skip to content

Conversation

MaxGhenis
Copy link
Contributor

Description

This PR adds parameter validation to the LanguageModel class to warn users about invalid or misspelled parameters that will be silently ignored.

Fixes #2184

Problem

Currently, EDSL accepts any parameter without validation, leading to silent failures when users:

  • Make typos (e.g., temprature instead of temperature)
  • Use unknown parameters (e.g., unicorn_mode=True)
  • Use parameters not supported by their chosen service

Solution

Added a parameter validation system that:

  1. Warns about unknown parameters
  2. Suggests corrections for common typos
  3. Validates parameters against service-specific lists
  4. Uses warnings instead of errors for backward compatibility

Changes

  • Added parameter_validator.py with validation logic and known parameters for each service
  • Updated language_model.py to validate kwargs before applying them
  • Added comprehensive tests in test_parameter_validation.py

Example

# This now generates a warning:
m = Model('gpt-4o-mini', temprature=0.7)
# UserWarning: Parameter validation issues:
#   'temprature' might be a typo. Did you mean 'temperature'?

# This also warns:
m = Model('gpt-4o-mini', unicorn_mode=True)
# UserWarning: Parameter validation issues:
#   Unknown parameters for openai: unicorn_mode
#   These parameters will be ignored and have no effect.

Testing

  • Added comprehensive test suite covering:
    • Valid parameters (no warnings)
    • Unknown parameters (warnings)
    • Typo detection
    • Service-specific validation
    • Backward compatibility (parameters still set as attributes)

Backward Compatibility

  • Uses warnings instead of errors
  • Unknown parameters are still set as attributes (existing behavior preserved)
  • Can be made stricter in future major version if desired

Fixes expectedparrot#2184

This PR adds parameter validation to the LanguageModel class to warn users when they:
- Use unknown parameters that will be ignored
- Make common typos in parameter names
- Use parameters not supported by their chosen service

Changes:
- Add parameter_validator.py with validation logic and known parameters for each service
- Update language_model.py to validate kwargs before applying them
- Add comprehensive tests for parameter validation
- Use warnings for backward compatibility (not errors)

The validation will help catch issues like:
- Typos: 'temprature' instead of 'temperature'
- Unknown params: 'unicorn_mode' or other nonsense
- Service mismatches: using 'top_k' with OpenAI (only valid for Anthropic/Google)

This addresses the issue where EDSL silently accepts any parameter without validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Model accepts arbitrary parameters without validation or warnings
1 participant