Skip to content

Document plugin registration and expand plugin tests#3

Open
Erotemic wants to merge 4 commits intoexplicit_plugins-refinefrom
codex/update-helm-documentation-for-plugins
Open

Document plugin registration and expand plugin tests#3
Erotemic wants to merge 4 commits intoexplicit_plugins-refinefrom
codex/update-helm-documentation-for-plugins

Conversation

@Erotemic
Copy link
Collaborator

@Erotemic Erotemic commented Jan 9, 2026

Motivation

  • The plugin docs did not explain run specs or how HELM discovers and registers different plugin types.
  • Existing tests covered import mechanics for modules and entry-points but did not verify that Scenario, Metric, Client, Tokenizer (and other ObjectSpec-backed components) can be registered and instantiated.
  • Provide users concrete examples and clear guidance for how to write and expose plugins so third-party code can be discovered by helm-run.

Description

  • Update docs/importing_custom_modules.md to enumerate plugin categories (run specs vs ObjectSpec-backed classes), document discovery/registration paths, add a compact example showing @run_spec_function(...) usage and ObjectSpec-referenced classes, and clarify PYTHONPATH guidance.
  • Add test_import_user_plugins_supports_object_spec_plugins to src/helm/benchmark/test_plugins.py which creates a temporary module that defines a run spec and CustomScenario, CustomMetric, CustomClient, and CustomTokenizer, then exercises import_user_plugins, create_scenario, create_metric, create_object(ClientSpec), create_object(TokenizerSpec) and get_run_spec_function.
  • Ensure the code paths used by HELM to resolve classes by name (via ObjectSpec + create_object / create_metric / create_scenario) are exercised by the new test.
  • Minor doc cleanup (removed an unused import reference) and committed changes.

Testing

  • No automated test suite was executed as part of this change.
  • A new unit test test_import_user_plugins_supports_object_spec_plugins was added to src/helm/benchmark/test_plugins.py but has not been run here.
  • Existing plugin tests in src/helm/benchmark/test_plugins.py were left intact and the new test complements their coverage by targeting ObjectSpec-backed components.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant