Skip to content

fix: resolve dataset path config and undeclared provider dependencies - #20

Merged
himanshu231204 merged 1 commit into
mainfrom
fix/dataset-path-config
Jul 10, 2026
Merged

fix: resolve dataset path config and undeclared provider dependencies#20
himanshu231204 merged 1 commit into
mainfrom
fix/dataset-path-config

Conversation

@himanshu231204

Copy link
Copy Markdown
Member

Summary

  • Dataset config bug: \config.yaml\ pointed \dataset.path\ at a non-existent \data/questions.json\ (and a bare directory \data/\ produced a misleading \InvalidDatasetError). Now points to the existing \data/sample_questions.json\ with an explicit \ ormat: json.
  • Clearer loader errors: \datasets/factory.py\ now detects directory paths and raises an actionable error instead of the misleading \set the format field\ message.
  • JSON loader: supports the common {\items: [...]}\ wrapper so \sample_questions.json\ loads (5 items).
  • Undeclared dependencies: LLM provider SDKs (\openai, \�nthropic, \groq, \google-genai, \ iktoken) were imported unconditionally but never declared. Added a \providers\ optional extra in \pyproject.toml\ (and to \�ll).
  • Lazy provider imports: the four LLM provider modules now import their SDKs lazily (guarded \ ry/except), so the modules import without the optional dependency and raise a clear \ProviderError\ with install instructions at instantiation — matching the project's existing lazy-import design.
  • Test fix: added the missing \import httpx\ in \ est_anthropic.py\ (was using \httpx.Response\ unimported → 3 NameError failures).

Test plan

  • Full suite: 526 passed, 3 skipped (no regressions).
  • Verified provider modules import cleanly without their SDK and raise \ProviderError: The '' package is required ... Install with: pip install openagent-eval[providers]\ when instantiated without it.

Notes

Only files related to this fix were committed; pre-existing unrelated modifications (e.g. README.md) and untracked artifacts (reports/, docs) were left out.

- Point dataset config at an existing file (data/sample_questions.json) with explicit format; previously referenced a non-existent path/directory.
- Harden dataset loader factory to give a clear error for directory paths instead of a misleading 'set format' message.
- Support the common {items: [...]} wrapper in the JSON dataset loader.
- Declare LLM provider SDKs (openai, anthropic, groq, google-genai, tiktoken) as a new 'providers' optional extra in pyproject.toml.
- Make provider SDK imports lazy so modules import without the optional dependency and raise a clear ProviderError at instantiation.
- Fix missing 'import httpx' in test_anthropic.py.
Copilot AI review requested due to automatic review settings July 10, 2026 09:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@himanshu231204
himanshu231204 merged commit 47053ba into main Jul 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants