Skip to content

Feature: Add eval push to env hub.#167

Merged
d42me merged 2 commits intomainfrom
feature/add-eval-push-env-hub-eng-2132
Oct 24, 2025
Merged

Feature: Add eval push to env hub.#167
d42me merged 2 commits intomainfrom
feature/add-eval-push-env-hub-eng-2132

Conversation

@d42me
Copy link
Copy Markdown
Contributor

@d42me d42me commented Oct 23, 2025

Closes ENG-2132

Adds support for --push-to-hub flag for Environment Hub

Example command:

prime env eval gsm8k -m meta-llama/llama-3.1-70b-instruct -n 2 -r 1 --push-to-hub


Note

Adds a --push-to-hub flag to prime env eval to upload vf-eval results to Prime Evals Hub, including environment ID resolution and async support in evals client.

  • CLI:
    • packages/prime/src/prime_cli/commands/env.py
      • Add --push-to-hub (-P) to prime env eval; on success, pushes results via helper.
      • Wire in push_eval_results_to_hub(environment, model, job_id) and handle errors.
  • Utility:
    • packages/prime/src/prime_cli/utils/eval_push.py (new)
      • Parse latest vf-eval outputs (metadata.json, results.jsonl), resolve environment, convert samples, create evaluation, push samples, and finalize via EvalsClient.
      • Prints hub URL when available.
  • SDK (prime-evals):
    • packages/prime-evals/src/prime_evals/evals.py
      • Add environment resolution via /environmentshub/resolve; raise EvalsAPIError on missing env.
      • Resolve environment IDs in create_evaluation (sync/async; async uses asyncio.gather).
      • Import APIError; expose new EvalsAPIError.

Written by Cursor Bugbot for commit 9985fd6. This will update automatically on new commits. Configure here.

@d42me d42me marked this pull request as ready for review October 23, 2025 16:44
@d42me d42me requested review from AmeenP, JannikSt and burnpiro October 23, 2025 16:46
@cursor

This comment was marked as off-topic.

AmeenP
AmeenP previously approved these changes Oct 24, 2025
Comment thread packages/prime-evals/src/prime_evals/evals.py Outdated
Comment thread packages/prime-evals/src/prime_evals/evals.py
Comment thread packages/prime/src/prime_cli/utils/eval_push.py
@d42me d42me merged commit ff527ad into main Oct 24, 2025
11 checks passed
@d42me d42me deleted the feature/add-eval-push-env-hub-eng-2132 branch October 24, 2025 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants