Skip to content

feat(spaceship-titanic): add leak-safe bundled evaluator#52

Open
Abhishek21g wants to merge 2 commits into
hexo-ai:mainfrom
Abhishek21g:pr/32-spaceship-evaluator
Open

feat(spaceship-titanic): add leak-safe bundled evaluator#52
Abhishek21g wants to merge 2 commits into
hexo-ai:mainfrom
Abhishek21g:pr/32-spaceship-evaluator

Conversation

@Abhishek21g

Copy link
Copy Markdown

Summary

Adds the missing evaluate.py for spaceship-titanic — the only bundled task without a first-party evaluator despite README stating all four ship one.

Builds on the approach in #32 with an additional integrity improvement: results.json uses summary scalars + items[] and never writes held-out Transported labels, aligned with the eval-summary contract in #36.

Why

Changes

  • sia/tasks/spaceship-titanic/data/public/evaluate.py — Kaggle-style CSV scoring
  • tests/test_spaceship_titanic_evaluator.py — contract + leak guard tests

Test plan

  • python -m pytest tests/test_spaceship_titanic_evaluator.py -q — 3 passed
  • python -m pytest tests/ -q — 132 passed
  • Assert expected absent from written results.json

Related

skyzer and others added 2 commits July 5, 2026 18:05
Ship evaluate.py for the fourth bundled task (README claims all four have
evaluators; this task was missing one). results.json uses summary scalars plus
items[] without held-out labels, aligned with the eval-summary contract in hexo-ai#36.

@Abhishek21g Abhishek21g left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally on Abhishek21g/sia pr/32-spaceship-evaluator: 132 tests passed. Closes the only bundled task missing an evaluator; leak-safe results.json pairs with #36.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants