feat(spaceship-titanic): add leak-safe bundled evaluator#52
Open
Abhishek21g wants to merge 2 commits into
Open
feat(spaceship-titanic): add leak-safe bundled evaluator#52Abhishek21g wants to merge 2 commits into
Abhishek21g wants to merge 2 commits into
Conversation
Ship evaluate.py for the fourth bundled task (README claims all four have evaluators; this task was missing one). results.json uses summary scalars plus items[] without held-out labels, aligned with the eval-summary contract in hexo-ai#36.
Abhishek21g
commented
Jul 6, 2026
Abhishek21g
left a comment
Author
There was a problem hiding this comment.
Verified locally on Abhishek21g/sia pr/32-spaceship-evaluator: 132 tests passed. Closes the only bundled task missing an evaluator; leak-safe results.json pairs with #36.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the missing
evaluate.pyforspaceship-titanic— the only bundled task without a first-party evaluator despite README stating all four ship one.Builds on the approach in #32 with an additional integrity improvement:
results.jsonusessummaryscalars +items[]and never writes held-outTransportedlabels, aligned with the eval-summary contract in #36.Why
sia run --task spaceship-titaniccannot feed scores into the self-improvement loopresults.jsonwould leak into feedback prompts on currentmain(Fix: stop leaking held-out reference answers to the meta/feedback agents #36 fixes harness-side; this fixes grader-side)Changes
sia/tasks/spaceship-titanic/data/public/evaluate.py— Kaggle-style CSV scoringtests/test_spaceship_titanic_evaluator.py— contract + leak guard testsTest plan
python -m pytest tests/test_spaceship_titanic_evaluator.py -q— 3 passedpython -m pytest tests/ -q— 132 passedexpectedabsent from writtenresults.jsonRelated