Repository navigation
test(github-tools): anonymous-404 grader accepts the skill's own fallback - #24
Merged
Merged
Conversation
…back The CI judge failed 2 of 3 correct replies that ended with the skill's fallback for a real failure (drag the image into the PR in the browser), reading it as "suggests a different hosting method". The rubric now allows that fallback and fails only hosting elsewhere as the fix. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
The eval run on #22 after #23 merged passed 18 of 19 cases on every run;
upload/anonymous-404-is-normalscored 0.33 / 0.33 / 1. The failing replies were correct: anonymous 404 is expected, check with the bearer token, the asset goes live once a saved body references it. They ended with the skill's own fallback for an upload that really failed — "don't commit it to a branch or host it as a release asset; drag the image into the PR in the browser" — and the judge read that as "suggests a different hosting method".The rubric now names that fallback as fine, and FAIL covers hosting the image elsewhere (branch, release asset, external host) as the fix for the 404.
#22's eval check stays red until this merges, since it evaluates
main's evals.Test Plan
claude plugin validate .claude plugin eval plugins/github-tools --case anonymous-404-is-normal(Sonnet 5.5 agent and judge, as CI): 3/3 runs, 9/9 judge votes PASS, $0.14