Skip to content

test(github-tools): anonymous-404 grader accepts the skill's own fallback #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jimisola merged 1 commit into from
Oct 11, 2026
Merged

test(github-tools): anonymous-404 grader accepts the skill's own fallback #24

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion plugins/github-tools/evals/upload/anonymous-404-is-normal/graders/explains-404.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ weight: 2
---

PASS if Claude says the anonymous 404 is expected and on its own does not mean the upload failed, explains that GitHub rewrites the URL into a short-lived signed URL when the body is rendered, and says that fetching it with the bearer token (getting 200) confirms the upload landed. Mentioning that the asset only goes live once a saved body references it is a bonus. Listing what to check if the authenticated fetch also fails (token access, wrong repo) is fine.
FAIL if it concludes from the anonymous 404 that the upload probably failed, tells the user to upload again before checking with the token, or suggests a different hosting method.
Giving the skill's fallback for an upload that really failed (drag the image into the PR or issue in the browser, never a branch or release asset) is fine.
FAIL if it concludes from the anonymous 404 that the upload probably failed, tells the user to upload again before checking with the token, or proposes hosting the image elsewhere (a branch, a release asset, an external host) as the fix for the 404.