From 1ae2930cbd784b6d29f95499a6b64a5c97d41b24 Mon Sep 17 00:00:00 2001 From: Jimisola Laursen Date: Sun, 11 Oct 2026 07:56:38 +0200 Subject: [PATCH] test(github-tools): anonymous-404 grader accepts the skill's own fallback The CI judge failed 2 of 3 correct replies that ended with the skill's fallback for a real failure (drag the image into the PR in the browser), reading it as "suggests a different hosting method". The rubric now allows that fallback and fails only hosting elsewhere as the fix. Signed-off-by: Jimisola Laursen --- .../upload/anonymous-404-is-normal/graders/explains-404.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plugins/github-tools/evals/upload/anonymous-404-is-normal/graders/explains-404.md b/plugins/github-tools/evals/upload/anonymous-404-is-normal/graders/explains-404.md index 3c3af9d..82bc8b3 100644 --- a/plugins/github-tools/evals/upload/anonymous-404-is-normal/graders/explains-404.md +++ b/plugins/github-tools/evals/upload/anonymous-404-is-normal/graders/explains-404.md @@ -4,4 +4,5 @@ weight: 2 --- PASS if Claude says the anonymous 404 is expected and on its own does not mean the upload failed, explains that GitHub rewrites the URL into a short-lived signed URL when the body is rendered, and says that fetching it with the bearer token (getting 200) confirms the upload landed. Mentioning that the asset only goes live once a saved body references it is a bonus. Listing what to check if the authenticated fetch also fails (token access, wrong repo) is fine. -FAIL if it concludes from the anonymous 404 that the upload probably failed, tells the user to upload again before checking with the token, or suggests a different hosting method. +Giving the skill's fallback for an upload that really failed (drag the image into the PR or issue in the browser, never a branch or release asset) is fine. +FAIL if it concludes from the anonymous 404 that the upload probably failed, tells the user to upload again before checking with the token, or proposes hosting the image elsewhere (a branch, a release asset, an external host) as the fix for the 404.