Enhance agent onboarding and usability testing documentation#4
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR expands the SerpApi documentation set to improve agent onboarding and correct engine/response-key selection, and introduces a new “agent-usability-test” methodology skill for evaluating whether agents can discover and successfully use tools under realistic conditions.
Changes:
- Enhanced
serpapi-web-searchdocumentation with setup verification, engine-selection gotchas, response-key mapping clarifications, and agent-loop usage patterns. - Added a new
agent-usability-testskill plus a concreteserpapi-clitrace-capture/analysis recipe and lessons. - Reworked repository-level onboarding docs (
README.md,AGENTS.md) around MCP-first integration and updated.gitignore.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/serpapi-web-search/SKILL.md | Adds detection/verification guidance, clarifies --fields vs --jq, and expands engine-selection gotchas and composition patterns. |
| skills/serpapi-web-search/rules/use-cases.md | Adds agent-loop integration patterns (compaction safety, delegation, budget-gated fan-out). |
| skills/serpapi-web-search/rules/response.md | Clarifies Google Maps result keys for list vs single-place responses. |
| skills/serpapi-web-search/rules/parameters.md | Improves q exceptions guidance and recommends explicit num usage. |
| skills/serpapi-web-search/rules/examples.md | Adds response-shape notes, curl fields= equivalence, and worked examples for Maps/Finance/non-standard query params. |
| skills/serpapi-web-search/LESSONS.md | Refines tags and adds new lessons on compaction resilience and delegation patterns. |
| skills/agent-usability-test/SKILL.md | Introduces AUT methodology: failure modes, protocol, scoring, and fix→retest loop. |
| skills/agent-usability-test/recipes/serpapi-cli.md | Provides an end-to-end AUT recipe (trace capture, analysis, pilot run evidence). |
| skills/agent-usability-test/LESSONS.md | Summarizes empirical findings and practical testing guidance. |
| README.md | Shifts onboarding to MCP-first, documents installation options, and adds AUT overview. |
| AGENTS.md | Updates repository discovery metadata and documents the new skill and conventions. |
| .gitignore | Ignores Python files and __pycache__ in addition to .sisyphus/. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…scope, MCP auth path
|
|
||
| ## Detection & Verification | ||
|
|
||
| **Prerequisites:** API key required. For MCP, the key is embedded in the server URL (`mcp.serpapi.com/your_key_here/mcp`) — no further auth needed. For CLI/SDK, authenticate via (in priority order): |
Contributor
Author
There was a problem hiding this comment.
Already addressed in 338f565 — the MCP URL was removed entirely during the compression pass. Current SKILL.md uses auth-check block instead.
| { "url": "https://mcp.serpapi.com/your_key_here/mcp" } | ||
| ``` | ||
|
|
||
| **Local** (full control, works offline): |
Contributor
Author
There was a problem hiding this comment.
Already addressed in 338f565 — "works offline" wording removed. Current README just says "Local Execution" with the flag.
Comment on lines
+135
to
+137
| results = serpapi.search({"engine": "google_light", "q": "topic", "num": 20}) | ||
| findings = [{"title": r["title"], "url": r["link"], "fact": r["snippet"]} | ||
| for r in results["organic_results"][:5]] |
Contributor
Author
There was a problem hiding this comment.
Already addressed in 338f565 — use-cases.md no longer has raw results["organic_results"] access. The SDK examples in sdks.md keep it for demonstration but those are SDK-specific patterns.
Comment on lines
+11
to
+14
| **Claude Code:** | ||
| ```bash | ||
| claude mcp add serpapi -- npx -y @serpapi/serpapi-mcp | ||
| ``` |
Contributor
Author
There was a problem hiding this comment.
Already addressed in 338f565 — Claude Code section now shows claude mcp env serpapi SERPAPI_KEY your_key_here right after the add command.
…truth Compression pass inspired by caveman-compress + ponytail principles: - Delete AUT recipe (321 lines theater, unrunnable, no injection path) - Delete AUT LESSONS.md (merged 10 useful lines into SKILL.md) - Rewrite SKILL.md as table-first artifact (128 lines, engine→result key mapping) - Rewrite README to MCP-only install (56 lines) - Rewrite AGENTS.md to repo map + editing rules (24 lines) - Compress all rules/ and docs/ files (kill duplicate prose) - Deduplicate LESSONS.md (remove items now in SKILL.md) Deduplication: each fact lives in exactly one file. - MCP discovery rate → README - Engine selection → SKILL.md table - --fields/--jq → rules/examples.md - Auth config → docs/api-key-setup.md Before: 2268 lines across 15 files (881 added by branch) After: 944 lines across 12 files (net -283 vs origin/master)
Engine table now shows exact JSON paths (.inline_links.cited_by.total, .place_results.phone, .best_flights[0].price). Added copy-paste --jq patterns for citations, maps, flights, shopping. Expanded gotchas with scholar citation path and flights params. Informed by N=216 AUT trials: Haiku failed on key extraction because paths were undocumented. Sonnet succeeded by exploring response JSON.
Trace evidence from 216 AUT trials showed: - Haiku -20pp caused by retrying invalid auth without actionable guidance - Shopping prices lag retailer's own site (Target $279 vs actual $189)
Trace evidence: Sonnet knows serpapi exists (parametric) but fails at auth. The bottleneck is callable, not findable. Front-load the auth handshake.
…inimum Protocol additions from 240-run Haiku retest (0 errors): - HOME controls skill loading — stripped HOME for WITHOUT, not just CWD - Pre-flight checklist: verify GTs, pilot both conditions, check scoring - Same-day GT verification required (stale GTs caused false negatives) - Randomize run order (temporal confounds from sequential batches) - Don't list expanded with 4 new anti-patterns from empirical failures - LESSONS.md: 9 findings from 240+306 trials across 3 models Retest result: Haiku +48pp aggregate lift (was -20pp before auth fix).
…gine Hotels: .total_rate.lowest → .rate_per_night.extracted_lowest (correct field) Added: check_in_date/check_out_date/adults params, sort_by=8 for price sort App Store: added to engine table + gotchas (term not q, nested rating path) Both gaps identified by 240-run AUT retest (0% correct despite 100% discovery).
Runs on SKILL.md changes + weekly schedule. Hits each engine in the table, verifies documented result_key exists in response. Catches stale paths (like the .total_rate.lowest bug) before agents do. Requires SERPAPI_KEY secret.
0pp lift on Hotels/App Store wasn't doc bugs — web search answers those tasks equally well via Booking.com/AppBrain. Fixed docs, retested N=12, still 0pp. Added to pre-flight checklist: if 1 WITHOUT run passes, the task has no information asymmetry and will never show lift.
((PASS++)) returns exit code 1 when PASS=0 because the post-increment expression evaluates to 0. Switch to PASS=$((PASS + 1)).
Comment on lines
+21
to
+23
| set -euo pipefail | ||
| PASS=0; FAIL=0; SKIP=0 | ||
|
|
Comment on lines
+70
to
+79
| # Check if result key exists and is non-empty | ||
| has_key=$(echo "$resp" | python3 -c " | ||
| import json, sys | ||
| d = json.load(sys.stdin) | ||
| key = '${expected_key}' | ||
| if key in d and d[key]: | ||
| print('yes') | ||
| else: | ||
| print('no') | ||
| " 2>/dev/null || echo "error") |
Comment on lines
+24
to
+26
| # Engine table: engine | result_key | fields | ||
| # Parse from SKILL.md between "## Engine selection" and "## Gotchas" | ||
| declare -A QUERIES=( |
…ix comment - Guard: skip gracefully when SERPAPI_KEY is missing (forks) - google_maps: accept place_results OR local_results - Remove misleading 'parsed from SKILL.md' comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refine
serpapi-web-searchto cover all engines with invocation examples, multi-engine patterns, and engine selection rules.Introduce
agent-usability-testmethodology. Think of it as Nielsen's User Testing but where agent watches other agent via tmux.Changes:
serpapi-web-search: CLI examples, use cases, composition patterns, engine selection table. Agent goes from goal to correct API call without guessing parameters.agent-usability-test(new): five failure modes, binary scoring, fix→retest loop. Includes recipe with trace capture for reproducible agent observation.Tested
agent-usability-testin 8 isolated uncoached trials - agents with the skill produced correct methodology 6/8, agents without produced 0/8 (p=0.007). On specialized search tasks, the skill triggers correct engine selection where generic search settles for approximations.