[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-25

[github-actions[bot]]

Summary

• Date of test: 2026-09-25
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start Guide)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a first-time visitor, the site was easy to follow — the home page immediately answers "what is this?" and points to a "Get Started with CLI" button, and the Quick Start guide walks through a concrete, copy-pasteable example (adding the repo-status sample workflow) rather than an abstract tutorial. I hit no dead ends or broken commands.

Critical Issues Found

None. I did not encounter any broken links, 404s, or commands/flags that don't exist in the actual CLI (I cross-checked gh aw add-wizard against the Go source and it matches the docs).

Confusing Areas

1. 🟡 Authentication section is long and branches by engine — the "Configuring authentication" block on the Quick Start page covers Copilot, Claude, Codex, Gemini, and Pi all in one scroll, with two sub-paths for Copilot alone (org billing vs. COPILOT_GITHUB_TOKEN). A brand-new user who just wants "the GitHub Copilot path" has to visually skip past 4 other engines' instructions to find the ~3 lines that apply to them. A collapsible/tabbed UI per engine (many Starlight docs use tabs for this) would reduce scroll fatigue.
2. 🟡 "Frontmatter" jargon appears before it's fully explained — the opening paragraph mentions "YAML frontmatter (the configuration block between the --- markers...)" inline, which is a nice touch, but the term reappears later ("If you changed the frontmatter...") without a link back to that definition or to a glossary — worth a quick anchor link.
3. 🟡 The "report will look something like this" image loads lazily — on first paint the image placeholder briefly shows before content loads (confirmed it resolves to HTTP 200), which is normal for a dev server but could look like a broken image to an impatient user on a slow connection.

What Worked Well

• 🟢 Home page hero states the value proposition in one sentence and has clear, prominently-placed "Get Started with CLI" and "Creating Workflows" buttons — no hunting required.
• 🟢 Quick Start's Prerequisites list is concrete and actionable (specific gh version, exact commands to check auth status, explicit OS support) rather than vague ("modern OS").
• 🟢 The step to add a workflow (gh aw add-wizard githubnext/agentics/repo-status) is copy-pasteable and the guide explains up front what the interactive wizard will do (6 clearly enumerated sub-steps), which sets good expectations before running an unfamiliar command.
• 🟢 CLI Commands page leads with a "Day-one commands" table and a callout telling new users to ignore the advanced/enterprise sections further down — this is a great way to avoid overwhelming beginners with the full command surface.
• 🟢 Version-pinning instructions (tag, SHA, upgrade flow) are given right after the basic install command, which is useful without being in the way.

Recommendations

Quick wins:

• Add a small note/anchor near the second "frontmatter" mention linking back to its first definition.
• Consider tabs (Starlight supports (Tabs)) for the per-engine authentication instructions so a Copilot-only user isn't scrolling past Claude/Codex/Gemini/Pi setup.

Longer-term:

• Add a one-line "estimated total time including auth setup" near the top, since the 10-minute estimate likely assumes secrets are already configured.
• A troubleshooting/FAQ callout specifically for the Quick Start page (e.g., "workflow run failed — check this first") would help beginners self-serve before filing an issue.

Screenshots

📎 home.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/795ef67c0bc653bfbac94d121e9882a7bfdb5ba169b9034762010e0351404624.png?raw=true

📎 quick-start-full.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8ef76fb7ad38fd43d6ace44849847184b943313766eb0619b70a43f9cfee6f9f.png?raw=true

📎 cli-commands.png — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5b69ac8eba0db6dc8a729129b723e921964be87b9bf4c254988935bc1cd4d374.png?raw=true

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• clients2.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "clients2.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 58.5 AIC · ⌖ 7.45 AIC · ⊞ 10.7K · ◷

• expires on Sep 25, 2026, 7:56 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-26T03:56:24.030Z.

Closed by Workflow