Skip to content

Commit 5325893

Browse files
authored
docs: multi-scenario evaluation example and failure classification for agentic workflow agent (#49891)
1 parent e8ef5d5 commit 5325893

1 file changed

Lines changed: 30 additions & 1 deletion

File tree

.github/aw/create-agentic-workflow.md

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,10 +86,39 @@ Use this mode for exploratory testing, persona walkthroughs, and "what workflow
8686
- Do not create or edit workflow files.
8787
- Return a compact recommendation covering trigger, any scoped `paths:` filters for file-event triggers, read tools, safe outputs, permissions, and explicit `noop` criteria.
8888
- For recurring reports or digests, always include the report window, grouping dimensions, and deduplication key. See [triggers.md](triggers.md) for key-format examples.
89-
- If custom-agent invocation is unavailable in the environment, use fallback evaluation mode: provide the recommendation directly from local gh-aw guidance, record scoring/response quality as unavailable, and continue with partial-results reporting.
9089
- Exit ad hoc evaluation mode only when the user explicitly asks to create, implement, or write the workflow file.
9190
- End by offering to turn the recommendation into `.github/workflows/<workflow-id>.md` if the user wants to proceed.
9291

92+
### Multi-Scenario Evaluation Example
93+
94+
To compare multiple persona or task slices in a single request, use the following prompt format:
95+
96+
> agentic-workflows evaluate these scenarios without creating files:
97+
> 1. Information Worker — weekly digest of open issues and PRs assigned to me
98+
> 2. Product Manager — recurring backlog triage report sorted by staleness
99+
> 3. Backend Engineer — API contract diff review on every pull request
100+
101+
For each scenario, return a compact recommendation table covering:
102+
103+
| Field | Value |
104+
|---|---|
105+
| Trigger | event + key options |
106+
| Scope | paths filter or scheduling window |
107+
| Read tools | gh-proxy toolsets or optional tools |
108+
| Safe outputs | add-comment / create-issue / noop |
109+
| Permissions | read-only scopes required |
110+
| Noop condition | when no action is needed |
111+
112+
### Failure Classification
113+
114+
When evaluating scenarios, classify any failure before stopping:
115+
116+
| Failure type | Symptom | Action |
117+
|---|---|---|
118+
| Transient issue | Network error, timeout, or quota exceeded | Retry once; if it persists, record `invocation_unavailable` and continue with partial results |
119+
| Unsupported command | Unknown subcommand or unrecognized option | Record `command_not_supported`, document the gap, and fall back to providing the recommendation directly from local gh-aw guidance |
120+
| Product gap | Invocation succeeds but returns no workflow-design guidance | Record `response_unavailable`, note the scenario, and surface it as a missing capability rather than treating it as an error |
121+
93122
## Design Checklist
94123

95124
### 1. Pick the workflow ID

0 commit comments

Comments
 (0)