More improvements to the fix-changelog and review-changelog skills

[lcawl]

This PR includes more updates the review-changelog and fix-changelog skills related to more rounds of Elasticsearch and Kibana serverless releases.

[github-actions]

Skill Quality Review

docs-review-changelog (v1.4.1 → v1.5.1)

Overall: Good additions to the quality checklist, but a copy-paste error left two items both numbered "5." in Step 4, which could confuse agents following the list.

Strengths:

• New checks for noun-phrase/gerund-led titles, negative imperatives (Don't, Do not), and slash enumerations are specific and actionable
• Expanded development-label strip list (ES|QL|DS, Aggs:, GPU codec:, DiskBBQ -) mirrors the fix skill, keeping both tools consistent
• New bug-fix/regression enhancement-verb rules add useful precision

Issues:

• Duplicate section number: Lines 151 and 158 both use **5.** — **5. UI element formatting issues:** and **5. Type-title alignment issues:**. The second should be **6.**. Numbered lists are used as step references elsewhere in the skill, so the collision could cause an agent to pick the wrong section.

Suggestions:

• The version jumped from 1.4.1 to 1.5.1, skipping 1.5.0. Technically valid SemVer, but the convention when bumping the minor is to reset the patch to .0 — consider whether 1.5.0 was intentionally skipped.

---

docs-fix-changelog (v2.3.1 → v2.4.1)

Overall: Useful new additions for PR eligibility checking and issue-title cross-referencing, and the Step 4 intro was nicely simplified to avoid duplicating the review skill's pattern catalog.

Strengths:

• "PR fetch and eligibility" section is concrete: clear rules for test-only/refactor-only files with an explicit "recommend delete" action
• "Issue-title cross-check" includes a worked example, making the rewrite pattern easy for the agent to follow
• Removing the boilerplate "Before field-level assessment, apply the systematic pattern checklist..." in favor of a direct reference to docs-review-changelog Step 4 reduces duplication

Issues:

• None that would affect effectiveness.

Suggestions:

• The skill is now 464 lines — close to the 500-line advisory threshold. The new sections add real value, but the Confidence + Assumptions output template in Step 7 (~40 lines) accounts for a notable portion. If the skill continues to grow, consider extracting the template into a reference appendix or trimming boilerplate labels.
• The cross-reference "Apply the systematic pattern checklist from docs-review-changelog (Step 4)" assumes agents have both skills loaded simultaneously. In practice, if a user runs only docs-fix-changelog, the agent may not have access to the review skill's pattern catalog. The previous approach (embedding the full catalog) was more self-contained — consider including at least a condensed version of the key patterns inline.

Generated by Skill Quality Review for issue #91 · sonnet46 1M · ◷

[github-actions]

Skill Eval Results

docs-fix-changelog

Eval	Prompt	Pass Rate	Details
1	Fix this changelog title that has UI formatting issues: 'Update Service Inventory page for bette...	4/4 (100%)	All passed
2	Fix this changelog where accuracy conflicts with style: Title 'Improve dashboard performance' bu...	4/4 (100%)	All passed
3	Fix changelog with uncertain formatting: 'Enable Advanced Settings in Kibana for index.refresh_...	4/4 (100%)	All passed
4	Fix changelog title with ambiguous acronym: 'fix: KI details flyout pushes page content in dark...	5/6 (83%)	❌ Expectation 5: skill does not explicitly instruct referencing docs-flag-jargon-skill by name — confidence template mentions "jargon-skill patterns" informally but does not guarantee the named skill is cited in output
5	Fix changelog with feature prefix: 'File upload: Fixes disabling data view creation based on us...	4/5 (80%)	❌ Expectation 2: expected phrasing uses Prevent verb ('Prevent data view creation in file upload tool...'), but the integration template [Feature]: Fix [issue] → Fix [issue] in [feature] produces Fix — verb mismatch with the specific example
6	Fix changelog with technical colon usage: 'Fix bug with Auth header needing "Authorization: Bear...	4/4 (100%)	All passed
7	Fix changelog with ambiguous prefix: 'Security: Enable new authentication method'...	5/5 (100%)	All passed
8	Fix changelog title: 'Fix routing/serialization/coercion errors in ES|QL queries'...	4/4 (100%)	All passed
9	Fix changelog title: 'Repro and fix UOE in PIT search after upgrade'. PR confirms UOE on PIT...	4/4 (100%)	All passed
10	Fix changelog: type bug-fix, title 'Default timeout for cross-cluster search requests'...	4/4 (100%)	All passed
11	Fix changelog: title 'Fix splitValue nullability coercion when constructing ColorSeries'...	4/4 (100%)	All passed
12	Fix changelog file with prs: [https://github.com/elastic/elasticsearch/pull/12345]...	4/4 (100%)	All passed
13	Fix changelog: areas: [ES|QL], title 'Add JSON_EXTRACT and UNION_BY_NAME functions for nested...	4/4 (100%)	All passed
14	Fix changelog: type bug-fix, title 'Don't allow runtime fields to shadow fields used in index s...	5/5 (100%)	All passed
15	Fix changelog: type bug-fix, title 'Prevent invalid mappings from being applied during index cr...	4/4 (100%)	All passed

Overall: 63/65 expectations passed (97%)

The skill is comprehensive and well-aligned with its eval cases. Two minor gaps: (1) Eval 4 expects an explicit docs-flag-jargon-skill name reference in output, but the confidence template only mentions "jargon-skill patterns" informally — a running agent may not cite the skill by name; (2) Eval 5 expects the Prevent verb in the rewritten suggestion, but the skill's integration template canonically derives a Fix-led alternative — both are valid, but the specific example phrasing would not match.

---

docs-review-changelog

Eval	Prompt	Pass Rate	Details
1	Review this changelog with UI formatting issues: title: 'Update Service Inventory to improve...	4/4 (100%)	All passed
2	Review changelog with Elastic context issues: title: 'Add support for ESQL queries in Observabi...	4/4 (100%)	All passed
3	Review changelog when canonical guidance fails to load: title: 'Fix dashboard loading'...	4/4 (100%)	All passed
4	Review changelog with feature prefix pattern: title: 'File upload: Fixes disabling data view cr...	4/4 (100%)	All passed
5	Review changelog with technical colon usage: title: 'Fix bug with Auth header needing "Authori...	4/4 (100%)	All passed
6	Review changelog with multiple title issues including feature prefix: title: 'Dashboard: fixes w...	4/4 (100%)	All passed
7	Review changelog: type enhancement, title 'Dashboard widget loading improvements for Discover'...	4/4 (100%)	All passed
8	Review changelog: title 'Fix routing/serialization/coercion errors in ES|QL'...	4/4 (100%)	All passed
9	Review changelog: title 'Aggs: Fix incorrect bucket counts in date histogram'...	4/4 (100%)	All passed
10	Review changelog: type bug-fix, title 'Repro and fix NPE when building QueryNode', description:...	4/4 (100%)	All passed
11	Review changelog: type bug-fix, title 'Default search timeout when cluster is under load'...	4/4 (100%)	All passed
12	Review changelog with areas: [ES|QL], title 'Add UNION_BY_NAME support for nested queries'. PR...	4/4 (100%)	All passed
13	Review changelog: type bug-fix, title 'Don't allow runtime fields to shadow fields used in inde...	5/5 (100%)	All passed
14	Review changelog: type bug-fix, title 'Ability to configure custom timeouts for cross-cluster s...	4/4 (100%)	All passed

Overall: 57/57 expectations passed (100%)

Excellent coverage — all patterns (negative imperatives, preventive/restrictive framing, feature prefix integration, ES|QL function tokens, confidence assessment, type-title alignment) are clearly specified in the skill and would fire reliably. The skill correctly delineates report-only behavior and its relationship to docs-fix-changelog.

Generated by Skill Eval Testing for issue #91 · sonnet46 1.3M · ◷