Distill security policy PR outcomes

Author: bmtriet

evals/2026-06-12-public-security-policy-pr-loop.md

@@ -0,0 +1,41 @@
+# Public Security Policy PR Loop Evaluation
+
+Date: 2026-06-12
+
+## What Was Checked
+
+The public security policy PR workflow after three public-safe attempts:
+
+- Dify: source issue plus merged docs PR.
+- Aider: source issue plus open docs PR.
+- mem0: source issue closed as fixed, docs PR closed unmerged.
+
+## Evidence
+
+- Dify PR merged, which validates that a small `SECURITY.md` contribution can be acceptable proof-of-work.
+- Aider PR remains open, so it is not yet a success or failure signal.
+- mem0 issue closed as fixed after maintainers provided or enabled the private reporting path, while the docs PR closed without merge.
+- LLMGate `gpt-5.5` reviewed the pattern and recommended narrower preflight rules, stop conditions, and outcome labels.
+
+## Passed
+
+- The workflow avoids public vulnerability details.
+- The docs-only PR shape is small and reviewable.
+- At least one large public repo accepted the contribution.
+
+## Failed Or Mixed
+
+- Counting every opened PR as positive evidence would overstate the skill quality.
+- mem0 shows that a valid trigger can still produce a stale or unnecessary PR once maintainers solve the need elsewhere.
+- The previous workflow did not explicitly require a last-minute issue timeline and linked-PR recheck before opening.
+
+## Changes
+
+- Keep `public-security-policy-pr` as `candidate`, not formal.
+- Add preflight checks for closed issues, active maintainer fixes, linked PRs, and newly added policy files.
+- Treat closed-unmerged PRs tied to fixed issues as trigger validation only.
+- Do not repeat this pattern unless the issue is still open and no maintainer-owned fix path is visible.
+
+## Next Decision
+
+Run at most one more carefully preflighted public security policy PR before promoting, retiring, or narrowing this candidate further.

memory/operating-notes.md

@@ -17,6 +17,8 @@ These are public-safe notes about how FreeCodex should operate. They are not raw
 - Keep all outbound inside `docs/autonomous-outreach-policy.md`.
 - Prefer proof-of-work PRs and concrete mini-audits over generic sales messages.
 - Never include secrets, private user data, raw chat logs, or private project material in public artifacts.
+- Treat mixed public PR outcomes honestly: a closed-unmerged PR can validate the problem while disproving the contribution path.
+- Before opening proof-of-work PRs, re-check issue state, linked maintainer work, and repo contents immediately so stale work does not create noise.
 
 ## Confirmed Decisions
 

ops/mission-report.md

@@ -2,7 +2,7 @@
 
 Generated by `python3 scripts/mission_control.py generate`.
 
-Generated at UTC: `2026-06-07T14:10:27Z`
+Generated at UTC: `2026-06-11T23:33:56Z`
 
 ## Recent Work
 

ops/outcomes.md

@@ -8,7 +8,7 @@ This file is generated from public sent logs only. It never posts outreach and d
 
 | Metric | Value |
 | --- | ---: |
-| Generated at UTC | 2026-06-07T14:10:27Z |
+| Generated at UTC | 2026-06-11T23:33:56Z |
 | Tracked outbound artifacts | 15 |
 | Pull requests tracked | 9 |
 | Issue comments tracked | 6 |
@@ -33,4 +33,4 @@ This file is generated from public sent logs only. It never posts outreach and d
 | pull_request | leads/sent/2026-06-05-github-outreach-008.md | https://github.com/modelcontextprotocol/servers/pull/4282 | https://github.com/modelcontextprotocol/servers/pull/4282 | OPEN | False | 1 | 0 | 2026-06-05T14:07:34Z | - |
 | pull_request | leads/sent/2026-06-05-github-outreach-009.md | https://github.com/Aider-AI/aider/pull/5218 | https://github.com/Aider-AI/aider/pull/5218 | OPEN | False | 1 | 0 | 2026-06-07T14:04:35Z | - |
 | issue_comment | leads/sent/2026-06-07-github-outreach-010.md | https://github.com/open-webui/open-webui/issues/19313#issuecomment-4642847395 | https://github.com/open-webui/open-webui/issues/19313 | OPEN | - | 39 | - | 2026-06-07T13:43:10Z | parent issue state only |
-| pull_request | leads/sent/2026-06-07-github-outreach-011.md | https://github.com/mem0ai/mem0/pull/5417 | https://github.com/mem0ai/mem0/pull/5417 | OPEN | False | 1 | 0 | 2026-06-07T14:09:31Z | - |
+| pull_request | leads/sent/2026-06-07-github-outreach-011.md | https://github.com/mem0ai/mem0/pull/5417 | https://github.com/mem0ai/mem0/pull/5417 | CLOSED | False | 1 | 0 | 2026-06-11T15:46:57Z | - |

ops/scoreboard.md

@@ -6,7 +6,7 @@ Generated by `python3 scripts/mission_control.py generate`.
 
 | Metric | Value |
 | --- | ---: |
-| Generated at UTC | 2026-06-07T14:10:27Z |
+| Generated at UTC | 2026-06-11T23:33:56Z |
 | Sent log files | 11 |
 | GitHub PRs opened | 9 |
 | GitHub issue comments sent | 5 |

skills/pipeline/public-security-policy-pr.md

@@ -28,11 +28,13 @@ Never include credentials, private reports, exploit details, or confidential vul
 
 1. Confirm neither `SECURITY.md` nor `.github/SECURITY.md` exists.
 2. Confirm the repository's existing private reporting path, or confirm that the public issue is asking maintainers to provide one.
-3. Add a concise `SECURITY.md` that points reporters to GitHub Private Vulnerability Reporting if enabled, or asks them to request a private channel without sharing details if it is not enabled yet.
-4. Warn reporters not to disclose vulnerabilities in public issues, discussions, or pull requests.
-5. Keep scope to reporting guidance, disclosure etiquette, and update guidance.
-6. Avoid unsupported claims about SLAs, bounties, supported versions, or guaranteed security response.
-7. Open a small PR that references the source issue and states that no sensitive details are included.
+3. Re-check the source issue state, linked PRs, maintainer comments, and repository contents immediately before opening a PR.
+4. Stop if the issue is already closed as fixed, has an active maintainer-owned fix path, or a policy appeared since triage.
+5. Add a concise `SECURITY.md` that points reporters to GitHub Private Vulnerability Reporting if enabled, or asks them to request a private channel without sharing details if it is not enabled yet.
+6. Warn reporters not to disclose vulnerabilities in public issues, discussions, or pull requests.
+7. Keep scope to reporting guidance, disclosure etiquette, and update guidance.
+8. Avoid unsupported claims about SLAs, bounties, supported versions, or guaranteed security response.
+9. Open a small PR that references the source issue and states that no sensitive details are included.
 
 ## Validation
 
@@ -59,14 +61,28 @@ Stop or revise when:
 - the proposed text claims unsupported SLA, bounty, or version support
 - the PR would expose vulnerability details or encourage public disclosure
 - the repository asks security reports to use a different official channel
+- the source issue is already resolved, assigned to a maintainer implementation, or has a linked maintainer fix in progress
 
 ## Evidence
 
-This candidate is based on repeated public-safe documentation PRs:
+This candidate is based on public-safe documentation attempts with mixed outcomes:
 
 - Dify source issue: https://github.com/langgenius/dify/issues/36692
 - Dify merged PR: https://github.com/langgenius/dify/pull/36873
 - Aider source issue: https://github.com/Aider-AI/aider/issues/5217
 - Aider PR: https://github.com/Aider-AI/aider/pull/5218
 - mem0 source issue: https://github.com/mem0ai/mem0/issues/5385
 - mem0 PR: https://github.com/mem0ai/mem0/pull/5417
+
+Outcome interpretation:
+
+- Dify is positive PR validation because the docs change merged.
+- Aider is unresolved evidence until maintainer action, merge, or closure.
+- mem0 validates the trigger, not the PR path: the source issue closed as fixed after maintainers handled the private reporting channel, while the docs PR closed unmerged.
+
+## Next Experiment Rules
+
+- Prefer repositories where the issue is still open and explicitly asks for `SECURITY.md` or private reporting guidance.
+- Only open a PR when no maintainer fix is visible after checking issue timeline, linked PRs, and repository contents.
+- Treat a closed-unmerged PR with a fixed related issue as "need solved elsewhere," not as merged-proof evidence.
+- Limit the next run to one carefully preflighted PR before promoting this candidate into a formal reusable skill.