Skip to content

fix(python): resume missing immutable portfolio artifacts safely - #1205

Merged
ruvnet merged 6 commits into
mainfrom
fix/python-portfolio-resume-20261012
Oct 13, 2026
Merged

ruvnet merged 6 commits into
mainfrom
fix/python-portfolio-resume-20261012

Conversation

@ruvnet

@ruvnet ruvnet commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

The companion 0.2.0 release stopped after uploading attention and cluster wheels because PyPI returned HTTP 429 while creating the next project. Blind retries would collide with immutable files, and rebuilding current main would change the original native dependency from 0.1.3 to 0.1.4.

This PR safely resumes the 22 original artifacts from source fccf36e9a76d35bce6750dfd3b25802cae3d683b, run 38169271248, artifact 11713787141. Archive SHA256: b4c92cae86c753305abef7aad395fdd9d91c7e84c3c887342defa7a3e4607241.

Changes:

  • Bounded, read-only preflight checks the pinned inventory, artifact metadata, exact native requirement and existing PyPI file hashes; it stages only missing originals.
  • Recovery PRs run three-platform artifact acceptance and a fresh frozen native audit. Main-only publication is explicit, defaults off, uses the existing protected pypi environment, and shares a publication lock with ordinary portfolio publishing. Registry reconciliation repeats under that lock; final checks verify every file and fresh PyPI-only installs.
  • Scoped LF checkout policy preserves the byte-pinned inventory on Windows. No hash checks, tests or release protections are weakened.
  • Targeted npm and Quinn dependency updates repair security gates. The single unmaintained YAML dependency is replaced with bounded serde-saphyr parsing, preserving metadata, boolean and merge-key behavior. Only the affected middleware declares Rust 1.89; frozen Python SDK dependencies are unchanged.
  • Fixes a same-shard replication deadlock, server readiness before bind, and the serving test's premature stdout-pipe closure. The ORT-dependent core CI shard uses Ubuntu 24.04 for its required ABI.
  • ADR, accessible animated SVG, truthful availability documentation, reproducible consumer tests and dated evidence are included. Existing tested tutorial gist: https://gist.github.com/ruvnet/2491c7631bf3e186e815196f52fd8954.

Validation and current state (2026-10-13):

  • Independent review approved the recovery design and all repairs; reviewer reran 62 recovery contracts, seven dependency consumer groups and 23 exact-source YAML tests.
  • 103 original artifact contract tests and 11 README quickstarts passed locally. Linux, macOS and Windows hosted recovery acceptance have each passed during repair; all must pass together on final head.
  • Both hosted Cargo vulnerability audits and npm audit passed after their respective repairs. Full latest-head supply-chain, middleware and workspace CI remain required.
  • All 27 replication tests pass; new deadlock regression fails against original code. Six strict-serving tests pass, including 50 fresh-process repetitions of the affected HTTP case.
  • actionlint 1.7.7 passes. Existing helper security scan found no issues; secret-scan signals were public provenance hashes. The npm consumer test uses an exact affected closure, not a claim of full workspace clean-install success.
  • Latest head: 3b37b41b8ebdf88c5556891ecaeb669d400526ae. CI: recovery 38391969695, supply-chain 38391969628, Workspace CI 38391969737. No merge, dispatch or upload has occurred. Two matching files and 20 missing files were last verified on PyPI.

Evidence: python/portfolio/evidence/2026-10-13-release-repair.md plus prior frozen inventory/security records. Failed or pending gates stop promotion even where GitHub does not enforce them. No retry loop, skip-existing workaround, credential expansion or fabricated rate-limit cooldown is used.

Rollback: stop the recovery workflow and revert this PR through review. Preserve original artifacts/receipts. Published files cannot be overwritten; any post-release failure requires stopping promotion and evaluating yanking or a separately reviewed patch version. New packages remain deferred until this backlog is delivered.

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

2026-10-13 fail-closed release reconciliation

Fresh state (06:31–06:32 UTC):

  • This draft remains the sole frozen companion recovery branch at d902ebe4d697fb25a446af4c5ba1e4cf96fd861e; no publisher or supervisor is running.
  • PyPI still has 2/22 expected companion 0.2.0 files (attention and cluster wheels); 20 original files remain missing. No new upload occurred.
  • The public attention gist is unchanged at revision 7c09fe56274683c0b838ed0a2b176abdd86ff079; its exact fetched script passed again.
  • All 59 hosted checks completed: 48 passed, 7 skipped, 4 failed. Recovery contract passed. Failures are npm audit plus Workspace CI core-and-rest (linker ABI symbols), core-and-rest-heavy (one localhost connection race), and core-platform (replication promotion timeout).

Fresh npm 10.8.2 production audit against lock SHA-256 6b55f7af387854b24e249bc1ea7de831f5cb5e51f899d96ee467038b2ef83db2 reports 1 critical, 3 high, 5 moderate nodes:

  • critical proxy-addr@2.0.7 (GHSA-jqcg-44mw-7w3h; fixed 2.0.8);
  • high @modelcontextprotocol/sdk@1.30.0 (fixed >=1.31), axios@1.18.1 (fixed 1.20.0), and compression@1.8.1 (fixed 1.8.2);
  • moderate fast-copy@3.0.2, ip-address@10.3.1, and qs@6.15.3 effects through Express/body-parser.

Existing Dependabot PRs overlap exactly: #1129, #1127, #1102, #1130 and #1128. They are stale/UNSTABLE and must not be merged blindly or duplicated by overwriting their branches. #1129's npm-audit job proves the critical lock bump clears the current critical-only workflow gate, but it leaves the three high nodes and the workflow's stated Q3 2026 backlog deadline has passed. Release acceptance therefore requires one current-main npm-security owner to consolidate the fixes, reach zero current production audit findings (or obtain explicit authorized risk acceptance with compensating control/deadline), run affected workspace/packed-consumer tests, and pass every required hosted check.

No generic npm audit fix, forced lock rewrite, merge, recovery dispatch or upload was accepted. Publication remains stopped.

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Publication follow-through, 2026-10-13 11:28 UTC: head 5a19341274e6e35497bf0f6bfac7021e366042d2 adds targeted repairs for the four observed gate failures. Seven npm dependency records and three SDK manifests updated; qs override makes the Express/body-parser resolution explicit. Production audit now reports zero findings. Independent reviewer approved the scoped dependency diff; clean affected-dependency consumer validation is continuing (this is not a claim that the full npm workspace installs cleanly). Rust repairs release the DashMap guard before accessing a second replica, bind the HTTP listener before printing readiness, and run only the ORT-containing CI shard on Ubuntu 24.04 for its libc/libstdc++ ABI. Existing coverage is retained and a bounded same-shard promotion regression added. Independent code review passed; targeted native compile/tests remain in progress. Recovery helper: 62 tests passed again; original artifact preflight validated the two existing files and selected20 missing originals without uploading. Three-OS original-artifact preflight and frozen native audit now run on PRs too; publish still requires explicit main dispatch, all gates and protected pypi environment. No merge/upload yet. Existing Dependabot #1102/#1127/#1128/#1129/#1130 branches are untouched; this commit integrates only the bounded current-source fixes needed for recovery.

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Release repair checkpoint (2026-10-13 11:42 UTC): head 3b37b41b8ebdf88c5556891ecaeb669d400526ae contains all independently reviewed fixes. No merge or upload yet.

  • Targeted npm repair passed hosted audit and seven functional consumer groups in a clean affected dependency closure; reproducible scripts committed. Full workspace clean installation is not claimed.
  • Quinn 0.11.18 cleared both hosted Cargo vulnerability audits on prior head fbf30da. Frozen SDK closure excludes Quinn; original native audit passed.
  • Recovery Linux and Windows preflight passed on fbf30da; macOS passed earlier run38390477781 and must pass again on final head. Scoped LF checkout policy preserves the pinned inventory hash.
  • Replication: 27 tests pass, with deterministic failing baseline for the new deadlock regression. Strict serve: six tests pass; affected HTTP case passes50/50 fresh-process runs. Assertions unchanged.
  • Replaced sole unmaintained serde_yaml/unsafe-libyaml use with bounded serde-saphyr1.3.0 parsing, preserving string metadata, merge-key and boolean semantics. Independent reviewer reran23/23 exact-source parser tests. Middleware explicitly requires Rust1.89; full middleware CI remains required. Frozen Python SDK unchanged.

Evidence and rollback: python/portfolio/evidence/2026-10-13-release-repair.md. Latest CI includes recovery run38391969695, supply-chain38391969628 and WorkspaceCI38391969737. Superseded PR-only runs were cancelled to release runners; no release workflow was cancelled or retried. Promotion remains blocked until all required final-head checks pass.

@ruvnet
ruvnet marked this pull request as ready for review October 13, 2026 11:43

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Automatic recovery supervisor started on October 13 at 12:33:40 UTC under the existing publication authorization. Owner: /root; this is now the sole merge/dispatch executor. No merge or upload has occurred.

Reviewed head: 3b37b41b8ebdf88c5556891ecaeb669d400526ae; tree 9a4782604f648dd19fda346d89510ddbf90fb293; base 31549f798997cbb16b1ec872ca30b9e690bbd3ff.

Current gates: 85 successful checks, 7 expected conditional skips, 2 Rust shards still running in Workspace CI. Prior successful durations were about 62 minutes for the WASM shard and up to 76 minutes for heavy ML. All three recovery preflights and supply-chain audits pass.

The supervisor polls every 10 minutes for up to four hours (deadline 16:33:40 UTC). It requires the full observed inventory of 94 checks and 12 workflows, all additional checks, review state and unchanged source identity. It performs at most one expected-head merge and one protected main recovery dispatch, reconciles uncertain responses without retry, and stops on failed gates or conflicting publishers. Only the known unchanged-version ordinary portfolio push is classified as nonpublishing. Publication success requires the recovery workflow's exact-file verification and all three fresh PyPI consumer jobs.

Independent review and 24 mocked failure/reconciliation tests passed locally and on the execution host. Supervisor SHA-256: d593188cb73800228fe0a17a606dd9cd23f1a1a9dddfb05a715c84900882175e. Work Desktop PID 1595; receipt /home/ruv/releases/python-recovery-20261013-1233/receipt.json. Independent first-cycle readback confirmed waiting_for_gates at 12:33:46 UTC. Keepawake covers the full deadline. Milestone and terminal receipts will be posted here; no second publisher should be launched.

GitHub does not provide expected-base CAS for merge or expected-SHA CAS for main workflow dispatch. The supervisor checks immediately before and reconciles resulting tree/run identity afterward; this narrow service race remains a documented limitation.

@ruvnet
ruvnet merged commit 25ed48e into main Oct 13, 2026
94 checks passed
@ruvnet

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

RuVector recovery supervisor evidence

Owner: root
Phase: merge_confirmed
Reason: none
Reviewed head: 3b37b41b8ebdf88c5556891ecaeb669d400526ae
Merge: 25ed48e9e31823dd8bf56533881ecebc56d131ba
Release run: not_confirmed
No direct upload. An uncertain comment or mutation is never blindly retried.

@ruvnet

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

RuVector recovery supervisor evidence

Owner: root
Phase: release_running
Reason: none
Reviewed head: 3b37b41b8ebdf88c5556891ecaeb669d400526ae
Merge: 25ed48e9e31823dd8bf56533881ecebc56d131ba
Release run: https://github.com/ruvnet/RuVector/actions/runs/38402061555
No direct upload. An uncertain comment or mutation is never blindly retried.

@ruvnet

ruvnet commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

RuVector recovery supervisor evidence

Owner: root
Phase: stopped
Reason: release_failed_no_retry
Reviewed head: 3b37b41b8ebdf88c5556891ecaeb669d400526ae
Merge: 25ed48e9e31823dd8bf56533881ecebc56d131ba
Release run: https://github.com/ruvnet/RuVector/actions/runs/38402061555
No direct upload. An uncertain comment or mutation is never blindly retried.

ruvnet commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner Author

October 14 verification update: this PR merged as 25ed48e9e31823dd8bf56533881ecebc56d131ba; final pre-merge checks were 87 success / 7 expected skips. The earlier PR-body statement that no merge/upload had occurred is historical.

Recovery run 38402061555 passed all three platform preflights and the frozen native audit, uploaded six additional wheels, then received PyPI HTTP 429, “Too many new projects created,” at ruvector-rvf. The supervisor stopped without retry. Fresh registry verification finds 8/22 original files with exact hashes/sizes and none yanked: attention, cluster, core, gnn, graph, rerank, router and ruvllm wheels at 0.2.0. Three wheels and all eleven sdists remain missing. Final post-publication platform jobs were skipped, not passed.

Fresh Linux CPython 3.12.3 / native 0.1.3 verification: 38 unchanged contract tests, eight README quickstarts, pip check and the exact gist pass. pip-audit 2.10.1 reports zero known vulnerabilities across 43 installed dependencies, with no skipped distributions. Coverage limits and reproducible commands are in draft evidence PR #1218, head 65a0083ababeb60228ac5e1e77310b9d97bef797.

Existing gist updated, then fetched back and rerun successfully: https://gist.github.com/ruvnet/2491c7631bf3e186e815196f52fd8954 revision dbe58b58696153172bcaa3730500e3d2f606ea43 (Python script unchanged).

No active publisher found; no upload, rerun, supervisor or new package was started today. No applicable fixed reset time is established. The next owner must recheck ownership, exact missing files and release gates before protected recovery; do not blindly replay the stopped supervisor or evade the project-creation limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant