perf: incremental metadata regen (stop syncing the full pool) (#10)

The publish re-downloaded the entire pool/channel from R2 on every run so
apt-ftparchive / createrepo_c could rescan all historical packages. The pool
is ~233 x ~80 MB (~18 GB) and grows every snapshot, so sync time ballooned
past 15 min and every run was one transient R2 failure away from dying.

Make the regen incremental instead:

- apt: pull only the channel's Packages index (~KBs), append the new .deb's
  stanza with apt-ftparchive (single file), and rebuild Release - which only
  needs the index files, not the pool.
- yum: pull only the channel's repodata, then merge it with a throwaway
  single-package repo via `mergerepo_c --all --omit-baseurl` (createrepo_c
  --update alone drops packages that aren't on disk, so it can't be used
  incrementally).

Both approaches were validated in containers against the live repo's exact
arch/index layout (Architecture:all packages land in both binary-amd64 and
binary-arm64; mergerepo_c preserves every historical version and keeps
location hrefs relative). Uploads keep the retry from the previous commit.

Sync drops from ~15 min / ~18 GB to ~seconds / ~KBs.

Signed-off-by: Peter Amiri <peter@alurium.com>

Author: bpamiri

.github/workflows/wheels-released.yml

@@ -130,54 +130,40 @@ jobs:
           echo "$FINGERPRINT:6:" | gpg --batch --yes --import-ownertrust
           echo "GPG_KEY_ID=$KEY_ID" >> "$GITHUB_ENV"
 
-      - name: Sync existing pool from R2
+      - name: Pull existing Packages index from R2
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CHANNEL: ${{ steps.inputs.outputs.channel }}
         run: |
           set -euo pipefail
-          # Retry transient R2 failures (timeouts / 5xx / connection resets).
-          # The pool is unbounded and re-downloaded in full on every publish, so
-          # the sync is a long sequence of single-shot `wrangler r2 object get`
-          # calls; one transient failure used to kill the whole run. 3 attempts
-          # with backoff absorbs that, and failures are logged instead of being
-          # discarded to /dev/null so a persistent error stays diagnosable.
-          r2get() {
+          # Incremental regen: download the channel's current Packages index
+          # (~KBs) instead of the whole pool (~18 GB). regenerate-apt-metadata.sh
+          # appends the new package's stanza and rebuilds Release, so the
+          # historical .deb files never need to be pulled again.
+          pull_index() {
             local key="$1" attempt out
             for attempt in 1 2 3; do
               if out="$(wrangler r2 object get "${R2_BUCKET}/${key}" --file="$key" --remote 2>&1)"; then
+                echo "  pulled ${key} ($(wc -c < "$key") bytes)"
                 return 0
               fi
-              echo "::warning::r2 object get failed for ${key} (attempt ${attempt}/3): ${out}" >&2
+              # Missing index → first publish on this channel: start empty.
+              if echo "$out" | grep -qiE "does not exist|NoSuchKey|NoSuchObject"; then
+                : > "$key"
+                echo "  ${key} absent — starting from an empty index"
+                return 0
+              fi
+              echo "::warning::index pull failed for ${key} (attempt ${attempt}/3): ${out}" >&2
               sleep $((attempt * 5))
             done
-            echo "::error::r2 object get failed permanently for ${key}" >&2
+            echo "::error::index pull failed permanently for ${key}" >&2
             return 1
           }
-          # List all R2 objects under pool/<channel>/ and download each.
-          # apt-ftparchive needs the actual .deb files locally so it can compute
-          # size + sha256 + read package metadata for the Packages index.
-          mkdir -p pool/${CHANNEL}
-          PREFIX="pool/${CHANNEL}/"
-          CURSOR=""
-          while :; do
-            URL="https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/r2/buckets/${R2_BUCKET}/objects?prefix=${PREFIX}&per_page=1000"
-            [ -n "$CURSOR" ] && URL="${URL}&cursor=${CURSOR}"
-            RESP=$(curl -sS "$URL" -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}")
-            echo "$RESP" | jq -r '.result[].key' | while IFS= read -r key; do
-              [ -z "$key" ] && continue
-              mkdir -p "$(dirname "$key")"
-              echo "  pulling $key"
-              r2get "$key"
-            done
-            CURSOR=$(echo "$RESP" | jq -r '.result_info.cursor // empty')
-            [ -z "$CURSOR" ] && break
+          for arch in amd64 arm64; do
+            key="dists/${CHANNEL}/main/binary-${arch}/Packages"
+            mkdir -p "$(dirname "$key")"
+            pull_index "$key"
           done
-          echo "Local pool/${CHANNEL}/ contents after sync:"
-          # Diagnostic only. `head` closes the pipe after 20 lines; find then
-          # gets SIGPIPE and `pipefail` would fail the step after a successful
-          # R2 pull (run 369: last pull was …/wheels-be_4.0.6-snapshot.2273_all.deb).
-          find pool/${CHANNEL}/ -type f | head -20 || true
 
       - name: Download new .deb from upstream Release
         env:
@@ -220,10 +206,11 @@ jobs:
         env:
           GPG_PASSPHRASE: ${{ secrets.WHEELS_REPO_GPG_PASSPHRASE }}
           GPG_KEY_ID: ${{ env.GPG_KEY_ID }}
-          # Scope regen to the dispatched channel only. The sync step above pulls
-          # just pool/<channel>/ from R2, so regenerating the OTHER channel would
-          # scan an empty local pool and clobber its R2 index on upload
-          # (#3218 / #2838). The other channel's dists are left untouched.
+          # Scope regen to the dispatched channel only. The workflow pulls only
+          # that channel's Packages index and slots only its new .deb, so
+          # regenerating the OTHER channel would append against a missing index
+          # and clobber its R2 dists on upload (#3218 / #2838). The other
+          # channel's dists are left untouched.
           CHANNELS: ${{ steps.inputs.outputs.channel }}
         run: |
           set -euo pipefail

scripts/regenerate-apt-metadata.sh

@@ -1,15 +1,30 @@
 #!/bin/bash
-# Regenerates apt metadata for both `stable` and `bleeding-edge` distributions
-# under dists/, then signs Release with GPG (detached → Release.gpg, inline →
+# Incrementally regenerates apt metadata for one dispatched distribution under
+# dists/, then signs Release with GPG (detached → Release.gpg, inline →
 # InRelease). Both signed forms are required: older apt clients read Release +
 # Release.gpg, newer clients prefer InRelease.
 #
+# This is the incremental replacement for the old full-pool scan. The old flow
+# ran `apt-ftparchive packages pool/<dist>` over EVERY historical .deb, which
+# forced the workflow to re-download the whole pool (~18 GB, hundreds of files,
+# ~15 min and growing) on every publish. Instead:
+#
+#   * the workflow slots the single new .deb into pool/<dist>/… and downloads
+#     the channel's current Packages index into dists/<dist>/main/binary-*/,
+#   * this script appends the new package's stanza (apt-ftparchive over just
+#     that one .deb) to each architecture index, gzips, and regenerates Release
+#     via `apt-ftparchive release` — which only needs the index files, not the
+#     pool.
+#
+# A package of Architecture `all` is appended to BOTH binary-amd64 and
+# binary-arm64 (apt-ftparchive includes `all` in every arch index, verified
+# against the live repo); an arch-specific package lands only in its own index.
+#
 # Inputs (env vars):
 #   GPG_PASSPHRASE  — passphrase for the imported signing key
 #   GPG_KEY_ID      — long-form key ID (set by the workflow after `gpg --import`)
-#
-# Idempotent: safe to run by hand against an existing tree to repair a torn
-# release. Re-reads everything in pool/ and rewrites dists/ from scratch.
+#   CHANNELS        — space-separated distributions (workflow passes the single
+#                     dispatched channel; default "stable bleeding-edge")
 
 set -euo pipefail
 
@@ -20,56 +35,47 @@ fi
 
 ARCHITECTURES="amd64 arm64"
 COMPONENTS="main"
-# Per-channel regen (CHANNELS env), default = both for manual full rebuilds.
-#
-# CRITICAL (#3218, recurrence of #2838): the publish workflow only syncs
-# pool/<dispatched-channel>/ from R2, so any channel NOT being published has an
-# empty local pool here. Regenerating it would emit an empty Packages, and the
-# upload step's `find dists` would then clobber that channel's good R2 index.
-# A bleeding-edge snapshot publish was wiping the stable index minutes after
-# every stable release. Scoping to the dispatched channel keeps the other
-# channel's R2 dists untouched. The workflow passes CHANNELS=<channel>; a bare
-# manual run still rebuilds both (only safe when both pools are present locally).
 DISTRIBUTIONS="${CHANNELS:-stable bleeding-edge}"
 
-# apt-ftparchive uses a config file to know where the pool lives. The same
-# config drives both distributions — only the dist-name and the scan path
-# change between invocations.
 APT_CONF_TEMPLATE="templates/aptftparchive.conf"
-
 if [ ! -f "$APT_CONF_TEMPLATE" ]; then
   echo "::error::Missing $APT_CONF_TEMPLATE — template is expected to ship in the bucket repo."
   exit 1
 fi
 
 for DIST in $DISTRIBUTIONS; do
-  echo "── Regenerating dists/${DIST}/ ──"
+  echo "── Incrementally updating dists/${DIST}/ ──"
   DIST_DIR="dists/${DIST}"
   mkdir -p "$DIST_DIR"
-  # First publish for a brand-new channel: the pool dir may not exist yet.
-  # apt-ftparchive aborts on a missing scan path, so create an empty pool
-  # for now — it'll be backfilled by the first publish dispatch on that channel.
   mkdir -p "pool/${DIST}"
 
+  # Only the just-slotted .deb is present locally (the pool is not synced).
+  NEW_DEB=$(find "pool/${DIST}" -type f -name '*.deb' | head -1 || true)
+  if [ -z "$NEW_DEB" ]; then
+    echo "::warning::No .deb under pool/${DIST} — nothing to append; skipping."
+    continue
+  fi
+  echo "  new package: ${NEW_DEB}"
+
   for COMPONENT in $COMPONENTS; do
     for ARCH in $ARCHITECTURES; do
       BIN_DIR="${DIST_DIR}/${COMPONENT}/binary-${ARCH}"
       mkdir -p "$BIN_DIR"
 
-      # apt-ftparchive packages <override> <pool-path> emits Packages on stdout.
-      # We don't use an override file (no priority overrides for now).
-      apt-ftparchive \
-        --arch "$ARCH" \
-        packages "pool/${DIST}" \
-        > "${BIN_DIR}/Packages"
+      # Existing index was downloaded by the workflow; first publish starts empty.
+      [ -f "${BIN_DIR}/Packages" ] || : > "${BIN_DIR}/Packages"
+
+      # Emit the new package's stanza and append. `--arch` matches the old full
+      # scan: an `all` package is emitted for both arches, `amd64` only for
+      # binary-amd64 (arm64 yields an empty append, which is a no-op).
+      apt-ftparchive --arch "$ARCH" packages "$NEW_DEB" >> "${BIN_DIR}/Packages"
 
       gzip -9 --keep --force "${BIN_DIR}/Packages"
     done
   done
 
-  # apt-ftparchive release emits the Release file metadata. The config template
-  # provides Origin/Label/Codename/Description etc.; we override -o APT::FTPArchive::Release::Codename
-  # per distribution so a single conf can drive both.
+  # apt-ftparchive release emits the Release metadata from the index files in
+  # dists/<dist>/ — it does not read pool/, so it works without the pool present.
   apt-ftparchive \
     -c "$APT_CONF_TEMPLATE" \
     -o "APT::FTPArchive::Release::Codename=${DIST}" \
@@ -96,7 +102,7 @@ for DIST in $DISTRIBUTIONS; do
     --output "${DIST_DIR}/InRelease" \
     "${DIST_DIR}/Release"
 
-  echo "  ✓ Release + Release.gpg + InRelease written for ${DIST}"
+  echo "  ✓ appended ${NEW_DEB}, Release + Release.gpg + InRelease written for ${DIST}"
 done
 
 echo "Done."