Merge pull request #492 from ModernRelay/codex/rustfs-ci-harness-stabilization

ci: stabilize configured RustFS integration

Author: aaltshuler

.github/workflows/ci.yml

@@ -474,15 +474,25 @@ jobs:
 
       - name: Create RustFS test bucket
         run: |
-          for _ in $(seq 1 30); do
+          set -euo pipefail
+          rustfs_ready=false
+          for attempt in $(seq 1 30); do
             if aws --endpoint-url "${AWS_ENDPOINT_URL_S3}" s3api list-buckets >/dev/null 2>&1; then
+              rustfs_ready=true
               break
             fi
+            echo "RustFS readiness check ${attempt}/30 failed; retrying"
             sleep 2
           done
+
+          if [ "$rustfs_ready" != true ]; then
+            echo "::error::RustFS did not become ready after 30 attempts"
+            exit 1
+          fi
+
           aws --endpoint-url "${AWS_ENDPOINT_URL_S3}" \
             s3api create-bucket \
-            --bucket "${OMNIGRAPH_S3_TEST_BUCKET}" >/dev/null 2>&1 || true
+            --bucket "${OMNIGRAPH_S3_TEST_BUCKET}"
 
       # Compile and execute the selected default targets in one Cargo resolve.
       # A superset prebuild followed by narrower commands would still rebuild
@@ -502,7 +512,7 @@ jobs:
             --test s3 \
             --test s3_cluster \
             --test system_local \
-            -- --nocapture 2>&1 | tee "$test_log"
+            -- --nocapture --test-threads=1 2>&1 | tee "$test_log"
 
           if grep -Eiq \
             "SKIP public_physical_ref_token_rejects_s3_same_version_aba|skipping s3 " \
@@ -553,6 +563,30 @@ jobs:
           echo "$output" | grep -Eq "test result: ok\. [1-9][0-9]* passed" \
             || { echo "::error::filter 's3_' matched no tests — vacuous pass"; exit 1; }
 
-      - name: Dump RustFS logs on failure
+      - name: Dump RustFS diagnostics on failure
         if: failure()
-        run: docker logs rustfs
+        run: |
+          set +e
+          echo "::group::docker inspect rustfs"
+          # Limit inspection to runtime state: full inspect output includes the
+          # container environment and would disclose the test credentials.
+          docker inspect --format 'State={{json .State}} Image={{.Image}}' rustfs
+          echo "::endgroup::"
+
+          echo "::group::docker logs rustfs"
+          docker logs rustfs
+          echo "::endgroup::"
+
+          diagnostics_dir=$(mktemp -d "${RUNNER_TEMP}/rustfs-logs.XXXXXX")
+          echo "::group::RustFS /logs"
+          if docker cp rustfs:/logs "$diagnostics_dir"; then
+            find "$diagnostics_dir" -type f -exec sh -c '
+              for log_file do
+                echo "===== ${log_file} ====="
+                tail -n 2000 "$log_file"
+              done
+            ' sh {} +
+          else
+            echo "Unable to copy /logs from the RustFS container"
+          fi
+          echo "::endgroup::"

docs/dev/ci.md

@@ -18,7 +18,14 @@
     fails if that exact test skips or if a broad filter matches another cell.
 - **AWS feature build job**: `cargo build/test -p omnigraph-server --features aws` on ubuntu-latest.
 - **Windows binary build job**: `cargo build --release --locked -p omnigraph-cli -p omnigraph-server` on windows-latest with smoke checks for `omnigraph.exe version`, `omnigraph-server.exe --help`, and PowerShell installer syntax.
-- **RustFS S3 integration**: starts RustFS and runs the configured bucket-gated engine/server/cluster/CLI correctness suites, including ordinary recovery failpoints. Cost/benchmark instruments remain on demand and are not promoted into correctness CI.
+- **RustFS S3 integration**: starts RustFS, requires a successful readiness
+  probe and bucket creation, and runs the configured bucket-gated
+  engine/server/cluster/CLI correctness suites, including ordinary recovery
+  failpoints. The default shard serializes only the outer libtest scenarios
+  (`--test-threads=1`); their internal Tokio work remains concurrent. Failures
+  capture `docker inspect`, container stdout/stderr, and RustFS's service logs
+  from `/logs`. Cost/benchmark instruments remain on demand and are not
+  promoted into correctness CI.
 - **release-edge.yml**: on every push to main, retags `edge`, builds Linux x86_64 / Linux arm64 / macOS arm64 archives and Windows x86_64 zip + sha256, publishes a rolling prerelease, then smoke-tests the Windows PowerShell installer against `edge`. The macOS arm64 matrix entry uses Rust's large code model because the release binary's text exceeds the architecture's +/-128 MiB direct-branch range; other platforms keep the workspace release profile unchanged.
 - **release.yml**: on `v*` tags, builds the Linux x86_64 / Linux arm64 / macOS arm64 archives and Windows x86_64 zip release matrix, updates the Homebrew tap (`scripts/update-homebrew-formula.sh`) by pushing the regenerated formula to `ModernRelay/homebrew-tap`, and smoke-tests the Windows PowerShell installer against the tag. It carries the same macOS-only large-code-model setting as the edge workflow so tagged and rolling artifacts cannot diverge at this linker boundary.
 - **package.yml**: manual ECR image build; emits two image tags per commit (`<sha>`, `<sha>-aws`) via CodeBuild.

docs/dev/testing.md

@@ -110,9 +110,14 @@ CI runs these S3-backed **correctness** tests against a containerized RustFS
 server (`.github/workflows/ci.yml` → `rustfs_integration` job) in two
 feature-graph shards. The default shard selects its five test binaries in one
 Cargo invocation so dependency features and large test links compile once,
-then checks the captured log for every required S3 cell and explicit skip. The
-failpoints shard runs the `s3_`-prefixed feature-gated cells in one
-invocation. These remain the focused local equivalents:
+then runs the outer libtest harness with `--test-threads=1` and checks the
+captured log for every required S3 cell and explicit skip. This serializes only
+the top-level scenarios sharing one RustFS container; each scenario keeps its
+ordinary internal Tokio concurrency. RustFS readiness and bucket creation are
+hard gates. A failed shard records container state, stdout/stderr, and the
+service files under `/logs`. The failpoints shard runs the `s3_`-prefixed
+feature-gated cells in one invocation. These remain the focused local
+equivalents:
 
 - `cargo test -p omnigraph-engine --test s3_storage` (lifecycle/branching, allowed external-S3 copy with 2 inputs/1 HEAD/1 GET and post-source-deletion ownership, plus the e_tag-present CSR topology cache-key reuse test — the path local FS can't reach since its e_tag is `None`)
 - `cargo test -p omnigraph-engine --test lance_surface_guards public_physical_ref_token_rejects_s3_same_version_aba -- --exact` (RFC-024's public current-HEAD witness across unchanged reopen plus main/named same-version ABA; the workflow additionally rejects a zero-test/vacuous match)