Skip to content

refactor(workflows): validate multiarch images once#23

Merged
imbajin merged 9 commits into
masterfrom
cx-pd-store-server-e2e
Jul 22, 2026
Merged

refactor(workflows): validate multiarch images once#23
imbajin merged 9 commits into
masterfrom
cx-pd-store-server-e2e

Conversation

@imbajin

@imbajin imbajin commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

address part of #7

  • build and load PD, Store, HStore Server, and standalone Server as amd64/arm64 candidates in one job
  • validate both platform variants, then run a low-memory 1 PD + 1 Store + 1 Server topology against the exact local final tags
  • import only the Server image's bundled scripts/example.groovy, assert 6V/6E, execute Gremlin CRUD, and restore 6V/6E
  • publish the already validated multi-platform indexes directly, without temporary architecture tags or manifest/cleanup jobs
  • decouple release source branch from explicit version_tag, so master can publish 1.7.0
  • retry transient QEMU build failures per module and transient final-tag pushes; dry-runs cannot update the latest hash

A/B result

All comparisons used apache/hugegraph@master, strict integration checks, read-only registry caches, and dry_run=true.

Design Active wall time Approx. runner time Active jobs Result
Previous 4-way ARM parallel baseline 6m09s 13m41s 6 fastest wall time, highest runner use / setup duplication
A: combined ARM + combined manifest, staged tags 10m44s 10m34s 3 passed; ARM job 7m48s
B: one multi-platform build/test job 10m30s 10m26s 2 passed
B repeat 11m06s ~11m06s 2 passed
B final hardened head 11m50s ~11m50s 2 passed

B is retained. Its observed runtime is in the same range as A (normal QEMU/network variance is larger than the difference), while it removes the separate ARM/manifest orchestration, eight temporary tags, and Docker Hub delete API. It also tests the final local tags directly.

A later run exposed the actual remaining ARM risk: QEMU segfaulted in the ARM libc-bin post-install script. The final workflow retries only that failed module with completed BuildKit layers retained; permanent failures still stop after three attempts. The final successful run needed no retry.

Verification

  • actionlint for reusable/latest/release workflows
  • git diff --check
  • independent final diff review: ready, no Critical or Important findings
  • A dry-run: 29897704499
  • initial B dry-run: 29898202758
  • captured QEMU failure: 29899521050
  • final head dry-run: 29900458519 — success
    • source SHA: 89b648a7f9a7f6c8d202bb199d3c2442d8d511f9
    • all four final tags contain locally inspectable amd64 and arm64 variants
    • enforced PD/Store/Server limits: 1 GiB / 1 GiB / 1.5 GiB
    • imported image-bundled scripts/example.groovy: 6V/6E
    • CRUD reached 7V/7E and returned to 6V/6E
    • standalone smoke passed; compose containers, networks, and volumes were removed
    • image/cache push and hash update were skipped

Summary by CodeRabbit

  • 新功能

    • 新增手动发布时的版本标签设置。
    • 新增 dry-run 模式,可完成多架构构建与完整验证,但不推送镜像或更新版本标记。
    • 发布前增加集成预检、健康检查、示例导入及 Gremlin 增删改查验证。
    • 增强独立服务的启动与接口烟测,并在失败时保留诊断日志。
  • 改进

    • 优化 amd64 与 arm64 镜像发布流程,确保仅发布通过验证的镜像。
    • 默认发布分支调整为 master,并完善发布流程文档。

- run low-memory HStore compose before image publication
- import the bundled example graph and verify Gremlin CRUD
- enforce runtime resource limits and strict cleanup checks
- document CI scope and the future 3x3 topology TODO
Copilot AI review requested due to automatic review settings July 21, 2026 08:14
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. enhancement New feature or request labels Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

新增 dry_runversion_tag 发布参数,扩展 AMD64 候选镜像集成预检和 standalone smoke test,并调整 ARM64、manifest 清理及 README 流程说明。

Changes

镜像发布流水线

Layer / File(s) Summary
输入与发布门控
.github/workflows/_publish_pd_store_server_reusable.yml, .github/workflows/publish_latest_pd_store_server_image.yml, .github/workflows/publish_release_pd_store_server_image.yml
新增 dry_runversion_tag,限制 release dry-run,并在 master latest dry-run 中禁用镜像发布与 hash gate。
AMD64 候选构建与集成预检
.github/workflows/_publish_pd_store_server_reusable.yml
候选镜像使用统一标签,通过本地 compose、资源/健康检查、图导入、Gremlin CRUD 和 standalone smoke test 后按条件推送。
ARM64 构建、manifest 与流程说明
.github/workflows/_publish_pd_store_server_reusable.yml, README.md
ARM64 构建依赖 AMD64 成功结果;manifest 创建后批量删除临时架构标签,README 同步更新关键路径和执行约束。

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant build_test_publish_amd64
  participant local_compose
  participant publish_arm64
  participant publish_manifest
  Workflow->>build_test_publish_amd64: 构建并验证 AMD64 候选镜像
  build_test_publish_amd64->>local_compose: 执行集成检查与 smoke test
  build_test_publish_amd64->>publish_arm64: 成功后允许 ARM64 构建
  publish_arm64->>publish_manifest: 提供 ARM64 镜像
  publish_manifest->>publish_manifest: 创建并检查多架构 manifest
  publish_manifest->>publish_manifest: 删除 AMD64/ARM64 临时标签
Loading

Possibly related PRs

Poem

小兔挥爪跑流水,
dry-run 下只验证。
AMD64 先过 Gremlin 门,
ARM64 随后踏云行。
manifest 合拢月光白。

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 标题与本次工作流重构和一次构建后验证多架构镜像的核心变化一致,且足够简洁具体。
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch cx-pd-store-server-e2e

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the README.md documentation to detail the strict integration precheck for PD, Store, and HStore Server images. It explains the low-memory test profile constraints, the functional graph check using the bundled example.groovy file, and the Gremlin CRUD validation. It also documents that the precheck uses a 1 PD + 1 Store + 1 Server topology to fit standard GitHub-hosted runners. There are no review comments, so we have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a stricter pd/store/server integration precheck to the reusable publish workflow, aimed at validating a low-resource Docker Compose topology and exercising a bundled example graph + Gremlin CRUD before any publish steps run.

Changes:

  • Adds compose-file auto-detection (docker-compose.dev.yml with legacy fallback) plus CI override constraints (CPU/memory/JVM tuning) for PD/Store/Server.
  • Imports the Server image’s bundled example.groovy graph and validates Gremlin read/create/update/delete, ensuring the graph returns to the baseline.
  • Improves failure diagnostics and enforces compose resource cleanup, and documents the new precheck scope in README.md.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
README.md Documents the new low-memory integration precheck, example graph import, and Gremlin CRUD validation scope.
.github/workflows/_publish_pd_store_server_reusable.yml Implements the compose-based precheck (resource limits, example import, CRUD checks) plus enhanced diagnostics and cleanup enforcement.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +539 to +550
docker compose -p hg-ci-precheck -f "$COMPOSE_FILE" -f /tmp/docker-compose.ci.override.yml ps || true
docker stats --no-stream hg-pd hg-store hg-server || true
for container in hg-pd hg-store hg-server; do
docker inspect --format '{{json .State.Health}}' "$container" || true
done
for artifact in /tmp/hg-ci-example-import.log /tmp/hg-ci-gremlin-*.json; do
if [ -f "$artifact" ]; then
echo "===== ${artifact} ====="
cat "$artifact"
fi
done
docker compose -p hg-ci-precheck -f "$COMPOSE_FILE" -f /tmp/docker-compose.ci.override.yml logs --no-color --tail=200 || true
Comment on lines 555 to 560
set -euo pipefail

docker compose \
-p hg-ci-precheck \
-f docker/docker-compose.yml \
-f "$COMPOSE_FILE" \
-f /tmp/docker-compose.ci.override.yml \
imbajin added 2 commits July 21, 2026 16:37
- avoid unbound COMPOSE_FILE in failure paths
- skip compose commands when setup never completed
- preserve strict leftover resource checks
- build and validate amd64 candidates once before pushing
- move arm64 builds to native GitHub runners without QEMU
- add a non-publishing exact-master dry-run path
- document cache isolation and Dockerfile follow-up
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jul 22, 2026
- use cached x86 build stages when Dockerfiles pin BUILDPLATFORM
- fall back to native ARM for legacy release Dockerfiles
- isolate native ARM caches and keep dry-runs read-only
- document measured QEMU and native runner tradeoffs
@imbajin imbajin changed the title feat(workflows): add graph publish precheck refactor(workflows): promote tested graph images Jul 22, 2026
- decouple release source branch from image version tag
- combine arm64 builds into one reusable runner job
- combine manifest creation and temporary tag cleanup
- keep amd64 integration validation ahead of publishing

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/workflows/_publish_pd_store_server_reusable.yml (1)

675-680: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

建议显式设置 persist-credentials: false

zizmor 提示该 actions/checkout 步骤未设置 persist-credentials: false,存在凭证持久化风险(artipacked)。该步骤 checkout 的是外部 REPOSITORY_URL/SOURCE_SHA,后续构建步骤并不需要 git 认证能力,建议显式关闭凭证持久化以降低意外泄露风险。

🔒 建议修复
     - name: Checkout source
       uses: actions/checkout@v7
       with:
         repository: ${{ env.REPOSITORY_URL }}
         ref: ${{ env.SOURCE_SHA }}
         fetch-depth: 2
+        persist-credentials: false
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/_publish_pd_store_server_reusable.yml around lines 675 -
680, Update the “Checkout source” step using actions/checkout@v7 to explicitly
set persist-credentials to false, while preserving the existing repository, ref,
and fetch-depth settings.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/_publish_pd_store_server_reusable.yml:
- Around line 785-820: Update the input validation in delete_tag_with_retry to
detect a missing slash from the original image_repo format rather than comparing
namespace and repository values. Preserve valid repositories where namespace and
repository are identical, including hugegraph/hugegraph, so the function
proceeds with deletion attempts.

---

Nitpick comments:
In @.github/workflows/_publish_pd_store_server_reusable.yml:
- Around line 675-680: Update the “Checkout source” step using
actions/checkout@v7 to explicitly set persist-credentials to false, while
preserving the existing repository, ref, and fetch-depth settings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: be1af3b3-6fe0-4645-a087-b80b903ab0e3

📥 Commits

Reviewing files that changed from the base of the PR and between 09c0e58 and ab66b10.

📒 Files selected for processing (3)
  • .github/workflows/_publish_pd_store_server_reusable.yml
  • .github/workflows/publish_release_pd_store_server_image.yml
  • README.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • README.md

Comment thread .github/workflows/_publish_pd_store_server_reusable.yml Outdated
imbajin added 4 commits July 22, 2026 14:52
- load amd64 and arm64 candidates into the containerd image store
- run integration checks against final local image tags
- push validated multi-platform indexes without temporary tags
- document the simplified publication and release inputs
- disable persisted credentials for external source checkout
- keep source builds read-only after checkout
- address the remaining applicable PR security comment
- prevent dry runs from updating the latest source hash
- retry final multi-platform image pushes on transient failures
- require local standalone images and remove dead matrix output
- align release documentation with branch-only source resolution
- consolidate multi-platform builds behind one helper
- retry transient QEMU runtime-stage failures per module
- reuse completed BuildKit layers between attempts
- preserve local multi-platform loading and cache policy
@imbajin imbajin changed the title refactor(workflows): promote tested graph images refactor(workflows): validate multiarch images once Jul 22, 2026
@imbajin
imbajin merged commit 4c57abb into master Jul 22, 2026
4 checks passed
@imbajin
imbajin deleted the cx-pd-store-server-e2e branch July 22, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants