Skip to content

refactor: split pd/server/store image stage & bump to V2 version #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
imbajin merged 11 commits into from
Mar 21, 2026
Merged

refactor: split pd/server/store image stage & bump to V2 version #13

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
53ca07e
chore(workflows): add strict_mode publish gate
imbajin Mar 21, 2026
a4b2759
Update publish_latest_pd_store_server_image.yml
imbajin Mar 21, 2026
9a867a6
Use resolved source SHA and improve publish workflow
imbajin Mar 21, 2026
d6d57f2
fix(workflows): allow publish when precheck skipped
imbajin Mar 21, 2026
de4c958
Update .github/workflows/publish_latest_pd_store_server_image.yml
imbajin Mar 21, 2026
1bfb2aa
fix(workflows): align self-check with image contracts
imbajin Mar 21, 2026
bacc9e3
Merge branch 'publish-gate' of https://github.com/hugegraph/actions i…
imbajin Mar 21, 2026
10f67a6
feat(workflows): add configurable wait timeout
imbajin Mar 21, 2026
e6f312d
chore(workflows): harden compose path and trim cache writes
imbajin Mar 21, 2026
9b2347d
fix(workflows): skip fragile module self-checks
imbajin Mar 21, 2026
a48f7e2
chore(workflows): skip unused x86 self-check builds
imbajin Mar 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
275 changes: 216 additions & 59 deletions .github/workflows/publish_latest_pd_store_server_image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,31 @@ on:
required: false
default: ''
description: 'mvn build args, like "MAVEN_ARGS=-P stage"'

strict_mode:
type: boolean
required: false
default: true
description: 'whether integration precheck is mandatory before publish'

jobs:
build_latest:
integration_precheck:
if: ${{ github.event.inputs.strict_mode == 'true' }}
Comment thread
imbajin marked this conversation as resolved.
Outdated
runs-on: ubuntu-latest
env:
REPOSITORY_URL: apache/hugegraph
BRANCH: master
PD_IMAGE_URL: hugegraph/pd:latest
STORE_IMAGE_URL: hugegraph/store:latest
SERVER_IMAGE_URL: hugegraph/server:latest
MVN_ARGS: ${{inputs.mvn_args}}
MVN_ARGS: ${{ github.event.inputs.mvn_args || '' }}

steps:
# - name: Maximize Build Space
# uses: easimon/maximize-build-space@master
# with:
# root-reserve-mb: 512
# swap-size-mb: 1024
# remove-dotnet: 'true'
- name: Checkout latest
uses: actions/checkout@v4
with:
repository: ${{ env.REPOSITORY_URL }}
ref: ${{ env.BRANCH }}
fetch-depth: 2
Comment thread
coderabbitai[bot] marked this conversation as resolved.

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
Expand All @@ -36,92 +42,243 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Checkout latest
uses: actions/checkout@v4
with:
repository: ${{ env.REPOSITORY_URL }}
ref: ${{ env.BRANCH }}
fetch-depth: 2
- name: Pre-build disk usage
run: |
df -h
docker system df || true

- name: Pre-build cleanup
run: |
docker system prune -af || true
docker builder prune -af || true

- name: Build x86 PD Image
- name: Build x86 PD image for integration check
uses: docker/build-push-action@v5
with:
context: .
Comment thread
imbajin marked this conversation as resolved.
file: ./hugegraph-pd/Dockerfile
load: true
tags: ${{ env.PD_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
cache-from: type=gha,scope=latest-pd
cache-to: type=gha,scope=latest-pd,mode=min

@imbajin imbajin Mar 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ mode=min will not export the build-stage cache graph from these multi-stage Dockerfiles. The expensive mvn package work lives in the throwaway build stage, so the later publish jobs will still end up rebuilding most of it instead of reusing the precheck output. If the point of integration_precheck is to amortize build time for publish_matrix, switch these exports to mode=max on all three modules.

build-args: ${{ env.MVN_ARGS }}

- name: Build x86 Store Image
- name: Build x86 Store image for integration check
uses: docker/build-push-action@v5
with:
context: .
file: ./hugegraph-store/Dockerfile
load: true
tags: ${{ env.STORE_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
cache-from: type=gha,scope=latest-store
cache-to: type=gha,scope=latest-store,mode=min
build-args: ${{ env.MVN_ARGS }}

- name: Build x86 Server Image
- name: Build x86 Server image for integration check
uses: docker/build-push-action@v5
with:
context: .
file: ./hugegraph-server/Dockerfile-hstore
load: true
tags: ${{ env.SERVER_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
cache-from: type=gha,scope=latest-server
cache-to: type=gha,scope=latest-server,mode=min
build-args: ${{ env.MVN_ARGS }}

- name: Test x86 Images
- name: Start compose stack with local images
run: |
docker images
docker run -itd --name=pd --network host $PD_IMAGE_URL
sleep 10s
curl 0.0.0.0:8620 || exit
docker run -itd --name=store --network host $STORE_IMAGE_URL
sleep 10s
curl 0.0.0.0:8520 || exit
docker run -itd --name=server --network host $SERVER_IMAGE_URL
sleep 10s
curl 0.0.0.0:8080 || exit
docker ps -a

- name: Push x86 & ARM PD Images
uses: docker/build-push-action@v5
cat > /tmp/docker-compose.ci.override.yml <<'COMPOSE_OVERRIDE'
services:
pd:
image: hugegraph/pd:latest
pull_policy: never
store:
image: hugegraph/store:latest
pull_policy: never
server:
image: hugegraph/server:latest
Comment thread
imbajin marked this conversation as resolved.
Outdated
pull_policy: never
COMPOSE_OVERRIDE

docker compose \
-p hg-ci-precheck \
-f docker/docker-compose.yml \
Comment thread
imbajin marked this conversation as resolved.
-f /tmp/docker-compose.ci.override.yml \
up -d

docker compose -p hg-ci-precheck -f docker/docker-compose.yml -f /tmp/docker-compose.ci.override.yml ps

- name: Verify integration endpoints
run: |
wait_for_http() {
local url="$1"
local retries="${2:-40}"
Comment thread
imbajin marked this conversation as resolved.
Outdated
local sleep_secs="${3:-5}"

for _ in $(seq 1 "$retries"); do
if curl -fsS "$url" >/dev/null; then
echo "Ready: $url"
return 0
fi
sleep "$sleep_secs"
Comment thread
imbajin marked this conversation as resolved.
Outdated
done

echo "Timeout waiting for: $url"
return 1
}

wait_for_http "http://127.0.0.1:8620/v1/health"
wait_for_http "http://127.0.0.1:8520/v1/health"
wait_for_http "http://127.0.0.1:8080/versions"

- name: Dump compose logs on failure
if: ${{ failure() }}
run: |
docker compose -p hg-ci-precheck -f docker/docker-compose.yml -f /tmp/docker-compose.ci.override.yml logs --no-color --tail=200 || true

- name: Stop compose stack
if: ${{ always() }}
run: |
docker compose \
-p hg-ci-precheck \
-f docker/docker-compose.yml \
-f /tmp/docker-compose.ci.override.yml \
down -v --remove-orphans || true

- name: Post-check disk usage
if: ${{ always() }}
run: |
docker system df || true
df -h

- name: Post-check cleanup
if: ${{ always() }}
run: |
docker system prune -af || true
docker builder prune -af || true

publish_matrix:
needs: integration_precheck
if: ${{ needs.integration_precheck.result == 'success' || needs.integration_precheck.result == 'skipped' }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
Comment thread
coderabbitai[bot] marked this conversation as resolved.
matrix:
include:
- module: pd
image_url: hugegraph/pd:latest
dockerfile: ./hugegraph-pd/Dockerfile
container_port: 8620
host_port: 18620
probe_path: /v1/health
- module: store
image_url: hugegraph/store:latest
dockerfile: ./hugegraph-store/Dockerfile
container_port: 8520
host_port: 18520
probe_path: /v1/health
- module: server
image_url: hugegraph/server:latest
dockerfile: ./hugegraph-server/Dockerfile-hstore
container_port: 8080
Comment thread
imbajin marked this conversation as resolved.
Outdated
host_port: 18080
Comment thread
imbajin marked this conversation as resolved.
Outdated
probe_path: /versions
Comment thread
imbajin marked this conversation as resolved.

env:
REPOSITORY_URL: apache/hugegraph
BRANCH: master
MVN_ARGS: ${{ github.event.inputs.mvn_args || '' }}

steps:
- name: Checkout latest
uses: actions/checkout@v4
with:
context: .
file: ./hugegraph-pd/Dockerfile
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ env.PD_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
build-args: ${{ env.MVN_ARGS }}
repository: ${{ env.REPOSITORY_URL }}
ref: ${{ env.BRANCH }}
fetch-depth: 2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Pre-build disk usage (${{ matrix.module }})
run: |
df -h
docker system df || true

- name: Push x86 & ARM Store Images
- name: Pre-build cleanup (${{ matrix.module }})
run: |
docker system prune -af || true
docker builder prune -af || true

- name: Build x86 image for self-check (${{ matrix.module }})
uses: docker/build-push-action@v5
with:
context: .
file: ./hugegraph-store/Dockerfile
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ env.STORE_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
file: ${{ matrix.dockerfile }}
load: true
tags: ${{ matrix.image_url }}
cache-from: type=gha,scope=latest-${{ matrix.module }}
cache-to: type=gha,scope=latest-${{ matrix.module }},mode=min
build-args: ${{ env.MVN_ARGS }}
Comment thread
imbajin marked this conversation as resolved.

- name: Push x86 & ARM Server Images
- name: Self-check x86 image (${{ matrix.module }})
env:
IMAGE_URL: ${{ matrix.image_url }}
CONTAINER_PORT: ${{ matrix.container_port }}
HOST_PORT: ${{ matrix.host_port }}
PROBE_PATH: ${{ matrix.probe_path }}
run: |
container_name="hg-ci-${{ matrix.module }}"
docker rm -f "$container_name" >/dev/null 2>&1 || true

docker run -d \
Comment thread
imbajin marked this conversation as resolved.
Outdated
--name "$container_name" \
-p "$HOST_PORT:$CONTAINER_PORT" \
"$IMAGE_URL"

for _ in $(seq 1 30); do
Comment thread
imbajin marked this conversation as resolved.
Outdated
if curl -fsS "http://127.0.0.1:$HOST_PORT$PROBE_PATH" >/dev/null; then
echo "Self-check passed: http://127.0.0.1:$HOST_PORT$PROBE_PATH"
exit 0
fi
sleep 5
Comment thread
imbajin marked this conversation as resolved.
Outdated
done

echo "Self-check failed: http://127.0.0.1:$HOST_PORT$PROBE_PATH"
docker logs "$container_name" || true
exit 1

- name: Stop self-check container (${{ matrix.module }})
if: ${{ always() }}
run: |
docker rm -f "hg-ci-${{ matrix.module }}" >/dev/null 2>&1 || true

- name: Build and push multi-arch image (${{ matrix.module }})
uses: docker/build-push-action@v5
with:
context: .
file: ./hugegraph-server/Dockerfile-hstore
file: ${{ matrix.dockerfile }}
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ env.SERVER_IMAGE_URL }}
# cache-from: type=gha
# cache-to: type=gha,mode=max
tags: ${{ matrix.image_url }}
cache-from: type=gha,scope=latest-${{ matrix.module }}
cache-to: type=gha,scope=latest-${{ matrix.module }},mode=min
build-args: ${{ env.MVN_ARGS }}

- name: Post-build disk usage (${{ matrix.module }})
if: ${{ always() }}
run: |
docker system df || true
df -h

- name: Post-build cleanup (${{ matrix.module }})
if: ${{ always() }}
run: |
docker system prune -af || true
docker builder prune -af || true
Loading