Skip to content

Commit

Permalink
Merge pull request #3 from OKDP/concurrency
Browse files Browse the repository at this point in the history
Maximize number of parallel jobs and readme update
  • Loading branch information
idirze authored Feb 13, 2024
2 parents 55b1ad9 + 5823b53 commit 45b18fc
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 13 deletions.
11 changes: 1 addition & 10 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Build, test, and push jupyter images
name: "[Main] Build, test, and push jupyter images"

on:
pull_request:
Expand Down Expand Up @@ -97,8 +97,6 @@ jobs:
build-base:
name: build-test-base (python-${{ matrix.python.python_version }})
strategy:
# 3 Jobs in //, the base jobs run in sequential
max-parallel: 3
matrix:
python: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.python) }}
uses: ./.github/workflows/build-test-base.yml
Expand All @@ -115,8 +113,6 @@ jobs:
build-datascience:
name: build-test-datascience (python-${{ matrix.python.python_version }})
strategy:
# 1 matrix call = +2 jobs in // (check the number here build-datascience.yml)
max-parallel: 1
matrix:
python: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.python) }}
uses: ./.github/workflows/build-test-datascience.yml
Expand All @@ -132,8 +128,6 @@ jobs:
build-spark:
name: build-test-spark (python-${{ matrix.spark.python_version }})
strategy:
# 2 jobs in //
max-parallel: 2
matrix:
spark: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.spark) }}
uses: ./.github/workflows/build-test-spark.yml
Expand All @@ -158,7 +152,6 @@ jobs:
if: github.ref == 'refs/heads/main'
name: push-base (python-${{ matrix.python.python_version }})
strategy:
max-parallel: 1
matrix:
image: [ docker-stacks-foundation, base-notebook, minimal-notebook, scipy-notebook,]
python: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.python) }}
Expand All @@ -176,7 +169,6 @@ jobs:
if: github.ref == 'refs/heads/main'
name: push-datascience (python-${{ matrix.python.python_version }})
strategy:
max-parallel: 1
matrix:
image: [r-notebook, datascience-notebook, ]
python: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.python) }}
Expand All @@ -193,7 +185,6 @@ jobs:
push-spark:
if: github.ref == 'refs/heads/main'
strategy:
max-parallel: 1
matrix:
image: [ pyspark-notebook, all-spark-notebook,]
spark: ${{ fromJson(needs.build-version-compatibility-matrix.outputs.spark) }}
Expand Down
23 changes: 20 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The [main](.github/workflows/main.yml) build pipeline contains 6 main reusable w

The build is based on the [version compatibility matrix](.build/.versions.yml).

The [build-matrix](.build/.versions.yml#L42) section defines the components versions to build. It behaves like a filter of the parent [version compatibility matrix](.build/.versions.yml) to limit the versions combintations to build. The build process ensures only the compatible versions are built:
The [build-matrix](.build/.versions.yml#L42) section defines the components versions to build. It behaves like a filter of the parent [compatibility-matrix](.build/.versions.yml#L5) section to limit the versions combintations to build. The build process ensures only the compatible versions are built:

For example, the following build-matrix:

Expand Down Expand Up @@ -62,7 +62,7 @@ Development images with tags ```-<GIT-BRANCH>-latest``` suffix (ex.: spark3.2.4-
The [official images](#tagging) are pushed to the [container registry](https://github.com/orgs/OKDP/packages) when:

1. The workflow is triggered on the main branch only and
2. The [tests](#build/test) are completed successfully
2. The [tests](docker-stacks/tests) are completed successfully

This prevents pull requests or developement branchs to push the official images before they are reviewed or tested. It also provides the flexibility to test against developement images ```-<GIT-BRANCH>-latest``` before they are officially pushed.

Expand Down Expand Up @@ -99,7 +99,24 @@ Here are some examples:

Please, check the [container registry](https://github.com/orgs/OKDP/packages) for more images and tags.

# Build locally with Act
# Running github actions
## Github container registry credentials

Create the following [secrets and configuration variables](https://docs.github.com/en/actions/learn-github-actions/variables#creating-configuration-variables-for-a-repository) when running with your own github account or organization:

| Variable | Type | Default | Description |
| -----------------------|-------------------------| ---------| ------------------------------------------- |
| `REGISTRY` | Configuration variable | ghcr.io | Container registry |
| `REGISTRY_USERNAME` | Secret variable | | Container registry username |
| `REGISTRY_ROBOT_TOKEN` | Secret variable | | Container registry password or access token `(Scopes: write:packages/delete:packages)` |
## Running with Github

By default, the [workflow](.github/workflows/main.yml) runs automatically on the following events:

- Push on the main branch with changes on the configured `paths` filters
- Pull request on any branch

## Running locally with act

[Act](https://github.com/nektos/act) can be used to build and test locally.

Expand Down
Binary file modified doc/_images/build-pipeline.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 45b18fc

Please sign in to comment.