-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Build/Test - Add new test with multiple filters and rename build work…
…flows
- Loading branch information
Showing
9 changed files
with
207 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,131 @@ | ||
| # kdp-docker-stacks | ||
| # OKDP Jupyter Images | ||
|
|
||
| kdp jupyter images based on https://github.com/jupyter/docker-stacks | ||
| [](https://github.com/OKDP/jupyterlab-docker/actions/workflows/main.yml) | ||
|
|
||
| OKDP jupyter docker images based on [jupyter docker-stacks](https://github.com/jupyter/docker-stacks) source dockerfiles. It includes (read only copy) [jupyter docker-stacks](https://github.com/jupyter/docker-stacks) repository as a [git-subtree](https://www.atlassian.com/git/tutorials/git-subtree) sub project. | ||
|
|
||
| # Extension | ||
| The project leverages the features provided by [jupyter docker-stacks](https://github.com/jupyter/docker-stacks): | ||
| - Build from the original [source docker files](docker-stacks/images) | ||
| - Customize the images by using docker ```build-arg``` build arguments | ||
| - Run the original [tests](docker-stacks/tests) at every pipeline trigger | ||
|
|
||
| The project provides an up to date jupyter lab images especially for pyspark. | ||
|
|
||
| # Initial setup | ||
| # Images build workflow | ||
| ## Build/Test | ||
|
|
||
| The [main](.github/workflows/main.yml) build pipeline contains 6 main reusable workflows: | ||
|
|
||
| 1. [build-test-base](.github/workflows/build-test-base.yml): docker-stacks-foundation, base-notebook, minimal-notebook, scipy-notebook | ||
| 2. [build-test-datascience](.github/workflows/build-test-datascience.yml): r-notebook, julia-notebook, tensorflow-notebook, pytorch-notebook | ||
| 3. [build-test-spark](.github/workflows/build-test-spark.yml): pyspark-notebook, all-spark-notebook | ||
| 4. [tag-push](.github/workflows/docker-tag-push.yml): push the built images to the container registry (main branch only) | ||
| 5. [auto-rerun](.github/workflows/auto-rerun.yml): partially re-run jobs in case of failures (github runner issues/main branch only) | ||
| 6. [unit-tests](.github/workflows/unit-tests.yml): run the unit tests (okdp extension) at every pipeline trigger | ||
|
|
||
|  | ||
|
|
||
| The build is based on the [version compatibility matrix](.build/.versions.yml). | ||
|
|
||
| The [build-matrix](.build/.versions.yml#L42) section defines the components versions to build. It behaves like a filter of the parent [version compatibility matrix](.build/.versions.yml) to limit the versions combintations to build. The build process ensures only the compatible versions are built: | ||
|
|
||
| For example, the following build-matrix: | ||
|
|
||
| ```yaml | ||
| build-matrix: | ||
| python_version: ['3.9', '3.10', '3.11'] | ||
| spark_version: [3.2.4, 3.3.4, 3.4.2, 3.5.0] | ||
| java_version: [11, 17] | ||
| scala_version: [2.12] | ||
| ``` | ||
| Will build the following versions combinations in regards to [compatibility-matrix](.build/.versions.yml#5) section: | ||
| - spark3.3.4-python3.10-java17-scala2.12 | ||
| - spark3.5.0-python3.11-java17-scala2.12 | ||
| - spark3.4.2-python3.11-java17-scala2.12 | ||
| - spark3.2.4-python3.9-java11-scala2.12 | ||
| By default, if no filter is specified: | ||
| ```yaml | ||
| build-matrix: | ||
| ``` | ||
| All compatible versions combinations are built. | ||
| Finally, all the images are tested against the original [tests](docker-stacks/tests) at every pipeline trigger | ||
| ## Push | ||
| Development images with tags ```-<GIT-BRANCH>-latest``` suffix (ex.: spark3.2.4-python3.9-java11-scala2.12-<GIT-BRANCH>-latest) are produced at every pipeline run regardless of the git branch (main or not). | ||
|
|
||
| The [official images](#tagging) are pushed to the [container registry](https://github.com/orgs/OKDP/packages) when: | ||
|
|
||
| 1. The workflow is triggered on the main branch only and | ||
| 2. The [tests](#build/test) are completed successfully | ||
|
|
||
| This prevents pull requests or developement branchs to push the official images before they are reviewed or tested. It also provides the flexibility to test against developement images ```-<GIT-BRANCH>-latest``` before they are officially pushed. | ||
|
|
||
| ## Tagging | ||
|
|
||
| The project builds the images with a long format tags. Each tag combines multiple compatible versions combinations. | ||
|
|
||
| There are multiple tags levels and the format to use is depending on your convenience in term of stability and reproducibility. | ||
|
|
||
| Here are some examples: | ||
|
|
||
| ### scipy-notebook: | ||
| - python-3.11-2024-02-06 | ||
| - python-3.11.7-2024-02-06 | ||
| - python-3.11.7-hub-4.0.2-lab-4.1.0 | ||
| - python-3.11.7-hub-4.0.2-lab-4.1.0-2024-02-06 | ||
|
|
||
| ### datascience-notebook: | ||
| - python-3.9-2024-02-06 | ||
| - python-3.9.18-2024-02-06 | ||
| - python-3.9.18-hub-4.0.2-lab-4.1.0 | ||
| - python-3.9.18-hub-4.0.2-lab-4.1.0-2024-02-06 | ||
| - python-3.9.18-r-4.3.2-julia-1.10.0-2024-02-06 | ||
| - python-3.9.18-r-4.3.2-julia-1.10.0-hub-4.0.2-lab-4.1.0 | ||
| - python-3.9.18-r-4.3.2-julia-1.10.0-hub-4.0.2-lab-4.1.0-2024-02-06 | ||
|
|
||
| ### pyspark-notebook: | ||
| - spark-3.5.0-python-3.11-java-17-scala-2.12 | ||
| - spark-3.5.0-python-3.11-java-17-scala-2.12-2024-02-06 | ||
| - spark-3.5.0-python-3.11.7-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0 | ||
| - spark-3.5.0-python-3.11.7-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0-2024-02-06 | ||
| - spark-3.5.0-python-3.11.7-r-4.3.2-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0 | ||
| - spark-3.5.0-python-3.11.7-r-4.3.2-java-17.0.9-scala-2.12.18-hub-4.0.2-lab-4.1.0-2024-02-06 | ||
|
|
||
| Please, check the [container registry](https://github.com/orgs/OKDP/packages) for more images and tags. | ||
|
|
||
| # Build locally with Act | ||
|
|
||
| [Act](https://github.com/nektos/act) can be used to build and test locally. | ||
|
|
||
| Here is an example command: | ||
|
|
||
| ```shell | ||
| git remote add docker-stacks https://github.com/jupyter/docker-stacks.git | ||
| git subtree add --prefix=docker-stacks --squash docker-stacks main | ||
| $ act --container-architecture linux/amd64 \ | ||
| -W .github/workflows/main.yml \ | ||
| --env ACT_SKIP_TESTS=<true|false> \ | ||
| --var REGISTRY=ghcr.io \ | ||
| --secret REGISTRY_USERNAME=<GITHUB_OWNER> \ | ||
| --secret REGISTRY_ROBOT_TOKEN=<GITHUB_CONTAINER_REGISTRY_TOKEN> | ||
| --rm | ||
| ``` | ||
|
|
||
| set the option ```--container-architecture linux/amd64``` if you are running locally with Apple's M1/M2 chips. | ||
|
|
||
| For more information: | ||
|
|
||
| ```shell | ||
| act --container-architecture linux/amd64 \ | ||
| -W .github/workflows/docker.yml \ | ||
| --artifact-server-path /tmp/act/artifacts \ | ||
| --env ACT_SKIP_TESTS=<true|false> \ | ||
| --env PUSH_TO_REGISTRY=true \ | ||
| --env REGISTRY=ghcr.io \ | ||
| --secret REGISTRY_USERNAME=<GITHUB_OWNER> \ | ||
| --secret REGISTRY_ROBOT_TOKEN=<GITHUB_CONTAINER_REGISTRY_TOKEN> | ||
| --rm | ||
| ``` | ||
| $ act --help | ||
| ``` | ||
|
|
||
| # OKDP custom extensions | ||
|
|
||
| 1. [Tagging extension](python/okdp/extension/tagging) is based on the original [jupyter docker-stacks](docker-stacks/tagging) source files | ||
| 2. [Patchs](python/okdp/patch/README.md) patchs the original [jupyter docker-stacks](docker-stacks/tests) in order to run the tests | ||
| 3. [Version compatibility matrix](python/okdp/extension/matrix) to generate all the compatible versions combintations for pyspark | ||
| 4. [Unit tests](python/tests) in order to test okdp extension at every pipeline run |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters