Add extension and config to enable templating in versions (#143)

jacobtomlinson · web-flow · commit fae81dfd8e42 · 2023-02-13T12:09:24.000Z
* Add extension and config to enable tamplting in versions

* Template in RAPIDS version

* Add example
diff --git a/.github/workflows/build-and-deploy.yml b/.github/workflows/build-and-deploy.yml
@@ -30,6 +30,8 @@ jobs:
           environment-file: conda/environments/deployment_docs.yml
 
       - name: Build
+        env:
+          DEPLOYMENT_DOCS_BUILD_STABLE: ${{ startsWith(github.event.ref, 'refs/tags/') && 'true' || 'false' }}
         run: make html
 
       # If on main in the upstream repo then push to docs repo
diff --git a/README.md b/README.md
@@ -102,6 +102,33 @@ This repository is continuously deployed to the [nightly docs at docs.rapids.ai]
 
 We can also update the [stable documentation at docs.rapids.ai](https://docs.rapids.ai/deployment/stable/) by creating and pushing a tag which will cause the `build-and-deploy` workflow to push to the [`deployment/stable` subdirectory](https://github.com/rapidsai/docs/tree/gh-pages/deployment) instead.
 
+The RAPIDS versions for things like container images and install instructions are templated into the documentation pages and are stored in `source/conf.py`.
+
+```python
+versions = {
+    "stable": {
+        "rapids_container": "rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9",
+    },
+    "nightly": {
+        "rapids_container": "rapidsai/rapidsai-core-nightly:23.02-cuda11.5-runtime-ubuntu20.04-py3.9",
+    },
+}
+```
+
+You can then use the value in any documentation page or notebook like this.
+
+```markdown
+# My doc page
+
+The latest container image is {{ rapids_container }}.
+```
+
+All builds will use the nightly section by default which allows you to test with the latest and greatest containers when developing locally or previewing nightly docs builds. To build the docs using the stable images you need to set the environment variable `DEPLOYMENT_DOCS_BUILD_STABLE` to `true`. This is done automatically when building from a tag in CI.
+
+Before you publish a new version for a release ensure that the latest container images are available and then update the `stable` config to use the new release version and update `nightly` to use the next upcoming nightly.
+
+Then you can push a tag to release.
+
 ```bash
 # Set next version number
 # See https://docs.rapids.ai/resources/versions/ and past releases for version scheme
diff --git a/extensions/rapids_version_templating.py b/extensions/rapids_version_templating.py
@@ -0,0 +1,31 @@
+def version_template(app, docname, source):
+    """Substitute versions into each page.
+
+    This allows documentation pages and notebooks to substiture in values like
+    the latest container image using jinja2 syntax.
+
+    E.g
+
+        # My doc page
+
+        The latest container image is {{ rapids_container }}.
+
+    """
+
+    # Make sure we're outputting HTML
+    if app.builder.format != "html":
+        return
+    src = source[0]
+    rendered = app.builder.templates.render_string(src, app.config.rapids_version)
+    source[0] = rendered
+
+
+def setup(app):
+    app.add_config_value("rapids_version", {}, "html")
+    app.connect("source-read", version_template)
+
+    return {
+        "version": "0.1",
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }
diff --git a/source/_includes/install-rapids-with-docker.md b/source/_includes/install-rapids-with-docker.md
@@ -6,9 +6,9 @@ On the release selector choose **Docker** in the **Method** column.
 Then copy the commands shown:
 
 ```bash
-docker pull rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+docker pull {{ rapids_container }}
 docker run --gpus all --rm -it \
     --shm-size=1g --ulimit memlock=-1 \
     -p 8888:8888 -p 8787:8787 -p 8786:8786 \
-    rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+    {{ rapids_container }}
 ```
diff --git a/source/cloud/aws/ec2-multi.md b/source/cloud/aws/ec2-multi.md
@@ -35,7 +35,7 @@ from dask_cloudprovider.aws import EC2Cluster
 
 cluster = EC2Cluster(
     instance_type="g4dn.12xlarge",  # 4 T4 GPUs
-    docker_image="rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9",
+    docker_image="{{ rapids_container }}",
     worker_class="dask_cuda.CUDAWorker",
     worker_options={"rmm-managed-memory": True},
     security_groups=["<SECURITY GROUP ID>"],
diff --git a/source/cloud/aws/sagemaker.md b/source/cloud/aws/sagemaker.md
@@ -31,7 +31,7 @@ set -e
 
 sudo -u ec2-user -i <<'EOF'
 
-mamba create -y -n rapids -c rapidsai -c conda-forge -c nvidia rapids=22.12 python=3.9 cudatoolkit=11.5 ipykernel
+mamba create -y -n rapids {{ rapids_conda_channels }} {{ rapids_conda_packages }} ipykernel
 conda activate rapids
 
 # optionally install AutoGluon for AutoML GPU demo
diff --git a/source/cloud/azure/azure-vm-multi.md b/source/cloud/azure/azure-vm-multi.md
@@ -38,7 +38,7 @@ cluster = AzureVMCluster(
     disk_size=100,
     n_workers=2,
     worker_class="dask_cuda.CUDAWorker",
-    docker_image="rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9",
+    docker_image={{rapids_container}},
     docker_args="-e DISABLE_JUPYTER=true -p 8787:8787 -p 8786:8786",
 )
 ```
diff --git a/source/cloud/gcp/vertex-ai.md b/source/cloud/gcp/vertex-ai.md
@@ -9,7 +9,7 @@ For new, user-managed notebooks, it is recommended to use a RAPIDS docker image
 Before configuring a new notebook, the [RAPIDS Docker image](#rapids-docker) will need to be built to expose port 8080 to be used as a notebook service.
 
 ```dockerfile
-FROM rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+FROM {{ rapids_container }}
 EXPOSE 8080
 
 ENTRYPOINT ["jupyter-lab", "--allow-root", "--ip=0.0.0.0", "--port=8080", "--no-browser", "--NotebookApp.token=''", "--NotebookApp.allow_origin='*'"]
@@ -18,8 +18,8 @@ ENTRYPOINT ["jupyter-lab", "--allow-root", "--ip=0.0.0.0", "--port=8080", "--no-
 Once you have built this image, it needs to be pushed to [Google Container Registry](https://cloud.google.com/container-registry/docs/pushing-and-pulling) for Vertex AI to access.
 
 ```console
-$ docker build -t gcr.io/<project>/<folder>/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9 .
-$ docker push gcr.io/<project>/<folder>/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+$ docker build -t gcr.io/<project>/<folder>/{{ rapids_container.replace('rapidsai/', '') }} .
+$ docker push gcr.io/<project>/<folder>/{{ rapids_container.replace('rapidsai/', '') }}
 ```
 
 ## Create a New Notebook
diff --git a/source/conf.py b/source/conf.py
@@ -20,6 +20,23 @@
 copyright = f"{datetime.date.today().year}, NVIDIA"
 author = "NVIDIA"
 
+versions = {
+    "stable": {
+        "rapids_container": "rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9",
+        "rapids_conda_channels": "-c rapidsai -c conda-forge -c nvidia",
+        "rapids_conda_packages": "rapids=22.12 python=3.9 cudatoolkit=11.5",
+    },
+    "nightly": {
+        "rapids_container": "rapidsai/rapidsai-core-nightly:23.02-cuda11.5-runtime-ubuntu20.04-py3.9",
+        "rapids_conda_channels": "-c rapidsai-nightly -c conda-forge -c nvidia",
+        "rapids_conda_packages": "rapids=23.02 python=3.9 cudatoolkit=11.5",
+    },
+}
+rapids_version = (
+    versions["stable"]
+    if os.environ.get("DEPLOYMENT_DOCS_BUILD_STABLE", "false") == "true"
+    else versions["nightly"]
+)
 
 # -- General configuration ---------------------------------------------------
 
@@ -36,6 +53,7 @@
     "rapids_notebook_files",
     "rapids_related_examples",
     "rapids_grid_toctree",
+    "rapids_version_templating",
 ]
 
 myst_enable_extensions = ["colon_fence"]
diff --git a/source/examples/rapids-sagemaker-higgs/Dockerfile b/source/examples/rapids-sagemaker-higgs/Dockerfile
@@ -1,5 +1,6 @@
-# FROM rapidsai/rapidsai-cloud-ml:latest
-FROM rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+ARG RAPIDS_IMAGE
+
+FROM $RAPIDS_IMAGE as rapids
 
 RUN apt-get update && apt-get install -y --no-install-recommends build-essential
 
diff --git a/source/examples/rapids-sagemaker-higgs/notebook.ipynb b/source/examples/rapids-sagemaker-higgs/notebook.ipynb
@@ -125,9 +125,9 @@
             "outputs": [],
             "source": [
                 "estimator_info = {\n",
-                "    \"rapids_container\": \"rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9\",\n",
-                "    \"ecr_image\": \"sagemaker-rapids-nightly\",\n",
-                "    \"ecr_repository\": \"sagemaker-rapids-nightly\",\n",
+                "    \"rapids_container\": \"{{ rapids_container }}\",\n",
+                "    \"ecr_image\": \"sagemaker-rapids\",\n",
+                "    \"ecr_repository\": \"sagemaker-rapids\",\n",
                 "}"
             ]
         },
@@ -156,17 +156,7 @@
             "metadata": {},
             "outputs": [],
             "source": [
-                "# FROM rapidsai/rapidsai-cloud-ml:latest\n",
-                "#!docker build -t sagemaker-rapids:latest docker"
-            ]
-        },
-        {
-            "cell_type": "code",
-            "execution_count": null,
-            "metadata": {},
-            "outputs": [],
-            "source": [
-                "!docker build -t sagemaker-rapids-nightly docker"
+                "!docker build -t sagemaker-rapids-nightly --build-arg RAPIDS_IMAGE={{ rapids_container }} ."
             ]
         },
         {
@@ -436,7 +426,7 @@
             "name": "python",
             "nbconvert_exporter": "python",
             "pygments_lexer": "ipython3",
-            "version": "3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37) \n[Clang 14.0.6 ]"
+            "version": "3.9.15"
         },
         "vscode": {
             "interpreter": {
diff --git a/source/examples/rapids-sagemaker-hpo/Dockerfile b/source/examples/rapids-sagemaker-hpo/Dockerfile
@@ -1,4 +1,6 @@
-FROM rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu18.04-py3.9 
+ARG RAPIDS_IMAGE
+
+FROM $RAPIDS_IMAGE as rapids
 
 ENV AWS_DATASET_DIRECTORY="10_year"
 ENV AWS_ALGORITHM_CHOICE="XGBoost"
@@ -7,17 +9,17 @@ ENV AWS_CV_FOLDS="10"
 
 # ensure printed output/log-messages retain correct order
 ENV PYTHONUNBUFFERED=True
-    
+
 # add sagemaker-training-toolkit [ requires build tools ], flask [ serving ], and dask-ml
-RUN apt-get update && apt-get install -y --no-install-recommends build-essential \ 
+RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
     && source activate rapids \
     && pip3 install sagemaker-training cupy-cuda115 flask dask-ml \
     && pip3 install --upgrade protobuf
 
 # path where SageMaker looks for code when container runs in the cloud
 ENV CLOUD_PATH="/opt/ml/code"
 
-# copy our latest [local] code into the container 
+# copy our latest [local] code into the container
 COPY . $CLOUD_PATH
 
 # make the entrypoint script executable
diff --git a/source/examples/rapids-sagemaker-hpo/notebook.ipynb b/source/examples/rapids-sagemaker-hpo/notebook.ipynb
@@ -713,9 +713,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "rapids_base_container = (\n",
-    "    \"rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu18.04-py3.9\"\n",
-    ")"
+    "rapids_base_container = \"{{ rapids_container }}\""
    ]
   },
   {
@@ -1347,7 +1345,7 @@
    ],
    "source": [
     "%%time\n",
-    "!docker build . -t $ecr_fullname -f Dockerfile"
+    "!docker build -t $ecr_fullname --build-arg RAPIDS_IMAGE={{ rapids_container }} ."
    ]
   },
   {
diff --git a/source/examples/xgboost-gpu-hpo-job-parallel-k8s/notebook.ipynb b/source/examples/xgboost-gpu-hpo-job-parallel-k8s/notebook.ipynb
@@ -90,7 +90,7 @@
    "outputs": [],
    "source": [
     "# Choose the same RAPIDS image you used for launching the notebook session\n",
-    "rapids_image = \"rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9\"\n",
+    "rapids_image = \"{{ rapids_container }}\"\n",
     "# Use the number of worker nodes in your Kubernetes cluster.\n",
     "n_workers = 4"
    ]
diff --git a/source/guides/azure/infiniband.md b/source/guides/azure/infiniband.md
@@ -252,7 +252,7 @@ Accept the default and allow conda init to run. Then start a new shell.
 Create a conda environment (see [UCX-Py](https://ucx-py.readthedocs.io/en/latest/install.html) docs)
 
 ```shell
-mamba create -n ucxpy -c rapidsai -c conda-forge -c nvidia rapids=22.10 python=3.9 cudatoolkit=11.5 ipython ucx-proc=*=gpu ucx ucx-py dask distributed numpy cupy pytest pynvml -y
+mamba create -n ucxpy {{ rapids_conda_channels }} {{ rapids_conda_packages }} ipython ucx-proc=*=gpu ucx ucx-py dask distributed numpy cupy pytest pynvml -y
 mamba activate ucxpy
 ```
 
@@ -299,7 +299,7 @@ UCX_TLS=tcp,cuda_copy,rc
 
 ## Run Benchmarks
 
-Finally, let's run the [merge benchmark](https://github.com/rapidsai/dask-cuda/blob/branch-22.12/dask_cuda/benchmarks/local_cudf_merge.py) from `dask-cuda`.
+Finally, let's run the [merge benchmark](https://github.com/rapidsai/dask-cuda/blob/HEAD/dask_cuda/benchmarks/local_cudf_merge.py) from `dask-cuda`.
 
 This benchmark uses Dask to perform a merge of two dataframes that are distributed across all the available GPUs on your
 VM. Merges are a challenging benchmark in a distributed setting since they require communication-intensive shuffle
diff --git a/source/platforms/databricks.md b/source/platforms/databricks.md
@@ -26,7 +26,7 @@ RUN conda create --name rapids --file /tmp/spec.txt && \
 ```
 
 ```console
-$ docker build --tag <username>/rapids_databricks:latest --build-arg RAPIDS_IMAGE=rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu18.04-py3.9 ./docker
+$ docker build --tag <username>/rapids_databricks:latest --build-arg RAPIDS_IMAGE={{ rapids_container }} ./docker
 ```
 
 Push this image to a Docker registry (DockerHub, Amazon ECR or Azure ACR).
diff --git a/source/platforms/kubeflow.md b/source/platforms/kubeflow.md
@@ -10,7 +10,7 @@ These instructions were tested against [Kubeflow v1.5.1](https://github.com/kube
 
 The [RAPIDS docker images](/tools/rapids-docker) can be used directly in Kubeflow Notebooks with no additional configuration. To find the latest image head to [the RAPIDS install page](https://docs.rapids.ai/install), as shown in below, and choose a version of RAPIDS to use. Typically we want to choose the container image for the latest release. Verify the Docker image is selected when installing the latest RAPIDS release.
 
-Be sure to match the CUDA version in the container image with that installed on your Kubernetes nodes. The default CUDA version installed on GKE Stable is 11.4 for example, so we would want to choose that. From 11.5 onwards it doesn’t matter as they will be backward compatible. Copy the container image name from the install command (i.e. `rapidsai/rapidsai-core:22.06-cuda11.5-runtime-ubuntu20.04-py3.9`).
+Be sure to match the CUDA version in the container image with that installed on your Kubernetes nodes. The default CUDA version installed on GKE Stable is 11.4 for example, so we would want to choose that. From 11.5 onwards it doesn’t matter as they will be backward compatible. Copy the container image name from the install command (i.e. `{{ rapids_container }}`).
 
 ````{note}
 You can [check your CUDA version](https://jacobtomlinson.dev/posts/2022/how-to-check-your-nvidia-driver-and-cuda-version-in-kubernetes/) by creating a pod and running `nvidia-smi`. For example:
@@ -146,7 +146,7 @@ from dask_kubernetes.experimental import KubeCluster
 
 cluster = KubeCluster(
     name="rapids-dask",
-    image="rapidsai/rapidsai-core:22.06-cuda11.4-runtime-ubuntu20.04-py3.9",
+    image="{{ rapids_container }}",
     worker_command="dask-cuda-worker",
     n_workers=2,
     resources={"limits": {"nvidia.com/gpu": "1"}},
diff --git a/source/platforms/kubernetes.md b/source/platforms/kubernetes.md
@@ -37,7 +37,7 @@ spec:
     fsGroup: 0
   containers:
     - name: rapids-notebook
-      image: rapidsai/rapidsai-core:22.06-cuda11.5-runtime-ubuntu20.04-py3.9
+      image: { { rapids_container } }
       resources:
         limits:
           nvidia.com/gpu: 1
@@ -231,7 +231,7 @@ spec:
     fsGroup: 0
   containers:
     - name: rapids-notebook
-      image: rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9
+      image: {{ rapids_container }}
       resources:
         limits:
           nvidia.com/gpu: 1
diff --git a/source/tools/kubernetes/dask-helm-chart.md b/source/tools/kubernetes/dask-helm-chart.md
@@ -12,8 +12,40 @@ This helm chart can be configured to run RAPIDS by providing GPUs to the Juptyte
 
 Built on top of the Dask Helm Chart, `rapids-config.yaml` file contains additional configurations required to setup RAPIDS environment.
 
-```{literalinclude} ./dask-helm-chart/rapids-config.yaml
-:language: yaml
+```yaml
+# rapids-config.yaml
+scheduler:
+  image:
+    repository: "{{ rapids_container.split(":")[0] }}"
+    tag: "{{ rapids_container.split(":")[1] }}"
+  env:
+    - name: DISABLE_JUPYTER
+      value: "true"
+
+worker:
+  image:
+    repository: "{{ rapids_container.split(":")[0] }}"
+    tag: "{{ rapids_container.split(":")[1] }}"
+  dask_worker: "dask_cuda_worker"
+  replicas: 3
+  env:
+    - name: DISABLE_JUPYTER
+      value: "true"
+  resources:
+    limits:
+      nvidia.com/gpu: 1
+
+jupyter:
+  image:
+    repository: "{{ rapids_container.split(":")[0] }}"
+    tag: "{{ rapids_container.split(":")[1] }}"
+  servicePort: 8888
+  # Default password hash for "rapids"
+  password: "argon2:$argon2id$v=19$m=10240,t=10,p=8$TBbhubLuX7efZGRKQqIWtw$RG+jCBB2KYF2VQzxkhMNvHNyJU9MzNGTm2Eu2/f7Qpc"
+  resources:
+    limits:
+      nvidia.com/gpu: 1
+
 ```
 
 `[jupyter|scheduler|worker].image.*` is updated with the RAPIDS "runtime" image from the stable release,
diff --git a/source/tools/kubernetes/dask-helm-chart/rapids-config.yaml b/source/tools/kubernetes/dask-helm-chart/rapids-config.yaml
diff --git a/source/tools/kubernetes/dask-kubernetes.md b/source/tools/kubernetes/dask-kubernetes.md
diff --git a/source/tools/kubernetes/dask-kubernetes/gpu-worker-spec.yaml b/source/tools/kubernetes/dask-kubernetes/gpu-worker-spec.yaml
diff --git a/source/tools/kubernetes/dask-operator.md b/source/tools/kubernetes/dask-operator.md

Original file line number	Diff line number	Diff line change
`@@ -38,7 +38,7 @@ cluster = AzureVMCluster(`
`38`	`38`	`disk_size=100,`
`39`	`39`	`n_workers=2,`
`40`	`40`	`worker_class="dask_cuda.CUDAWorker",`
`41`		`- docker_image="rapidsai/rapidsai-core:22.12-cuda11.5-runtime-ubuntu20.04-py3.9",`
	`41`	`+ docker_image={{rapids_container}},`
`42`	`42`	`docker_args="-e DISABLE_JUPYTER=true -p 8787:8787 -p 8786:8786",`
`43`	`43`	`)`
`44`	`44`	```