Merge pull request #103 from inab/full_circle

Version 1.0.0a0

.pre-commit-config.yaml

@@ -85,10 +85,16 @@ repos:
         args: [ wfexs_backend/schemas ]
 
   - repo: https://github.com/jmfernandez/pre-commit_mirrors-actionlint.git
-    rev: v1.6.25
+    rev: v1.7.1
     hooks:
       - id: actionlint
 
+  - repo: https://github.com/jmfernandez/pre-commit_mirrors-hadolint.git
+    rev: v2.12.0
+    hooks:
+      - id: hadolint
+        args: [ -t, warning]
+
   - repo: https://github.com/ambv/black.git
     rev: 23.3.0
     hooks:

CITATION.cff

@@ -28,4 +28,4 @@ message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/inab/WfExS-backend"
 type: software
 title: "WfExS-backend"
-version: 0.99.9
+version: 1.0.0a0

INSTALL.md

@@ -1,11 +1,139 @@
 # "Easy" install and setup a Workflow Execution Service backend instance
 
-## "Easy" setup of core and main software dependencies
+## Easy creation of WfExS container image
 
-There is an automated installer at [full-installer.bash](full-installer.bash):
+This section describes how to build a container image containing WfExS and its preconditions.
+
+### Docker
+
+The precondition is having Docker properly setup and running.
+
+You can build the Docker image for an specific version (release, tag, branch or commit)
+without fetching a full copy of the repo or the Dockerfile recipe,
+just using next bash pattern:
+
+```bash
+# WFEXS_VER can be either a branch, a tag or a commit hash
+WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+docker build -t inab/wfexs-backend:${WFEXS_VER} \
+--build-arg wfexs_checkout="${WFEXS_VER}" \
+https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
+```
+
+Alternatively, if the docker client does not accept URLs, you need to have
+a local copy of the recipe, and next command line from the project root will help you:
+
+```bash
+# WFEXS_VER can be either a branch, a tag or a commit hash
+WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+mkdir WfExS_docker_build
+cd WfExS_docker_build
+curl -O https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
+
+docker build -t inab/wfexs-backend:${WFEXS_VER} \
+--build-arg wfexs_checkout="${WFEXS_VER}" \
+Dockerfile
+```
+
+### Podman
+
+The precondition is having Podman properly setup and running.
+
+Mimicking what it can be performed with Docker, you can build the Podman
+image for an specific version (release, tag, branch or commit)
+without fetching a full copy of the repo or the recipe,
+just using next bash pattern:
+
+```bash
+# WFEXS_VER can be either a branch, a tag or a commit hash
+WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+podman build -t inab/wfexs-backend:${WFEXS_VER} \
+--build-arg wfexs_checkout="${WFEXS_VER}" \
+https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
+```
+
+Alternatively, if the podman client does not accept URLs, you need to have
+a local copy of the recipe, and next command line from the project root will help you:
+
+```bash
+# WFEXS_VER can be either a branch, a tag or a commit hash
+WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+mkdir WfExS_podman_build
+cd WfExS_podman_build
+curl -O https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Dockerfile
+
+podman build -t inab/wfexs-backend:${WFEXS_VER} \
+--build-arg wfexs_checkout="${WFEXS_VER}" \
+Dockerfile
+```
+
+### SIF image
+
+The precondition is having either Apptainer or Singularity properly setup. There are three different routes to create a SIF image of WfExS:
+
+* First approach requires either using curl or having a local copy of the repository
+  **and** a modern enough version of either apptainer (1.3 or later)
+  or singularity (4.0 or later). 
+
+  ```bash
+  # WFEXS_VER can be either a branch, a tag or a commit hash
+  WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+  mkdir WfExS_SIF_build
+  cd WfExS_SIF_build
+  curl -O https://raw.githubusercontent.com/inab/WfExS-backend/${WFEXS_VER}/container_recipes/Singularity.def
+  singularity build \
+  --build-arg wfexs_checkout="${WFEXS_VER}" \
+  wfexs-backend-${WFEXS_VER}.sif Singularity.def
+  ```
+
+  ```bash
+  # WFEXS_VER can be either a branch, a tag or a commit hash
+  WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+  singularity build \
+  --build-arg wfexs_checkout="${WFEXS_VER}" \
+  wfexs-backend-${WFEXS_VER}.sif container_recipes/Singularity.def
+  ```
+
+* Second approach involves to first create the WfExS docker image locally,
+  following the pattern previously described, and then telling apptainer / singularity
+  to build it:
+
+  ```bash
+  # Remember to use the correct tag!!!
+  WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+  singularity build wfexs-${WFEXS_VER}.sif docker-daemon://inab/wfexs-backend:${WFEXS_VER}
+  ```
+
+* Third approach involves to first create either the local docker or podman image,
+  as it was described above. Then, you have to save it to an image file,
+  which will be used to build the SIF image.
+
+  for the WfExS podman image locally,
+  following the pattern previously described, and then telling apptainer / singularity
+  to build it:
+
+  ```bash
+  mkdir WfExS_SIF_build
+  cd WfExS_SIF_build
+
+  # Remember to use the correct tag!!!
+  WFEXS_VER=574fe343c0b59eecd95afbc67894456359ebe649
+
+  # Next command should be used if you used podman to build the local image
+  podman save -o wfexs-backend-${WFEXS_VER}.tar inab/wfexs-backend:${WFEXS_VER}
+
+  # Next command should be used if you used docker to build the local image
+  docker save -o wfexs-backend-${WFEXS_VER}.tar inab/wfexs-backend:${WFEXS_VER}
+
+  singularity build wfexs-${WFEXS_VER}.sif docker-archive:wfexs-backend-${WFEXS_VER}.tar
+  ```
+
+## "Easy" local setup of core and main software dependencies
+
+There is an automated installer at [full-installer.bash](container_recipes/full-installer.bash), which is also used inside the docker:
 
 ```bash
-./full-installer.bash
+container_recipes/full-installer.bash
 ```
 
 which assumes both essential build dependencies
@@ -17,21 +145,21 @@ The automated installer installs both core dependencies and it fetches and insta
   * A static bash copy: needed by Nextflow runner to monkey-patch some containers which do not have bash, or whose bash copy is buggy.
 
 If you also want to install [singularity](https://sylabs.io/singularity/) or
-[apptainer](https://apptainer.org) at the WfExS-backend virtual environment, and you are using Ubuntu Linux, a rootless setup is achieved using either [singularity-local-installer.bash](singularity-local-installer.bash)
-or [apptainer-local-installer.bash](apptainer-local-installer.bash).
+[apptainer](https://apptainer.org) at the WfExS-backend virtual environment, and you are using Ubuntu Linux, a rootless setup is achieved using either [singularity-local-installer.bash](container_recipes/singularity-local-installer.bash)
+or [apptainer-local-installer.bash](container_recipes/apptainer-local-installer.bash).
 At most only one of them can be locally installed, because as of
 September 2022 workflow engines like `cwltool` or `nextflow` still use the
 hardcoded name of `singularity`. So, the apptainer installer has to create
 a "singularity" symlink pointing to "apptainer".
 
 ```bash
 # For singularity
-./singularity-local-installer.bash
+container_recipes/singularity-local-installer.bash
 ```
 
 ```bash
 # For apptainer
-./apptainer-local-installer.bash
+container_recipes/apptainer-local-installer.bash
 ```
 
 This setup will only work on Linux systems with cgroups v2 enabled. You will also need to install the package which provides `mksquashfs`, which is `squashfs-tools` both in Debian and Ubuntu.
@@ -40,12 +168,12 @@ The scripts only install singularity or apptainer when it is not available. If y
 
 ```bash
 # For singularity
-./singularity-local-installer.bash force
+container_recipes/singularity-local-installer.bash force
 ```
 
 ```bash
 # For apptainer
-./apptainer-local-installer.bash force
+container_recipes/apptainer-local-installer.bash force
 ```
 
 ## Core Dependencies
@@ -59,7 +187,7 @@ This workflow execution service backend is written for Python 3.7 and later.
 * The creation of a virtual environment where to install WfExS backend dependencies can be done running:
 
 ```bash
-./basic-installer.bash
+container_recipes/basic-installer.bash
 ```
 
 * If you upgrade your Python installation (from version 3.8 to 3.9 or later, for instance), or you move this folder to a different location after following this instructions, you may need to remove and reinstall the virtual environment.
@@ -101,10 +229,8 @@ All the development dependencies are declared at [dev-requirements.txt](dev-requ
 ```bash
 python3 -m venv .pyWEenv
 source .pyWEenv/bin/activate
-pip install --upgrade pip wheel
-pip install -r requirements.txt
-pip install -r dev-requirements.txt
-pip install -r mypy-requirements.txt
+pip install --require-virtualenv --upgrade pip wheel
+pip install --require-virtualenv -r requirements.txt -r dev-requirements.txt -r mypy-requirements.txt
 ```
 
 One of these dependencies is [pre-commit](https://pre-commit.com/), whose rules are declared at [.pre-commit-config.yaml](.pre-commit-config.yaml) (there are special versions of these rules for GitHub).
@@ -147,6 +273,6 @@ flake8 --ignore E501 wfexs_backend
 ```
 
 # License
-* © 2020-2022 Barcelona Supercomputing Center (BSC), ES
+* © 2020-2024 Barcelona Supercomputing Center (BSC), ES
 
 Licensed under the Apache License, version 2.0 <https://www.apache.org/licenses/LICENSE-2.0>, see the file `LICENSE.txt` for details.