bird-house · fmigneault · Feb 23, 2024 · Dec 15, 2023 · Dec 15, 2023 · Dec 15, 2023
@@ -29,3 +29,4 @@ venv/
 
 ## Testing
 .pytest_cache/
+*.log
@@ -15,13 +15,43 @@
 [Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
 ------------------------------------------------------------------------------------------------------------------
 
-## Fixes
+## Changes
+- Compose script utilities:
+  * Add `BIRDHOUSE_COLOR` option and various logging/messaging definitions in `birdhouse/scripts/logging.include.sh`.
+  * Replace all explicit color "logging" related `echo` in scripts by a utility `log {LEVEL} {message}` function
+    that employs variables `LOG_DEBUG`, `LOG_INFO`, `LOG_WARN`, `LOG_ERROR` and `LOG_CRITICAL` as applicable per
+    respective messages to report logging messages in a standard approach.
+    Colors can be disabled with `BIRDHOUSE_COLOR=0` and logging level can be set with `BIRDHOUSE_LOG_LEVEL={LEVEL}`
+    where all levels above or equal to the configured one will be displayed (default logging level is `INFO`).
+  * Move `read-configs.include.sh` into `birdhouse/scripts` along other include scripts.
+  * Add `info` option (ie: `pavics-compose.sh info`) that will stop processing just before `docker-compose` call.
+    This can be used to perform a "dry-run" of the command and validate that was is loaded is as expected, by inspecting
+    provided log messages.
+  * Replace older backtick (``` ` ```) executions by `$(...)` representation except for `eval` calls that require
+    them for backward compatibility of `sh` on some server instances.
+  * Modify the `sh -x` calls to scripts listed in `COMPONENT_PRE_COMPOSE_UP` and `COMPONENT_POST_COMPOSE_UP` to employ
+    the `-x` flag (showing commands) only when `BIRDHOUSE_LOG_LEVEL=DEBUG`.
+
+- Defaults:
+  * Add multiple `SERVER_[...]` variables with defaults using previously hard coded values referring to PAVICS.
+    These variables use a special combination of `DELAYED_EVAL` and `OPTIONAL_VARS` definitions that can make use
+    of a variable formatted as `<ANY_NAME>='${__DEFAULT__<ANY_NAME>}'` that will print a warning messages indicating
+    that the default is employed, although *STRONGLY* recommended to be overridden. This allows a middle ground between
+    backward-compatible `env.local` while flagging potentially misused configurations.
 
+## Fixes
+- Canarie-API: updated references
+  * Use the new `SERVER_[...]` variables.
+  * Replace the LICENSE URL of the server node pointing
+    at [Ouranosinc/pavics-sdi](https://github.com/Ouranosinc/pavics-sdi) instead
+    of intended [bird-house/birdhouse-deploy](https://github.com/bird-house/birdhouse-deploy).
+- Magpie: ensure that the `MAGPIE_ADMIN_USERNAME` variable is respected
+  * When determining the `JUPYTERHUB_ADMIN_USERS` variable
+  * Double check that it is being respected everywhere else
 - env.local.example: fix `JUPYTERHUB_CONFIG_OVERRIDE` comment section
 
   `JUPYTERHUB_CONFIG_OVERRIDE` was disconnected from its sample code.
 
-
 [2.0.6](https://github.com/bird-house/birdhouse-deploy/tree/2.0.6) (2024-02-15)
 ------------------------------------------------------------------------------------------------------------------
 

@@ -148,7 +148,7 @@ instructions below.
 Manual instructions:
 
 * Go to
-  ``https://<PAVICS_FQDN>/magpie/ui/login`` and login with the ``admin`` user. The password should be in ``env.local``.
+  ``https://<PAVICS_FQDN>/magpie/ui/login`` and login with the ``MAGPIE_ADMIN_USERNAME`` user. The password should be in ``env.local``.
 
 * Then go to ``https://<PAVICS_FQDN>/magpie/ui/users/add``.
 

@@ -304,7 +304,7 @@ birdhouse-deploy software stack and the machine that it is running on. It is hig
 make these routes available to anyone who does not have proper access permissions.
 
 Add existing users to the ``monitoring`` group to allow them access to the various monitoring WebUI.
-This way, we do not need to share the ``admin`` user account and do not have to add them to the
+This way, we do not need to share the ``MAGPIE_ADMIN_USERNAME`` user account and do not have to add them to the
 ``administrators`` group, which would give them too much permissions.
 
 

@@ -111,23 +111,23 @@ SERVICES = {
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
             'version': '2.0.6',
             'releaseTime': '2024-02-15T16:29:01Z',
-            'institution': 'Ouranos',
-            'researchSubject': 'Climatology',
+            'institution': '${SERVER_INSTITUTION}',
+            'researchSubject': '${SERVER_SUBJECT}',
             'supportEmail': '${SUPPORT_EMAIL}',
             'category': 'Resource/Cloud Management',
-            'tags': ['Climatology']
+            'tags': [tag.strip() for tag in "${SERVER_TAGS}".split(",") if tag.strip()],
         },
         'stats': {
             'method': '.*',
             'route': '(?!)'  # this will be set by CANARIE_STATS_ROUTES (see below)
         },
         'redirect': {
-            'doc': 'https://pavics-sdi.readthedocs.io/en/latest/arch/backend.html',
-            'releasenotes': 'https://github.com/bird-house/birdhouse-deploy/blob/master/CHANGES.md',
-            'support': 'https://github.com/bird-house/birdhouse-deploy/issues',
+            'doc': '${SERVER_DOCUMENTATION_URL}',
+            'releasenotes': '${SERVER_RELEASE_NOTES_URL}',
+            'support': '${SERVER_SUPPORT_URL}',
             'source': 'https://github.com/bird-house/birdhouse-deploy',
             'tryme': 'https://${PAVICS_FQDN_PUBLIC}',
-            'licence': 'https://pavics-sdi.readthedocs.io/en/latest/license.html',
+            'licence': '${SERVER_LICENSE_URL}',
             'provenance': 'https://pavics-sdi.readthedocs.io/en/latest/provenance/index.html'
         },
         'monitoring': {}  # filled in after processing everything, see end of script
@@ -144,10 +144,10 @@ PLATFORMS = {
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
             'version': '2.0.6',
             'releaseTime': '2024-02-15T16:29:01Z',
-            'institution': 'Ouranos',
-            'researchSubject': 'Climatology',
+            'institution': '${SERVER_INSTITUTION}',
+            'researchSubject': '${SERVER_SUBJECT}',
             'supportEmail': '${SUPPORT_EMAIL}',
-            'tags': ['Climatology', 'Cloud']
+            'tags': [tag.strip() for tag in "${SERVER_TAGS}".split(",") if tag.strip()],
         },
         'stats': {
             'method': '.*',

@@ -1,10 +1,17 @@
 #!/bin/sh
 
-THIS_FILE="`realpath "$0"`"
-THIS_DIR="`dirname "$THIS_FILE"`"
+THIS_FILE="$(readlink -f "$0" || realpath "$0")"
+THIS_DIR="$(dirname "$THIS_FILE")"
 COMPOSE_DIR="$THIS_DIR/../.."
 
+if [ -f "${COMPOSE_DIR}/scripts/read-configs.include.sh" ]; then
+  . "${COMPOSE_DIR}/scripts/read-configs.include.sh"
+
+  # resolve GEOSERVER_DATA_DIR
+  read_configs
+fi
+
 if [ ! -f "${GEOSERVER_DATA_DIR}/global.xml" ]; then
-  echo "fix GeoServer data dir permission on first run only, when data dir do not exist yet."
-  FIRST_RUN_ONLY=1 "$COMPOSE_DIR"/deployment/fix-geoserver-data-dir-perm
+  log INFO "fix GeoServer data dir permission on first run only, when data dir do not exist yet."
+  FIRST_RUN_ONLY=1 "${COMPOSE_DIR}"/deployment/fix-geoserver-data-dir-perm
 fi
@@ -68,9 +68,13 @@ export JUPYTERHUB_CRYPT_KEY=
 # JUPYTERHUB_CRYPT_KEY is set.
 export JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE=60
 
+# Usernames that should be given admin access in jupyterhub
+export JUPYTERHUB_ADMIN_USERS='{\"${MAGPIE_ADMIN_USERNAME}\"}'  # python set syntax
+
 export DELAYED_EVAL="
   $DELAYED_EVAL
   JUPYTERHUB_USER_DATA_DIR
+  JUPYTERHUB_ADMIN_USERS
 "
 
 # add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here

@@ -3,12 +3,11 @@
 # Create JSON files containing the version information, available services, and
 # enabled components. These files will be served by the nginx proxy as static files.
 
-THIS_FILE="$(realpath "$0")"
+THIS_FILE="$(readlink -f "$0" || realpath "$0")"
 THIS_DIR="$(dirname "$THIS_FILE")"
 
 mkdir -p "${THIS_DIR}/static"
 
 echo "${BIRDHOUSE_VERSION_JSON}" > "${THIS_DIR}/static/version.json"
 echo "${BIRDHOUSE_DEPLOY_SERVICES_JSON}" > "${THIS_DIR}/static/services.json"
 echo "${BIRDHOUSE_DEPLOY_COMPONENTS_JSON}" > "${THIS_DIR}/static/components.json"
-
@@ -31,17 +31,76 @@ export DELAYED_EVAL="
   PAVICS_FQDN_PUBLIC
   DATA_PERSIST_SHARED_ROOT
   WPS_OUTPUTS_DIR
+  SERVER_NAME
+  SERVER_DESCRIPTION
+  SERVER_INSTITUTION
+  SERVER_SUBJECT
+  SERVER_TAGS
+  SERVER_DOCUMENTATION_URL
+  SERVER_RELEASE_NOTES_URL
+  SERVER_SUPPORT_URL
+  SERVER_LICENSE_URL
 "
 
-
-export SERVER_NAME=PAVICS
-export SERVER_DESCRIPTION="
+# Server Identification Details
+#   Following definitions should definitenly be updated.
+#   Previous defaults are defined for backward-compatibility.
+#   If not overridden explicitly by their non '__' prefixed variant,
+#   a WARN message will be displayed by pavics-compose.
+export __DEFAULT__SERVER_NAME="PAVICS"
+export __DEFAULT__SERVER_DESCRIPTION="
 The PAVICS (Power Analytics for Visualization of Climate Science) platform is a collection of
 climate analysis services served through Open Geospatial Consortium (OGC) protocols.
 These services include data access, processing and visualization. Both data and algorithms
 can be accessed either programmatically, through OGC-compliant clients such as QGIS or ArcGIS,
 or a custom web interface.
 "
+export __DEFAULT__SERVER_INSTITUTION="Ouranos"
+export __DEFAULT__SERVER_SUBJECT="Climatology"
+# below can be a CSV list of tags
+export __DEFAULT__SERVER_TAGS="Climatology"
+export __DEFAULT__SERVER_DOCUMENTATION_URL="https://pavics-sdi.readthedocs.io/en/latest/arch/backend.html"
+export __DEFAULT__SERVER_RELEASE_NOTES_URL="https://github.com/bird-house/birdhouse-deploy/blob/master/CHANGES.md"
+export __DEFAULT__SERVER_SUPPORT_URL="https://github.com/bird-house/birdhouse-deploy/issues"
+# NOTE:
+#   This value does not use the previously hard coded default.
+#   Previous default pointed at the wrong repository with a mismatching LICENSE file.
+export __DEFAULT__SERVER_LICENSE_URL="https://github.com/bird-house/birdhouse-deploy/blob/master/LICENSE"
+
+# apply overrides or fallback above defaults with delayed evaluation
+# see also: 'SUPPORT_EMAIL'
+# This parameter is relevant to server details, but uses the old name for backward compatibility.
+export SERVER_NAME='${__DEFAULT__SERVER_NAME}'
+export SERVER_DESCRIPTION='${__DEFAULT__SERVER_DESCRIPTION}'
+export SERVER_INSTITUTION='${__DEFAULT__SERVER_INSTITUTION}'
+export SERVER_SUBJECT='${__DEFAULT__SERVER_SUBJECT}'
+export SERVER_TAGS='${__DEFAULT__SERVER_TAGS}'
+export SERVER_DOCUMENTATION_URL='${__DEFAULT__SERVER_DOCUMENTATION_URL}'
+export SERVER_RELEASE_NOTES_URL='${__DEFAULT__SERVER_RELEASE_NOTES_URL}'
+export SERVER_SUPPORT_URL='${__DEFAULT__SERVER_SUPPORT_URL}'
+export SERVER_LICENSE_URL='${__DEFAULT__SERVER_LICENSE_URL}'
+
+# Defaults for required variables recommended for override for security reasons.
+# Those will not be set explicitly as defaults to ensure they are overridden explicitly by the instance.
+# These values would be detected only if the instance was configured using a copy of 'env.local.example'.
+# Any default value change here should be synchronized with the corresponding value in 'env.local.example'.
+export __DEFAULT__MAGPIE_SECRET="itzaseekrit"
+export __DEFAULT__MAGPIE_ADMIN_USERNAME="admin"
+export __DEFAULT__MAGPIE_ADMIN_PASSWORD="qwertyqwerty!"
+export __DEFAULT__POSTGRES_PAVICS_USERNAME="postgres-pavics"
+export __DEFAULT__POSTGRES_PAVICS_PASSWORD="postgres-qwerty"
+export __DEFAULT__POSTGRES_MAGPIE_USERNAME="postgres-magpie"
+export __DEFAULT__POSTGRES_MAGPIE_PASSWORD="postgres-qwerty"
+export __DEFAULT__GEOSERVER_ADMIN_USER="admingeo"
+export __DEFAULT__GEOSERVER_ADMIN_PASSWORD="geoserverpass"
+#############################################################################
+# Deprecated vars (for components in the ./deprecated-components directory)
+#############################################################################
+export __DEFAULT__TOMCAT_NCWMS_PASSWORD="ncwmspass"
+export __DEFAULT__CATALOG_USERNAME="admin-catalog"
+export __DEFAULT__CATALOG_PASSWORD="qwerty"
+export __DEFAULT__PHOENIX_PASSWORD="phoenix_pass"
+export __DEFAULT__PHOENIX_PASSWORD_HASH="sha256:123456789012:1234567890123456789012345678901234567890123456789012345678901234"
 
 export DEFAULT_CONF_DIRS='
   ./components/proxy

@@ -46,8 +46,8 @@ certbotwrapper START_TIME=$START_TIME"
 
 set -x
 
-THIS_FILE="`realpath "$0"`"
-THIS_DIR="`dirname "$THIS_FILE"`"
+THIS_FILE="$(readlink -f "$0" || realpath "$0")"
+THIS_DIR="$(dirname "$THIS_FILE")"
 SAVED_PWD="`pwd`"
 
 . "$THIS_DIR/../read-configs.include.sh"

@@ -79,8 +79,8 @@ else
     shift
 fi
 
-COMPOSE_DIR="`realpath "$COMPOSE_DIR"`"
-REPO_ROOT="`realpath "$COMPOSE_DIR/.."`"
+COMPOSE_DIR="$(realpath "$COMPOSE_DIR")"
+REPO_ROOT="$(realpath "$COMPOSE_DIR/..")"
 
 if [ ! -f "$COMPOSE_DIR/docker-compose.yml" -o \
      ! -f "$COMPOSE_DIR/pavics-compose.sh" ]; then
@@ -105,7 +105,7 @@ cd $COMPOSE_DIR
 START_TIME="`date -Isecond`"
 echo "deploy START_TIME=$START_TIME"
 
-. "$COMPOSE_DIR/read-configs.include.sh"
+. "${COMPOSE_DIR}/scripts/read-configs.include.sh"
 
 # Read AUTODEPLOY_EXTRA_REPOS
 read_basic_configs_only
@@ -171,7 +171,7 @@ done
 cd $COMPOSE_DIR
 
 # reload again after git pull because this file could be changed by the pull
-. "$COMPOSE_DIR/read-configs.include.sh"
+. "${COMPOSE_DIR}/scripts/read-configs.include.sh"
 
 # reload again after default.env since env.local can override default.env
 # (ex: JUPYTERHUB_USER_DATA_DIR)

@@ -8,13 +8,13 @@
 # global.xml will exist and this script will not execute.  Without
 # FIRST_RUN_ONLY, this script will always execute.
 
-THIS_FILE="`realpath "$0"`"
-THIS_DIR="`dirname "$THIS_FILE"`"
+THIS_FILE="$(readlink -f "$0" || realpath "$0")"
+THIS_DIR="$(dirname "$THIS_FILE")"
 
 # Go to repo root.
 cd $THIS_DIR/../..
 
-. birdhouse/read-configs.include.sh
+. birdhouse/scripts/read-configs.include.sh
 
 # Get BASH_IMAGE
 # Get GEOSERVER_DATA_DIR

@@ -37,13 +37,13 @@
 # So the setfacl solution is the simplest, most portable/generic and most
 # localized (only the directories we need) solution.
 
-THIS_FILE="`realpath "$0"`"
-THIS_DIR="`dirname "$THIS_FILE"`"
+THIS_FILE="$(readlink -f "$0" || realpath "$0")"
+THIS_DIR="$(dirname "$THIS_FILE")"
 
 # Go to repo root.
 cd $THIS_DIR/../..
 
-. birdhouse/read-configs.include.sh
+. birdhouse/scripts/read-configs.include.sh
 
 # Get GEOSERVER_DATA_DIR, JUPYTERHUB_USER_DATA_DIR, MAGPIE_PERSIST_DIR
 read_configs

@@ -29,7 +29,7 @@ if [ ! -e "$REPO_ROOT/birdhouse/deployment/trigger-deploy-notebook" ]; then
     exit 2
 fi
 
-. "$REPO_ROOT/birdhouse/read-configs.include.sh"
+. "$REPO_ROOT/birdhouse/scripts/read-configs.include.sh"
 
 # Get JUPYTERHUB_USER_DATA_DIR
 read_configs

@@ -34,8 +34,8 @@ if [ -z "$COMPOSE_DIR" ]; then
     COMPOSE_DIR="$(dirname -- "$(dirname -- "$(realpath "$0")")")"
 fi
 
-if [ -f "$COMPOSE_DIR/read-configs.include.sh" ]; then
-    . "$COMPOSE_DIR/read-configs.include.sh"
+if [ -f "${COMPOSE_DIR}/scripts/read-configs.include.sh" ]; then
+    . "${COMPOSE_DIR}/scripts/read-configs.include.sh"
 
     # Get JUPYTERHUB_USER_DATA_DIR
     read_configs

@@ -59,7 +59,7 @@ else
     shift
 fi
 
-COMPOSE_DIR="`realpath "$COMPOSE_DIR"`"
+COMPOSE_DIR="$(realpath "$COMPOSE_DIR")"
 
 if [ ! -f "$COMPOSE_DIR/docker-compose.yml" ]; then
     echo "ERROR: missing docker-compose.yml in '$COMPOSE_DIR'" 1>&2
@@ -173,7 +173,7 @@ START_TIME="`date -Isecond`"
 echo "==========
 triggerdeploy START_TIME=$START_TIME"
 
-. "$COMPOSE_DIR/read-configs.include.sh"
+. "${COMPOSE_DIR}/scripts/read-configs.include.sh"
 
 # Read AUTODEPLOY_EXTRA_REPOS
 read_basic_configs_only