add optional-components/stac-data-proxy

CHANGES.md

@@ -15,7 +15,20 @@
 [Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
 ------------------------------------------------------------------------------------------------------------------
 
-[//]: # (list changes here, using '-' for each new entry, remove this when items are added)
+- `optional-components/stac-data-proxy`: add a new feature to allow hosting of local STAC assets.
+
+  The new component defines variables `STAC_DATA_PROXY_DIR_PATH` (default `${DATA_PERSIST_ROOT}/stac-data`) and
+  `STAC_DATA_PROXY_URL_PATH` (default `/data/stac`) that are aliased (mapped) under `nginx` to provide a URL
+  where locally hosted STAC assets can be downloaded from. This allows a server node to be a proper data provider,
+  where its STAC-API can return Catalog, Collection and Item definitions that points at these local assets available
+  through the `STAC_DATA_PROXY_URL_PATH` endpoint.
+
+  When enabled, this component can be combined with `optional-components/secure-data-proxy` to allow per-resource
+  access control of the contents under `STAC_DATA_PROXY_DIR_PATH` by setting relevant Magpie permissions under service
+  `secure-data-proxy` for children resources that correspond to `STAC_DATA_PROXY_URL_PATH`. Otherwise, the path and
+  all of its contents are publicly available, in the same fashion that WPS outputs are managed without
+  `optional-components/secure-data-proxy`. More details are provided under the component's
+  [README](./birdhouse/optional-components/README.rst#provide-a-proxy-for-local-stac-asset-hosting).
 
 [1.37.2](https://github.com/bird-house/birdhouse-deploy/tree/1.37.2) (2023-11-10)
 ------------------------------------------------------------------------------------------------------------------

birdhouse/optional-components/README.rst

@@ -326,6 +326,51 @@ To enable this optional-component:
 - Add ``./optional-components/stac-public-access`` to ``EXTRA_CONF_DIRS``.
 
 
+Provide a proxy for local STAC asset hosting
+--------------------------------------------------------
+
+STAC data proxy allows to host the URL location defined by ``PAVICS_FQDN_PUBLIC`` and ``STAC_DATA_PROXY_URL_PATH``
+to provide access to files contained within ``STAC_DATA_PROXY_DIR_PATH``.
+
+The ``STAC_DATA_PROXY_DIR_PATH`` location can be used to hold STAC Assets defined by the current server node
+(in contrast to STAC definitions that would refer to remote locations), such that the node can be the original
+location of new data, or to make a new local replication of remote data.
+
+To enable this optional-component:
+
+- Edit ``env.local`` (a copy of `env.local.example`_)
+- Add ``./optional-components/stac-data-proxy`` to ``EXTRA_CONF_DIRS``.
+- Optionally, add any other relevant components to control access as desired (see below).
+
+When using this component, access to the endpoint defined by ``STAC_DATA_PROXY_URL_PATH``, and therefore all
+corresponding files contained under mapped ``STAC_DATA_PROXY_DIR_PATH`` will depend on how this
+feature is combined with ``./optional-components/stac-public-access`` and ``./optional-components/secure-data-proxy``.
+Following are the possible combinations and obtained behaviors:
+
+.. list-table::
+    :widths: 40,60
+
+    * - Enabled Components
+      - Only ``./optional-components/stac-data-proxy`` is enabled.
+      - Both ``./optional-components/stac-data-proxy`` and ``./optional-components/stac-public-access`` are enabled.
+      - Both ``./optional-components/stac-data-proxy`` and ``./optional-components/secure-data-proxy`` are enabled.
+      - All of ``./optional-components/stac-data-proxy``, ``./optional-components/stac-public-access`` and
+        ``./optional-components/secure-data-proxy`` are enabled.
+    * - Obtained Behaviors
+      - All data under ``STAC_DATA_PROXY_URL_PATH`` is publicly accessible without authorization control
+        and specific resource access cannot be managed per content. However, since STAC-API itself is not made public,
+        the STAC Catalog, Collections and Items cannot be accessed publicly
+        (*note*: this is most probably never desired).
+      - All data under ``STAC_DATA_PROXY_URL_PATH`` is publicly accessible without possibility to manage per-resource
+        access. However, this public access is aligned with publicly accessible STAC-API endpoints and contents.
+      - All data under ``STAC_DATA_PROXY_URL_PATH`` is protected (by default, admin-only), but can be granted access
+        on a per-user, per-group and per-resource basis according to permissions applied by the administrator.
+        Since STAC-API is not made public by default, the administrator can decide whether they grant access only to
+        STAC metadata (Catalog, Collection, Items) with permission applied on the ``stac`` Magpie service, only to
+        assets data with permission under the ``stac-data-proxy``, or both.
+      - Similar to the previous case, allowing full authorization management control by the administrator, but contents
+        are publicly accessible by default. To revoke access, a Magpie administrator has to apply a ``deny`` permission.
+
 X-Robots-Tag Header
 ---------------------------
 

birdhouse/optional-components/stac-data-proxy/.gitignore

@@ -0,0 +1,2 @@
+config/magpie/permissions.cfg
+config/proxy/conf.extra-service.d/stac-proxy-data.conf

birdhouse/optional-components/stac-data-proxy/config/magpie/docker-compose-extra.yml

@@ -0,0 +1,5 @@
+version: "3.4"
+services:
+  magpie:
+    volumes:
+      - ./optional-components/stac-data-proxy/config/magpie/permissions.cfg:${MAGPIE_PERMISSIONS_CONFIG_PATH}/stac-data-proxy.cfg:ro

birdhouse/optional-components/stac-data-proxy/config/magpie/permissions.cfg.template

@@ -0,0 +1,12 @@
+# NOTE:
+# Assume 'secure-data-proxy' would exist if needed.
+# Since 'secure-data-proxy' < 'stac-data-proxy', it should be loaded first.
+permissions:
+  # following permission does not change anything technically (full access for admins)
+  # it is employed only to set up the relevant resource path and make permission customization easier by Magpie API/UI
+  - service: secure-data-proxy
+    resource: ${STAC_DATA_PROXY_URL_PATH}
+    type: route
+    permission: read
+    group: administrators
+    action: create

birdhouse/optional-components/stac-data-proxy/config/proxy/conf.extra-service.d/stac-proxy-data.conf.template

@@ -0,0 +1,5 @@
+    location ${STAC_DATA_PROXY_URL_PATH}/ {
+        ${SECURE_DATA_PROXY_AUTH_INCLUDE}
+
+        alias /stac-data-proxy/;
+    }

birdhouse/optional-components/stac-data-proxy/config/proxy/docker-compose-extra.yml

@@ -0,0 +1,7 @@
+version: "3.4"
+services:
+  proxy:
+    volumes:
+    - ./optional-components/stac-data-proxy/config/proxy/conf.extra-service.d:/etc/nginx/conf.extra-service.d/stac-data-proxy:ro
+    # NOTE: data for hosted STAC assets, not to be confused with 'stac-db' for internal STAC catalog definitions
+    - ${STAC_DATA_PROXY_DIR_PATH}:/stac-data-proxy

birdhouse/optional-components/stac-data-proxy/default.env

@@ -0,0 +1,42 @@
+#!/bin/sh
+
+# All env in this default.env can be overridden by env.local.
+
+# All env in this default.env must NOT depend on any other env.  If they do, they
+# must use single quotes to avoid early expansion before overrides in env.local
+# are applied and must be added to the list of DELAYED_EVAL.
+
+# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
+# single quotes are important in below list to keep variable names intact until 'pavics-compose' parses them
+EXTRA_VARS='
+  $STAC_DATA_PROXY_DIR_PATH
+  $STAC_DATA_PROXY_URL_PATH
+'
+
+# extend the original 'VARS' from 'birdhouse/pavics-compose.sh' to employ them for template substitution
+# adding them to 'VARS', they will also be validated in case of override of 'default.env' using 'env.local'
+VARS="$VARS $EXTRA_VARS"
+
+# Direcotry path that will be used as volume mount for storing hosted STAC assets data
+# NOTE:
+#   Hosting is not performed by the API itself. Data is expected to already reside in that
+#   location when referenced by STAC Collections and Items to make them accessible externally.
+export STAC_DATA_PROXY_DIR_PATH='${DATA_PERSIST_ROOT}/stac-data'
+
+# URL path (after PAVICS_FQDN_PUBLIC) that will be used to proxy local STAC assets data
+export STAC_DATA_PROXY_URL_PATH="/data/stac"
+
+DELAYED_EVAL="
+  $DELAYED_EVAL
+  STAC_DATA_PROXY_DIR_PATH
+"
+
+# add any component that this component requires to run
+# NOTE:
+#   './optional-component/secure-data-proxy' is purposely omitted from dependencies
+#   if 'EXTRA_CONF_DIRS' enabled it as well, the proxy path/alias will have relevant auth request enabled
+#   otherwise, it will use by default the public access with no prior nginx auth validation
+COMPONENT_DEPENDENCIES="
+  ./components/stac
+  ./config/proxy
+"

birdhouse/optional-components/stac-public-access/.gitignore

@@ -0,0 +1 @@
+config/stac-data-proxy/permissions.cfg

birdhouse/optional-components/stac-public-access/config/stac-data-proxy/docker-compose-extra.yml

@@ -0,0 +1,5 @@
+version: "3.4"
+services:
+  magpie:
+    volumes:
+    - ./optional-components/stac-public-access/config/stac-data-proxy/permissions.cfg:${MAGPIE_PERMISSIONS_CONFIG_PATH}/stac-data-proxy-public.cfg:ro

birdhouse/optional-components/stac-public-access/config/stac-data-proxy/permissions.cfg.template

@@ -0,0 +1,10 @@
+# NOTE:
+# Assume 'secure-data-proxy' would exist if needed.
+# Since 'secure-data-proxy' < 'stac-data-proxy-public', it should be loaded first.
+permissions:
+  - service: secure-data-proxy
+    resource: ${STAC_DATA_PROXY_URL_PATH}
+    type: route
+    permission: read
+    group: anonymous
+    action: create