fix: make installation relocatable (#215)

Author: c-dilks

.github/workflows/ci.yml

@@ -342,6 +342,13 @@ jobs:
           echo '' >> $GITHUB_STEP_SUMMARY
           echo '- for details, see the `coverage-report` artifact' >> $GITHUB_STEP_SUMMARY
           echo '- to compare to the report from the `main` branch, see <https://jeffersonlab.github.io/iguana/coverage-report>' >> $GITHUB_STEP_SUMMARY
+      ### test relocatability
+      - name: test relocatability
+        run: |
+          mv iguana relocated
+          source relocated/bin/this_iguana.sh --verbose # do not use --githubCI option, since we want this environment to be for only this step
+          relocated/bin/iguana-example-basic test_data.hipo ${{ env.num_events }}
+          mv relocated iguana
       ### set iguana environment, since the next steps will check the iguana installation
       - name: set iguana environment
         run: source iguana/bin/this_iguana.sh --verbose --githubCI

README.md

@@ -16,6 +16,7 @@ Iguana is not a framework for _reading_ data, rather it is a set of algorithms t
 ### For Users
 1. [Setup Guide](doc/setup.md)
 1. [Examples](examples/README.md)
+1. [Configuring the Algorithms](doc/configuration.md)
 1. [Troubleshooting](doc/troubleshooting.md)
 1. [API documentation](https://jeffersonlab.github.io/iguana/doxygen)
 

doc/configuration.md

@@ -0,0 +1,58 @@
+# Configuring the Algorithms
+
+Most of the algorithms are configurable using a YAML configuration file. If you are using Iguana for your analysis, you likely want to customize the algorithm configurations.
+
+The default configuration is installed in the `etc/` subdirectory of the Iguana installation. If you have set the Iguana environment variables using, _e.g._ `source this_iguana.sh`, or if you are using the version of Iguana installed on `ifarm`, you will have the environment variable `$IGUANA_CONFIG_PATH` set to include this `etc/` directory.
+
+There are a few ways to configure the algorithms; see the sections below for the options
+
+> [!IMPORTANT]
+> While algorithm developers are encouraged _not_ to make breaking changes to their algorithms or configuration, in some cases certain changes cannot be prevented. Thus if you have your own algorithm configurations, you may want to keep up-to-date on any changes of the algorithm. We will try to announce all breaking changes in the Iguana release notes.
+
+> [!NOTE]
+> If the Iguana installation is relocated, the environment variable `$IGUANA_CONFIG_PATH` _must_ be used at runtime.
+
+## Option 1: Copy the default directory, and modify
+
+First, copy the default configuration directory to your work area, or to your analysis code source tree; we'll call the copied directory `my_iguana_config`, as an example. If `$IGUANA_CONFIG_PATH` is the default configuration directory (_i.e._ you have not set or modified this variable yourself), you may run:
+```bash
+cp -r $IGUANA_CONFIG_PATH my_iguana_config
+```
+
+You may then freely modify any configuration file within `my_iguana_config/`. To use this directory in your algorithms, you may do any one of the following:
+
+1. Since `$IGUANA_CONFIG_PATH` allows multiple paths, delimited by colons (`:`), prepend `my_iguana_config/` to `$IGUANA_CONFIG_PATH`; the safest way is to use an absolute path:
+```bash
+export IGUANA_CONFIG_PATH=`pwd`/my_iguana_config:$IGUANA_CONFIG_PATH   # bash or zsh
+setenv IGUANA_CONFIG_PATH `pwd`/my_iguana_config:$IGUANA_CONFIG_PATH   # tcsh or csh
+```
+The algorithms will then search `my_iguana_config` for the configuration before searching the default paths. You may add multiple paths, if needed.
+Paths which appear first in `$IGUANA_CONFIG_PATH` will be prioritized when the algorithm searches for configuration parameters; this behavior is similar to that of `$PATH` or `$LD_LIBRARY_PATH`.
+
+2. Use `Algorithm::SetConfigDirectory` instead of prepending `$IGUANA_CONFIG_PATH`.
+
+## Option 2: Write your own YAML file
+
+Make a new YAML file, containing just the options you want to override; use `Algorithm::SetConfigFile` to use this file for each algorithm.
+See existing algorithms' configuration, and just copy what you need into your own YAML file; be mindful of the indentation, and that the top-level set of YAML nodes are the algorithm names. For example:
+
+Algorithm default configuration files:
+```yaml
+### default configuration file 1
+physics::AlgorithmA
+  cuts: [-1, 1]
+```
+```yaml
+### default configuration file 2
+physics::AlgorithmB
+  value: 3
+```
+Custom YAML file, widening `AlgorithmA`'s `cuts` and increasing `AlgorithmB`'s `value`:
+```yaml
+### custom YAML file
+physics::AlgorithmA
+  cuts: [-2, 2]
+
+physics::AlgorithmB
+  value: 5
+```

doc/setup.md

@@ -162,4 +162,5 @@ The following environment variables are set or modified; not all of them are nee
 | `LD_LIBRARY_PATH` (Linux) or `DYLD_LIBRARY_PATH` (macOS) | adds paths to dependency and Iguana libraries                                                                                             |
 | `PYTHONPATH`                                             | adds paths to dependency and Iguana Python packages, if Python bindings are installed                                                     |
 | `ROOT_INCLUDE_PATH`                                      | adds paths to dependency and Iguana header files, for usage in ROOT                                                                       |
+| `IGUANA_CONFIG_PATH`                                     | path to iguana algorithm configuration files (`.yaml`); users may override this with their own path; multiple paths may be specified, delimited by colons (`:`), where paths listed first will override paths listed later (similar behavior as `$PATH`); this variable is _only_ necessary if the Iguana installation has been relocated |
 | `IGUANA`                                                 | the path to the Iguana installation prefix, equivalent to `pkg-config iguana --variable prefix`; this is only for consumers that do not use `pkg-config` or the other standard environment variables, however usage of this variable is _discouraged_ since the installation layout may vary |

examples/iguana-example-basic.cc

@@ -57,7 +57,7 @@ int main(int argc, char** argv)
   seq.SetOption("clas12::EventBuilderFilter", "log", "debug");
   seq.SetOption("clas12::MomentumCorrection", "log", "debug");
 
-  // set algorithm options
+  // set algorithm options (overrides configuration files)
   seq.SetOption<std::vector<int>>("clas12::EventBuilderFilter", "pids", {11, 211, -211});
 
   // start the algorithms

meson.build

@@ -171,7 +171,7 @@ project_test_env.set(
 
 # set preprocessor macros
 add_project_arguments(
-  '-DIGUANA_ETC="' + get_option('prefix') / project_etc + '"',
+  '-DIGUANA_ETCDIR="' + get_option('prefix') / project_etc + '"',
   language: ['cpp'],
 )
 
@@ -218,6 +218,7 @@ if get_option('z_install_envfile')
       'python':  get_option('bind_python') ? 'true' : 'false',
       'libdir':  get_option('libdir'),
       'root':    ROOT_dep.found() ? 'true' : 'false',
+      'etcdir':  project_etc,
     },
   )
   foreach shell : ['csh', 'tcsh']

meson/this_iguana.sh.in

@@ -82,6 +82,9 @@ if @root@; then
   unset var
 fi
 
+# set config file path; append, rather than prepend, in case the user already has their own
+export IGUANA_CONFIG_PATH=${IGUANA_CONFIG_PATH:+${IGUANA_CONFIG_PATH}:}$IGUANA/@etcdir@
+
 # print environment variables
 if $arg_verbose; then
   print_var() { echo "$@" | sed 's/:/\n  /g'; }
@@ -107,6 +110,9 @@ ROOT_INCLUDE_PATH -- $(if @root@; then echo 'ADDED iguana and dependency include
 PYTHONPATH -- $(if @python@; then echo 'ADDED iguana python bindings:'; else echo 'has NOT been changed'; fi)
   $(print_var ${PYTHONPATH-})
 
+IGUANA_CONFIG_PATH -- ADDED iguana algorithms' default configuration directory
+  $(print_var ${IGUANA_CONFIG_PATH-})
+
 ----------------------------
 """
   unset -f print_var
@@ -117,6 +123,7 @@ if $arg_githubCI; then
   echo PKG_CONFIG_PATH=$PKG_CONFIG_PATH >> $GITHUB_ENV
   echo PATH=$PATH >> $GITHUB_ENV
   echo @ld_path@=$@ld_path@ >> $GITHUB_ENV
+  echo IGUANA_CONFIG_PATH=$IGUANA_CONFIG_PATH >> $GITHUB_ENV
   if @python@; then
     echo PYTHONPATH=$PYTHONPATH >> $GITHUB_ENV
   fi

src/iguana/services/ConfigFileReader.cc

@@ -1,18 +1,39 @@
 #include "ConfigFileReader.h"
 #include <filesystem>
+#include <cstdlib>
+#include <sstream>
 
 namespace iguana {
 
   ConfigFileReader::ConfigFileReader(std::string_view name)
       : Object(name)
   {
-    // add config files from installation prefix
+    // the algorithms need to know where the config files are;
+    // first, add config files from installation prefix; since this
+    // is added first, it's the lowest priority in the configuration
+    // search path
     AddDirectory(GetConfigInstallationPrefix());
+    // next, add `IGUANA_CONFIG_PATH` paths, which provides:
+    // - user override of the configuration path(s)
+    // - a fallback, if `GetConfigInstallationPrefix` is wrong, which happens
+    //   if the Iguana installation is relocated
+    auto user_paths_env_var = std::getenv("IGUANA_CONFIG_PATH");
+    if(user_paths_env_var != nullptr) {
+      std::istringstream user_paths_stream(user_paths_env_var);
+      std::string user_path_token;
+      decltype(m_directories) user_paths;
+      while(getline(user_paths_stream, user_path_token, ':')) // tokenize `IGUANA_CONFIG_PATH`
+        user_paths.push_front(user_path_token);
+      for(auto const& user_path : user_paths) // add the paths in the correct order
+        AddDirectory(user_path);
+      // after these default paths have been added, the user
+      // may still override by calling `AddDirectory` etc. themselves
+    }
   }
 
   std::string ConfigFileReader::GetConfigInstallationPrefix()
   {
-    return IGUANA_ETC;
+    return IGUANA_ETCDIR;
   }
 
   void ConfigFileReader::AddDirectory(std::string const& dir)

src/iguana/services/ConfigFileReader.h

@@ -15,6 +15,9 @@ namespace iguana {
       ConfigFileReader(std::string_view name = "config");
 
       /// Get the config files' _fixed_ installation prefix
+      /// @warning if the Iguana installation is _relocated_, this directory will **not** be correct,
+      /// since it is compiled in the shared library; as a fallback, you may use the environment variable
+      /// `$IGUANA_CONFIG_PATH`.
       /// @return the absolute path to the installed configuration file directory
       static std::string GetConfigInstallationPrefix();