[ENH] Increase configurability of API/graph server parameters (#146)

* Make graph + API server parameters configurable as env vars - host + container ports - graph server Docker image - graph root paths on host + container - full graph database query endpoint * update table of env vars in README * update README instructions for using .env * increase timeout for query to 30s * expand notes about default env var values in README * add note to Dockerfile about env var for API port * add template .env --------- Co-authored-by: Sebastian Urchs <[email protected]>
neurobagel · Aug 21, 2023 · 24e6322 · 24e6322
1 parent 0b805c7
commit 24e6322
Show file tree

Hide file tree

Showing 6 changed files with 93 additions and 50 deletions.
diff --git a/.template-env b/.template-env
@@ -0,0 +1,15 @@
+# THIS IS A TEMPLATE .env FILE. MAKE A COPY OF THIS FILE NAMED .env AND MODIFY VALUES AS NEEDED.
+NB_GRAPH_USERNAME=DBUSER  # REPLACE DBUSER WITH YOUR GRAPH DATABASE USERNAME
+NB_GRAPH_PASSWORD=DBPASSWORD  # REPLACE DBPASSWORD WITH YOUR GRAPH DATABASE PASSWORD
+NB_GRAPH_DB=test_data/query
+NB_RETURN_AGG=true
+NB_API_TAG=latest
+NB_GRAPH_IMG=stardog/stardog:8.2.2-java11-preview
+## ADDITIONAL CONFIGURABLE PARAMETERS: Uncomment and modify values of the below variables as needed to use non-default values.
+# NB_API_PORT_HOST=8000
+# NB_API_PORT=8000
+# NB_API_ROOT_HOST=~/stardog-home
+# NB_GRAPH_ROOT_CONT=/var/opt/stardog
+# NB_GRAPH_ADDRESS=graph
+# NB_GRAPH_PORT_HOST=5820
+# NB_GRAPH_PORT=5820
diff --git a/Dockerfile b/Dockerfile
@@ -8,6 +8,7 @@ RUN pip install --no-cache-dir --upgrade -r /usr/src/app/requirements.txt
 
 COPY ./app /usr/src/app
 
-ENTRYPOINT ["uvicorn", "app.main:app", "--proxy-headers", "--host", "0.0.0.0"]
-
-CMD ["--port", "8000"]
+# NB_API_PORT, representing the port on which the API will be exposed, 
+# is an environment variable that will always have a default value of 8000 when building the image
+# but can be overridden when running the container.
+ENTRYPOINT uvicorn app.main:app --proxy-headers --host 0.0.0.0 --port ${NB_API_PORT:-8000}
diff --git a/README.md b/README.md
@@ -23,6 +23,7 @@ The Neurobagel API is a REST API, developed in [Python](https://www.python.org/)
 
 - [Quickstart](#quickstart)
 - [Local installation](#local-installation)
+    - [Environment variables](#set-the-environment-variables)
     - [Docker](#docker)
     - [Python](#python)
 - [Testing](#testing)
@@ -36,79 +37,95 @@ Example: **I want to query for only female participants in the graph.** The URL
 
 Interactive documentation for the API is available at https://api.neurobagel.org/docs.
 
-NOTE: Currently, to access the API, you must be connected to the McGill network.
-
 ## Local installation
-The below instructions assume that you have a local instance of or access to a remotely hosted graph database to be queried. If this is not the case and you need to first build a graph from data, refer to the instructions for getting started locally with [Stardog Studio](https://docs.stardog.com/stardog-applications/dockerized_access#stardog-studio).
+The below instructions assume that you have a local instance of or access to a remotely hosted graph database to be queried. 
+If this is not the case and you need to first build a graph from data, refer to our documentation for [getting started locally with a graph backend](https://neurobagel.org/infrastructure/).
 
 ### Clone the repo
 ```bash
 git clone https://github.com/neurobagel/api.git
 ```
 
 ### Set the environment variables
-Create a `.env` file in the root of the repository to house the environment variables used by the app. 
+Create a file called `.env` in the root of the repository will house the environment variables used by the app. 
 
 To run API requests against a graph, at least two environment variables must be set, `NB_GRAPH_USERNAME` and `NB_GRAPH_PASSWORD`.  
-The contents of a minimal `.env` file:
-```bash
-NB_GRAPH_USERNAME=someuser
-NB_GRAPH_PASSWORD=somepassword
-```
-
-Below are all the possible Neurobagel environment variables that you can set in `.env`, depending on your mode of installation of the API.
-| Environment variable | Required in .env? | Description                                                                                                                              | Default value            | Relevant installation mode |
-|----------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------------------------|
-| `NB_GRAPH_USERNAME`  | Yes               | Username to access Stardog graph database that API will communicate with                                                                 | -                        | Docker, Python             |
-| `NB_GRAPH_PASSWORD`  | Yes               | Password to access Stardog graph database that API will communicate with                                                                 | -                        | Docker, Python             |
-| `NB_GRAPH_ADDRESS`   | No                | IP address for the graph database (or container name, if graph is hosted locally)                                                        | `206.12.99.17` (`graph`) | Docker, Python             |
-| `NB_GRAPH_DB`        | No                | Name of graph database to query                                                                                                          | `test_data`              | Docker, Python             |
-| `NB_RETURN_AGG`      | No                | Whether to return only dataset-level query results (including data locations) and exclude subject-level attributes. One of [true, false] | `true`                   | Docker, Python             |
-| `NB_API_TAG` | No | Tag for API Docker image | `latest` | Docker
-| `STARDOG_TAG` | No | Tag for Stardog Docker image | `8.2.2-java11-preview` | Docker |
-| `STARDOG_ROOT` | No | Path to directory on host machine containing a Stardog license file | `~/stardog-home` | Docker |
-
-NOTE: To avoid conflicts related to [Docker's environment variable precedence](https://docs.docker.com/compose/environment-variables/envvars-precedence/), ensure that any variables defined in your `.env` file are not already set in your current shell environment with **different** values.
-
-To export all the variables defined in your `.env` file in one step, run the following:
-```bash
-export $(cat .env | xargs)
-```
 
-The below instructions for Docker and Python assume that you have at least set `NB_GRAPH_USERNAME` and `NB_GRAPH_PASSWORD` in your current environment.
+This repository contains a [template `.env` file](/.template-env) that you can copy and edit.
+
+Below are explanations of all the possible Neurobagel environment variables that you can set in `.env`, depending on your mode of installation of the API and graph server software.
+| Environment variable | Required in .env? | Description                                                                                                                              | Default value if not set               | Relevant installation mode(s) |
+| -------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ----------------------------- |
+| `NB_GRAPH_USERNAME`  | Yes               | Username to access Stardog graph database that API will communicate with                                                                 | -                                      | Docker, Python                |
+| `NB_GRAPH_PASSWORD`  | Yes               | Password to access Stardog graph database that API will communicate with                                                                 | -                                      | Docker, Python                |
+| `NB_GRAPH_ADDRESS`   | No                | IP address for the graph database (or container name, if graph is hosted locally)                                                        | `206.12.99.17` (`graph`) **               | Docker, Python                |
+| `NB_GRAPH_DB`        | No                | Name of graph database endpoint to query (e.g., for a Stardog database, this will take the format of `{database_name}/query`)            | `test_data/query`                      | Docker, Python                |
+| `NB_RETURN_AGG`      | No                | Whether to return only dataset-level query results (including data locations) and exclude subject-level attributes. One of [true, false] | `true`                                 | Docker, Python                |
+| `NB_API_TAG`         | No                | Tag for API Docker image                                                                                                                 | `latest`                               | Docker                        |
+| `NB_API_PORT_HOST`   | No                | Port number on the _host machine_ to map the API container port to                                                                       | `8000`                                 | Docker                        |
+| `NB_API_PORT`        | No                | Port number on which to run the API                                                                                                      | `8000`                                 | Docker, Python                |
+| `NB_GRAPH_IMG`       | No                | Graph server Docker image                                                                                                                | `stardog/stardog:8.2.2-java11-preview` | Docker                        |
+| `NB_GRAPH_ROOT_HOST` | No                | Path to directory containing a Stardog license file on the _host machine_                                                                | `~/stardog-home`                       | Docker                        |
+| `NB_GRAPH_ROOT_CONT` | No                | Path to directory for graph databases in the _graph server container_                                                                    | `/var/opt/stardog` *                   | Docker                        |
+| `NB_GRAPH_PORT_HOST` | No                | Port number on the _host machine_ to map the graph server container port to                                                              | `5820`                                 | Docker                        |
+| `NB_GRAPH_PORT`      | No                | Port number used by the _graph server container_                                                                                         | `5820` *                               | Docker, Python                |
+
+_* These defaults are configured for a Stardog backend - you should not have to change them if you are running a Stardog backend._
+
+_** If using the [docker compose installation route](#option-1-recommended-use-the-docker-composeyaml-file), 
+do not change `NB_API_ADDRESS` from its default value (`graph`) as this corresponds to the preset container name of the graph database server within the docker compose network._
+
+---
+**IMPORTANT:** 
+- Variables set in the shell environment where the API is launched **_should not be used as a replacement for the `.env` file_** to configure options for the API or graph server software.
+- To avoid conflicts related to [Docker's environment variable precedence](https://docs.docker.com/compose/environment-variables/envvars-precedence/), 
+also ensure that any variables defined in your `.env` are not already set in your current shell environment with **different** values.
+---
+
+The below instructions for Docker and Python assume that you have at least set `NB_GRAPH_USERNAME` and `NB_GRAPH_PASSWORD` in your `.env`.
 
 ### Docker
 First, [install docker](https://docs.docker.com/get-docker/).
 
 You can then run a Docker container for the API in one of three ways:
-#### Option 1: Use the `docker-compose.yaml` file
+#### Option 1 (RECOMMENDED): Use the `docker-compose.yaml` file
 
 First, [install docker compose](https://docs.docker.com/compose/install/).
 
 If needed, update your `.env` file with optional environment variables for the docker compose configuration.
 
+**TIP:** Double check that the environment variables are resolved with your expected values using the command `docker compose config`.
+
 Use Docker Compose to spin up the containers by running the following in the repository root (where the `docker-compose.yml` file is):
 ```bash
 docker compose up -d
 ```
 
 #### Option 2: Use the latest image from Docker Hub
 ```bash
-export $(cat .env | xargs)  # export your environment variables 
 docker pull neurobagel/api
-docker run -d --name api -p 8000:8000 --env NB_GRAPH_USERNAME --env NB_GRAPH_PASSWORD neurobagel/api
+docker run -d --name=api -p 8000:8000 --env-file=.env neurobagel/api
+```
+**NOTE:** The above `docker run` command uses recommended default values for `NBI_API_PORT_HOST` and `NB_API_PORT` in the `-p` flag.
+If you wish to set different port numbers, modify your `.env` file accordingly and run the below commands instead:
+```bash
+export $(cat .env | xargs)  # export your .env file to expose your set port numbers to the -p flag of docker run
+docker run -d --name=api -p ${NB_API_PORT_HOST}:${NB_API_PORT} --env-file=.env neurobagel/api
 ```
+
 #### Option 3: Build the image using the Dockerfile
 After cloning the current repository, build the Docker image locally:
 ```bash
-export $(cat .env | xargs)  # export your environment variables
-docker build -t <image_name> .
-docker run -d --name api -p 8000:8000 --env NB_GRAPH_USERNAME --env NB_GRAPH_PASSWORD neurobagel/api
+docker build -t neurobagel/api .
+docker run -d --name=api -p 8000:8000 --env-file=.env neurobagel/api
+```
+**NOTE:** The above `docker run` command uses recommended default values for `NBI_API_PORT_HOST` and `NB_API_PORT` in the `-p` flag. 
+If you wish to set different port numbers, modify your `.env` file accordingly and run the below commands instead:
+```bash
+docker build -t neurobagel/api .
+export $(cat .env | xargs)  # export your .env file to expose your set port numbers to the -p flag of docker run
+docker run -d --name=api -p ${NB_API_PORT_HOST}:${NB_API_PORT} --env-file=.env neurobagel/api
 ```
-
-For Options 2 or 3, if you wish to also set `NB_GRAPH_ADDRESS`, make sure to pass it to the container in the `docker run` command using the `--env` flag.
-
-NOTE: In case you're connecting to the McGill network via VPN and you started the container before connecting to the VPN, make sure to configure your VPN client to allow local (LAN) access when using the VPN.
 
 #### Send a test query to the API
 By default, after running the above steps, the API should be served at localhost, http://127.0.0.1:8000/query, on the machine where you launched the Dockerized app. To check that the API is running and can access the knowledge graph as expected, you can navigate to the interactive API docs in your local browser (http://127.0.0.1:8000/docs) and enter a sample query, or send an HTTP request in your terminal using `curl`:
@@ -131,6 +148,11 @@ $ pip install -r requirements.txt
 
 To launch the API make sure you're in repository's main directory and in your environment where the dependencies are installed and environment variables are set.
 
+Export the variables defined in your `.env` file:
+```bash
+export $(cat .env | xargs)
+```
+
 You can then launch the API by running the following command in your terminal:
 
 ```bash

diff --git a/app/api/crud.py b/app/api/crud.py
@@ -85,6 +85,8 @@ async def get(
                 os.environ.get(util.GRAPH_USERNAME.name),
                 os.environ.get(util.GRAPH_PASSWORD.name),
             ),
+            # TODO: Revisit timeout value when query performance is improved
+            timeout=30.0,
         )
     except httpx.ConnectTimeout as exc:
         raise HTTPException(

diff --git a/app/api/utility.py b/app/api/utility.py
@@ -15,16 +15,18 @@
 GRAPH_ADDRESS = EnvVar(
     "NB_GRAPH_ADDRESS", os.environ.get("NB_GRAPH_ADDRESS", "206.12.99.17")
 )
-GRAPH_DB = EnvVar("NB_GRAPH_DB", os.environ.get("NB_GRAPH_DB", "test_data"))
+GRAPH_DB = EnvVar(
+    "NB_GRAPH_DB", os.environ.get("NB_GRAPH_DB", "test_data/query")
+)
+GRAPH_PORT = EnvVar("NB_GRAPH_PORT", os.environ.get("NB_GRAPH_PORT", 5820))
 # TODO: Environment variables can't be parsed as bool so this is a workaround but isn't ideal.
 # Another option is to switch this to a command-line argument, but that would require changing the
 # Dockerfile also since Uvicorn can't accept custom command-line args.
 RETURN_AGG = EnvVar(
     "NB_RETURN_AGG", os.environ.get("NB_RETURN_AGG", "True").lower() == "true"
 )
 
-GRAPH_PORT = 5820
-QUERY_URL = f"http://{GRAPH_ADDRESS.val}:{GRAPH_PORT}/{GRAPH_DB.val}/query"
+QUERY_URL = f"http://{GRAPH_ADDRESS.val}:{GRAPH_PORT.val}/{GRAPH_DB.val}"
 QUERY_HEADER = {
     "Content-Type": "application/sparql-query",
     "Accept": "application/sparql-results+json",

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -4,16 +4,17 @@ services:
   api:
     image: "neurobagel/api:${NB_API_TAG:-latest}"
     ports:
-      - "8000:8000"
+      - "${NB_API_PORT_HOST:-8000}:${NB_API_PORT:-8000}"
     environment:
       NB_GRAPH_USERNAME: ${NB_GRAPH_USERNAME}
       NB_GRAPH_PASSWORD: ${NB_GRAPH_PASSWORD}
       NB_GRAPH_ADDRESS: ${NB_GRAPH_ADDRESS:-graph}
-      NB_GRAPH_DB: ${NB_GRAPH_DB:-test_data}
+      NB_GRAPH_DB: ${NB_GRAPH_DB:-test_data/query}
       NB_RETURN_AGG: ${NB_RETURN_AGG:-true}
+      NB_API_PORT: ${NB_API_PORT:-8000}
   graph:
-    image: "stardog/stardog:${STARDOG_TAG:-8.2.2-java11-preview}"
+    image: "${NB_GRAPH_IMG:-stardog/stardog:8.2.2-java11-preview}"
     volumes:
-      - "${STARDOG_ROOT:-~/stardog-home}:/var/opt/stardog"
+      - "${NB_GRAPH_ROOT_HOST:-~/stardog-home}:${NB_GRAPH_ROOT_CONT:-/var/opt/stardog}"
     ports:
-      - "5820:5820"
+      - "${NB_GRAPH_PORT_HOST:-5820}:${NB_GRAPH_PORT:-5820}"