Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Germinate connection #435

Merged
merged 10 commits into from
Nov 27, 2024
86 changes: 86 additions & 0 deletions doc/adminGuides/docker-compose.prod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# NOTE refer to the accompanying '.env' file in this folder to access
# environment variables which are passed through to docker-compose.yaml
# at run time

# NOTE this has been updated to the docker compose V2 format, this will not work with docker-compose

name: pretzel-prod

networks:
pretzel-prod:
driver: bridge

services:
database: # mongo database
# MongoDb up to v5 has been tested OK
image: 9b5c4a4fdcb5 # 4.2.24. library/mongo-4.2.24 # base image off dockerhub
# environment:
# - "MONGO_INITDB_ROOT_USERNAME=${DB_USER}"
# - "MONGO_INITDB_ROOT_PASSWORD=${DB_PASS}"
volumes:
- ${DATA_DIR}:/data/db
expose:
- "${MONGO_DEFAULT_PORT}"
networks:
- pretzel-prod

api: # node environment
depends_on:
- database
# Could have depends_on: blastserver, but it is not a critical dependency
build:
context: .
dockerfile: ./scripts/Dockerfile
image: plantinformaticscollaboration/pretzel:v3.1.0
command: node /app/lb3app/server/server.js
environment:
- "API_HOST=${API_HOST}"
- "API_PORT_EXT=${API_PORT_EXT}"
- "API_PORT_PROXY=${API_PORT_PROXY}"
- "hostIp=${hostIp}"
# Flask port is now internal to the compose network, so use fixed (default) 4000;
# no need to configure via FLASK_PORT / BLASTSERVER_PORT.
- "FLASK_PORT=4000" # ${BLASTSERVER_PORT}"
- "DB_HOST=database"
- "DB_PORT=${MONGO_DEFAULT_PORT}"
- "DB_NAME=${DB_NAME}"
- "DB_USER=${DB_USER}"
- "DB_PASS=${DB_PASS}"
- "EMAIL_HOST=${EMAIL_HOST}"
- "EMAIL_PORT=${EMAIL_PORT}"
- "EMAIL_USER=${EMAIL_USER}"
- "EMAIL_PASS=${EMAIL_PASS}"
- "EMAIL_FROM=${EMAIL_FROM}"
- "EMAIL_VERIFY=${EMAIL_VERIFY}"
- "EMAIL_ADMIN=${EMAIL_ADMIN}"
- "mntData=${mntData}"
- "handsOnTableLicenseKey=${handsOnTableLicenseKey}"
volumes:
# landingPage
- $landingPage:/app/client/landingPageContent
# blastVolume
- $mntData/blast:$mntData/blast
# vcfVolume
- $mntData/vcf:$mntData/vcf
- ${resultsCacheDir}:/app/node_modules/flat-cache/.cache

ports:
# match ext / int ports for loopback
- "${API_PORT_EXT}:${API_PORT_EXT}"
networks:
- pretzel-prod

blastserver: # Python Flask blastn server, based on a python image, used for DNA Sequence Search
image: plantinformaticscollaboration/blastserver:latest
environment:
- "FLASK_PORT=${BLASTSERVER_PORT}"
volumes:
# mntData=/mnt/data_blast
- $mntData/blast:/mnt/data/blast
# Enables scripts/blastn_cont.bash to run blastn via docker
- /usr/bin/docker:/usr/bin/docker
- /var/run/docker.sock:/var/run/docker.sock
expose:
- "4000"
networks:
- pretzel-prod
142 changes: 142 additions & 0 deletions doc/adminGuides/install_and_configure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
### Audience

This guide is for system administrators tasked with installation and configuration of Pretzel, and administration of software systems and data.

This documentation will guide them through configuration of docker compose yaml files and related environment files, and provide templates of these as a starting point.

There are separate documentation resources for other audiences :
- User Documentation : https://docs.plantinformatics.io
- Development team : pretzel/doc/notes/

---

### Background

The method of installation of Pretzel has evolved along with the application.
Our early recommendations were to use individual docker containers for the mongoDb database and Pretzel API server, but with the addition of the blastserver (Python Flask micro-service), and access to additional external databases (Blast and VCF), we have moved to docker compose to package the configuration and connection of these components.

### Docker Compose elements

In the same directory as this document there is a sample docker compose yaml file, and sample environment files.

The following section discusses configurations the system administrator may make to these files.

### Configuration of docker compose .yaml

The most important configuration is the version of the Pretzel API server :
```
api: # node environment
...
image: plantinformaticscollaboration/pretzel:v3.1.0
```

These configurations support optional features :

```
api: # node environment
...
volumes:
# landingPage
- $landingPage:/app/client/landingPageContent
# blastVolume
- $mntData/blast:$mntData/blast
# vcfVolume
- $mntData/vcf:$mntData/vcf
- ${resultsCacheDir}:/app/node_modules/flat-cache/.cache
```
#### landingPage
A directory which contains index.html and other html and image files referenced by index.html. If provided, this content is displayed in the Pretzel home page before the user logs in.

#### blastVolume
A directory containing blastn databases.
If mntData is provided, then the default value of blastDir is $mntData/blast.

#### vcfVolume
A directory containing VCF genotype data.
If mntData is provided, then the default value of vcfDir is $mntData/vcf.

#### resultsCacheDir
The Pretzel API server will cache results for common requests, enabling them to be served more quickly and efficiently. This directory will be created within the Pretzel API server container, or if it is configured as a shared volume, then the resultsCache will be stored in the directory passed in. The benefit of this is that when the container is re-created, the resultsCache will be preserved. This is not essential, but will improve performance in particular for histograms where the chromosomes (Blocks) contain 1e5 - 1e7 features.

---

### Configuration of pretzel.compose.prod.env

The template .env file has these variables commented out; you will need to provide values for them :
- DATA_DIR
- mntData
- API_HOST
- EMAIL_*


---

Configuration for compose down & up :
```
logDate=$(date +%Y%b%d)
echo $logDate
# Location of docker compose templates
export Dc=~/pretzel/doc/adminGuides
# Use other values if you are running multiple Pretzel servers, e.g. stage=dev
# You can simplify the naming by omitting $stage if only running a single server.
stage=prod
L=~/log/compose/$stage/$logDate
```

Docker compose down and up :
```
docker compose --progress=plain --file $Dc/docker-compose.$stage.yaml --env-file $Dc/pretzel.compose.$stage.env down
nohup docker compose --progress=plain --file $Dc/docker-compose.$stage.yaml --env-file $Dc/pretzel.compose.$stage.env up > $L &
```

The above log file `$L` will contain the output from the 3 containers, api database and blastserver. To record the logs into separate log files the following can be used. This would be done before replacing them; i.e. before compose down.
```
for i in api database blastserver; do \
docker logs pretzel-$stage-$i-1 >& ~/log/compose/$stage/$i.$logDate; done
```

---

### Preparing a custom Pretzel API server container image

This section describes how to build a Pretzel API server image from a particular version or branch, e.g. if a hot-fix branch has been provided.

Image build
```
# cd '<build_directory>'
git clone https://github.com/plantinformatics/pretzel.git
cd pretzel

export GIT_PAGER=cat
# replace 'master' here with the name of the branch or version to build
git checkout master

logDate=$(date +%Y%b%d)
echo $logDate
export DOCKER_BUILDKIT=1

app=pretzel
# you can use some other label name; this example uses the build date.
export PRETZEL_VERSION=v$logDate
# e.g. export PRETZEL_VERSION=v3.1.0
image=$app:$PRETZEL_VERSION

mkdir -p ~/log/build/docker
time nohup sudo docker build -t $image . > ~/log/build/docker/$logDate &
```

Observe the image tag name; this will be used in docker compose .yaml : api : image.
```
docker image inspect $image | jq '.[] | .RepoTags'
```

Tag the built image for push to dockerhub
```
# If pushing the result to dockerhub :
# baseName='<dockerhub_organisation_name>'/$app
docker tag $image $baseName:$PRETZEL_VERSION
docker tag $image $baseName:latest
docker image inspect $image | jq '.[] | .RepoTags'
```

---
32 changes: 32 additions & 0 deletions doc/adminGuides/pretzel.compose.prod.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Prod

# DATA_DIR= # directory for mongoDb database
# mntData= # directory for Blast and VCF databases
# landingPage= # directory containing index.html and web page content to display on the home page before the user logs in.
# This dir maps to /app/node_modules/flat-cache/.cache and contains 1 file : resultsCache
resultsCacheDir=/home/ec2-user/home/resultsCache/prod
#PORT=3010

# Not used ?
# INSTANCE=agg
DB_NAME=pretzel
# API_HOST=agg.plantinformatics.io
API_PORT_PROXY=80
API_PORT_EXT=3010
hostIp=blastserver
# Flask port is now internal to the compose network, so use fixed (default) 4000;
# no need to configure via FLASK_PORT / BLASTSERVER_PORT.
BLASTSERVER_PORT=4000
# The value of API_PORT_PROXY is not (currently) a port, it is just defined or undefined.
MONGO_DEFAULT_PORT=27017
EMAIL_VERIFY=ADMIN
# [email protected]
# EMAIL_HOST=email-smtp.<region>.amazonaws.com
EMAIL_PORT=25
# EMAIL_PASS=...
# EMAIL_USER=...
# [email protected]
# This enables using Feature.value_0 as an index field; all datbases now contain this field and this option can be made to default to true (1).
use_value_0=1
# If a handsOnTableLicenseKey is not provided, a prompt message will be displayed in the GUI. Insert your license here; Hands On Table allows non-commercial and evaluation use.
# handsOnTableLicenseKey=non-commercial-and-evaluation
2 changes: 1 addition & 1 deletion frontend/app/components/draw/axis-1d.js
Original file line number Diff line number Diff line change
Expand Up @@ -1441,7 +1441,7 @@ export default Component.extend(Evented, AxisEvents, AxisPosition, {
as.classed("extended", this.get('extended'));
},
buttonStateEffect : computed('brushed', 'zoomed', function () {
later(() => this.isDestroying && this.showZoomResetButtonState(), 200);
later(() => ! this.isDestroying && this.showZoomResetButtonState(), 200);
}),
showZoomResetButtonState() {
let
Expand Down
2 changes: 1 addition & 1 deletion frontend/app/components/form/dataset-graph.hbs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

<BsButton @type="info" class="margin-top" @onClick={{@close}}>Close</BsButton>

{{input type="search" value=naturalQuery
{{input type="search" value=this.naturalQuery
enter=(action this.naturalQueryChanged)
placeholder="Natural language search" }}

Expand Down
6 changes: 1 addition & 5 deletions frontend/app/components/panel/manage-genotype.hbs
Original file line number Diff line number Diff line change
Expand Up @@ -838,12 +838,8 @@
{{!-- ----------------------------------------------------------------- --}}

{{#if this.vcfGenotypeLookupDomain}}
{{!-- disabled if ! All && no samples selected, or All && Common && no samples to select --}}
<button type="button" onclick={{action this.vcfGenotypeLookup}}
disabled={{or
(and (not @userSettings.requestSamplesAll) (not this.vcfGenotypeSamplesSelected.length))
(and @userSettings.requestSamplesAll @userSettings.samplesIntersection (not this.samples.length))
}}
disabled={{this.vcfGenotypeLookupButtonDisabled}}
class="btn btn-info">VCF Lookup</button>
{{else}}
{{#if this.urlOptions.gtIntervalLimit}}
Expand Down
Loading
Loading