diff --git a/docs/README.md b/docs/README.md index 29d778cc2..a2694fc2c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,4 +15,4 @@ We welcome contributions to our documentation! Please see the [Contributing](con ## License -This project is licensed under the [LICENSE](../LICENSE) file in the root directory of this source tree. \ No newline at end of file +This project is licensed under the [LICENSE](../LICENSE) file in the root directory of this source tree. diff --git a/docs/changelog.md b/docs/changelog.md index bd07fda3a..49c4487de 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -8,4 +8,4 @@ The changelog documents all notable changes to the project. This file is updated ## [Version] - [Release Date] -- List changes for each version. \ No newline at end of file +- List changes for each version. diff --git a/docs/contributing.md b/docs/contributing.md index d63eecf62..3dc7bd0d5 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -3,9 +3,10 @@ We love contributions from the community! This document explains how you can contribute to the project. ## How to Contribute -#### Generate and install the python installable API -After making any changes to the logic, it is worth regenerating the API with the OpenAPI Generator. +### Generate and install the python installable API + +After making any changes to the logic, it is worth regenerating the API with the OpenAPI Generator. Generating the installable APIs (Python + Typescript) involves running the server, getting the `/openapi.json`, and running `openapi-generator`. @@ -16,6 +17,7 @@ The `regenerate_api.py` script does this for us in a few ways: 3. Spins up the server itself. You can simply run: + ```bash # this will start the api.server, so make sure you have the dependencies installed, python regenerate_api.py \ @@ -23,6 +25,7 @@ python regenerate_api.py \ ``` or if you prefer the Docker approach (eg: for CI), this command will build the docker container and supply it to `regenerate_api.py`: + ```bash # SM_DOCKER is a known env variable to regenerate_api.py export SM_DOCKER="cpg/metamist-server:dev" @@ -30,7 +33,8 @@ docker build --build-arg SM_ENVIRONMENT=local -t $SM_DOCKER -f deploy/api/Docker python regenerate_api.py ``` -#### Developing the UI +### Developing the UI + ```bash # Ensure you have started metamist server locally on your computer already, then in another tab open the UI. # This will automatically proxy request to the server. @@ -39,6 +43,7 @@ npm install npm run compile npm start ``` + This will start a web server using Vite, running on `localhost:5173`. ### OpenAPI and Swagger @@ -70,4 +75,4 @@ The deploy github action builds the container, and is deployed. Additionally you can access metamist through the identity-aware proxy (IAP), which handles the authentication through OAuth, allowing you to access the -front-end. \ No newline at end of file +front-end. diff --git a/docs/installation.md b/docs/installation.md index e594933c4..fcb47f16a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -3,12 +3,10 @@ This document provides detailed instructions on how to install the project. Follow these steps to set up the project on your local system. ## Prerequisites -#### Mac -[Homebrew](https://brew.sh) is the simplest way to install system dependencies needed for this project. +[Homebrew](https://brew.sh) is the simplest way to install system dependencies needed for this project. -#### Windows -[Chocolatey](https://community.chocolatey.org/) is a good equivalent to Homebrew for package management in Windows. +[Chocolatey](https://community.chocolatey.org/) is a good equivalent to Homebrew for package management in Windows. ## System Requirements @@ -21,7 +19,8 @@ This document provides detailed instructions on how to install the project. Foll - **pyenv** - **wget** *(optional)* -#### Mac +### Mac + ```bash brew install nvm brew install java @@ -30,11 +29,12 @@ brew install pyenv brew install wget # skip if you wish to install via docker -brew install mariadb@10.8 +brew install mariadb@10.8 ``` -#### Windows +### Windows + ```bash # Assuming you have Chocolatey choco install nvm @@ -45,7 +45,6 @@ choco install wget # skip if you wish to install via docker choco install mariadb --version=10.8.3 - ``` ```bash @@ -55,19 +54,20 @@ nvm install --lts # Once you have npm set up, install the OpenAPI generator: npm install @openapitools/openapi-generator-cli -g openapi-generator-cli version-manager set 5.3.0 - ``` ## Installation Steps -#### Creating the environment +### Creating the environment + - Python dependencies for the `metamist` API package are listed in `setup.py`. - Additional dev requirements are listed in `requirements-dev.txt`. - Packages for the sever-side code are listed in `requirements.txt`. -We _STRONGLY_ encourage the use of `pyenv` for managing Python versions. Debugging and the server will run on a minimum python version of 3.10. +We *STRONGLY* encourage the use of `pyenv` for managing Python versions. Debugging and the server will run on a minimum python version of 3.10. Use of a virtual environment to contain all requirements is highly recommended: + ```bash virtualenv venv source venv/bin/activate @@ -79,6 +79,7 @@ pip install --editable . ``` You will also need to set the following environment variables. Adjust the paths if you installed the dependencies using an alternate means: + ```bash # homebrew should export this on an M1 Mac # the intel default is /usr/local @@ -112,7 +113,7 @@ export SM_DEV_DB_USER=sm_api # makes it easier to copy liquibase update command You can also add these to your shell config file e.g `.zshrc` or `.bashrc` for persistence to new sessions. -#### Database Setup - Native Installation +### Database Setup - Native Installation Set the following environment variables: @@ -124,7 +125,7 @@ export SM_DEV_DB_PORT=3306 # default mariadb port export SM_DEV_DB_NAME=sm_dev; ``` -Next, create the database `sm_dev` in MariaDB. +Next, create the database `sm_dev` in MariaDB. > In newer versions of MariaDB, the root user is protected. @@ -159,21 +160,24 @@ liquibase \ popd ``` -#### Database Setup - Docker Installation +### Database Setup - Docker Installation Ensure you have Docker installed or follow [this guide](https://docs.docker.com/engine/install/) to setup. Pull the image: + ```bash docker pull mariadb:10.8.3 ``` Run the container on port 3306: + ```bash docker run --name mariadb-p3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -p 3306:3306 -d docker.io/library/mariadb:10.8.3 ``` If you have a local MySQL instance already running on port 3306, you can map the docker container to run on 3307: + ```bash docker run --name mariadb-p3307 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -p 3307:3306 -d docker.io/library/mariadb:10.8.3 ``` @@ -185,6 +189,7 @@ docker exec -it mariadb-p3306 bash ``` Set up the database with the `sm_api` user and appropriate permissions: + ```bash mysql -u root --execute " CREATE DATABASE sm_dev; @@ -215,6 +220,7 @@ popd ``` Ensure the database port environment variable matches the mapping above: + ```bash export SM_DEV_DB_PORT=3306 # or 3307 ``` @@ -223,7 +229,7 @@ export SM_DEV_DB_PORT=3306 # or 3307 You'll want to set the following environment variables (permanently) in your local development environment. -The `SM_ENVIRONMENT`, `SM_LOCALONLY_DEFAULTUSER` and `SM_ALLOWALLACCESS` environment variables allow access to a local metamist server without providing a bearer token. +The `SM_ENVIRONMENT`, `SM_LOCALONLY_DEFAULTUSER` and `SM_ALLOWALLACCESS` environment variables allow access to a local metamist server without providing a bearer token. This will allow you to test the front-end components that access data. This happens automatically on the production instance through the Google identity-aware-proxy. @@ -238,10 +244,13 @@ export SM_LOCALONLY_DEFAULTUSER=$(whoami) ``` With those variables set, it is a good time to populate some test data if this is your first time running this server: + ```bash python3 test/data/generate_data.py ``` + You can now run the server: + ```bash # start the server python3 -m api.server @@ -249,7 +258,8 @@ python3 -m api.server # uvicorn --port 8000 --host 0.0.0.0 api.server:app ``` -#### Running and Debugging in VS Code +### Running and Debugging in VS Code + The following `launch.json` is a good base to debug the web server in VS Code: ```json @@ -279,13 +289,5 @@ You can now place breakpoints anywhere and debug the API with "Run API" under th ## Contributing -See the [Contributing](contributing.md) page for instructions on how to make changes and regenerate the API. - - - - - - - - +See the [Contributing](contributing.md) page for instructions on how to make changes and regenerate the API. diff --git a/docs/introduction.md b/docs/introduction.md index 6ea831870..dffc21142 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -1,7 +1,7 @@ # Introduction -**Metamist** is a database designed for storing de-identified -omics metadata. +**Metamist** is a database designed for storing **de-identified** -omics metadata. ## Purpose @@ -14,7 +14,7 @@ The project provides an interface to interact with the -omics database via the P - `dataset-sample-metadata-main-read`: For read-only access. - `dataset-sample-metadata-main-write`: For write access. - **Efficiency Note**: Members of Google Groups are cached in a blob to optimize performance, as group-membership identity lookups can be slow. - + ## High-Level Architecture It comprises three key components: @@ -45,5 +45,6 @@ The codebase contains the following modules worth noting: - `api/routes`: HTTP + OpenAPI And metamist maintains two clients: + - `web` -> React app that consumes a generated Typescript API + GraphQL -- `metamist` -> autogenerated Python API \ No newline at end of file +- `metamist` -> autogenerated Python API