diff --git a/README.md b/README.md
index ec3150ae4..5916d3282 100644
--- a/README.md
+++ b/README.md
@@ -1,453 +1,62 @@
-# Sample Metadata
+# Metamist
 
 [![codecov](https://codecov.io/gh/populationgenomics/metamist/branch/dev/graph/badge.svg?token=OI3XZYR9HK)](https://codecov.io/gh/populationgenomics/metamist)
 
-Metamist is database that stores **de-identified** -omics metadata.
 
-There are three components to the metamist system:
+## Introduction
 
-- System-versioned MariaDB database,
-- Python web API to manage permissions, and store frequently used queries,
-  - Including a GraphQL API for querying the database
-- An installable python library that wraps the Python web API (using OpenAPI generator)
+**Metamist** is a database designed for storing **de-identified** -omics metadata.
 
-Every resource in metamist belongs to a project. All resources are access
-controlled through membership of the google groups:
-`$dataset-sample-metadata-main-{read,write}`. Note that members of google-groups
-are cached in a blob as group-membership identity lookups are slow.
+## Purpose
 
-## API
+The project provides an interface to interact with the -omics database via the Python client as well as the GraphQL + HTTP APIs.
 
-There are two ways to query metamist in Python:
+## Features
 
-1. Use the REST interface with the predefined requests
-2. Use the GraphQL interface.
+- **Project-Based Resource Organization**: Every resource in Metamist is associated with a specific project.
+- **Access Control**: Access to resources is controlled through membership in specific Google Groups:
+  - `dataset-sample-metadata-main-read`: For read-only access.
+  - `dataset-sample-metadata-main-write`: For write access.
+- **Efficiency Note**: Members of Google Groups are cached in a blob to optimize performance, as group-membership identity lookups can be slow.
 
-To use the GraphQL interface in Python with the `sample_metadata` library, you can do the following:
+## High-Level Architecture
 
-```python
-from sample_metadata.graphql import query
+It comprises three key components:
 
-_query = """
-query YourQueryNameHere($sampleId: String!) {
-  sample(id: $sampleId) {
-    id
-    externalId
-  }
-}
-"""
+1. **System-Versioned MariaDB Database**: A robust database system for managing -omics metadata.
 
-print(query(_query, {"sampleId": "CPG18"}))
-```
+2. **Python Web API**: This component is responsible for:
+   - Managing permissions.
+   - Storing frequently used queries.
+   - Providing a GraphQL/HTTP API for efficient querying of the database.
 
-## Structure
+3. **Installable Python Library**: Wraps the Python Web API using the OpenAPI generator, facilitating easier interaction with the system.
 
-![Database structure](resources/2021-10-27_db-diagram.png)
+### Schema
 
-### Sample IDs
+As of Jan 15, 2024 this schema should reflect the data structure on the tables:
 
-In an effort to reduce our dependency on potentially mutable external sample IDs with inconsistent format,
-the metamist server generates an internal sample id for every sample. Internally they're an
-incrementing integer, but they're transformed externally to have a prefix, and checksum - this allows durability
-when transcribing sample IDs to reduce mistypes, and allow to quickly check whether a sample ID is valid.
+![Database Structure](resources/2024-01-15_db-diagram.png)
 
-> NB: The prefix and checksums are modified per environment (production, development, local) to avoid duplicates from these environments.
+You can also find this at [DbDiagram](https://dbdiagram.io/d/Metamist-Schema-v6-6-2-65a48ac7ac844320aee60d16).
 
-For example, let's consider the production environment which uses the prefix of `CPG` and a checksum offset of 0:
+The codebase contains the following modules worth noting:
 
-> A sample is given the internal ID `12345`, we calculate the Luhn checksum to be `5` (with no offset applied).
-> We can then concatenate the results, for the final sample ID to be `CPG123455`.
+- `models` -> General data models + enums
+- `db/python/tables` -> Interaction with MariaDB / BigQuery
+- `db/python/layers` -> Logic
+- `api/graphql` : GraphQL
+- `api/routes`: HTTP + OpenAPI
 
-### Reporting sex
+And metamist maintains two clients:
 
-To avoid ambiguity in reporting of gender, sex and karyotype - the sample metadata system
-stores these values separately on the `participant` as:
+- `web`  -> React app that consumes a generated Typescript API + GraphQL
+- `metamist` -> autogenerated Python API
 
-- `reported_gender` (string, expected `male` | `female` | _other values_)
-- `reported_sex` (follows pedigree convention: `unknown=0 | null`, `male=1`, `female=2`)
-- `inferred_karyotype` (string, eg: `XX` | `XY` | _other karyotypes_)
+## Installation and Running Locally
 
-If you import a pedigree, the sex value is written to the `reported_sex` attribute.
+- [Installation and developer setup](docs/installation.md)
 
-## Local develompent of SM
+## License
 
-The recommended way to develop the metamist system is to run a local copy of SM.
-
-> There have been some reported issues of running a local SM environment on an M1 mac.
-
-You can run MariaDB with a locally installed docker, or from within a docker container.
-You can configure the MariaDB connection with environment variables.
-
-### Creating the environment
-
-Python dependencies for the `metamist` API package are listed in `setup.py`.
-Additional dev requirements are listed in `requirements-dev.txt`, and packages for
-the sever-side code are listed in `requirements.txt`.
-
-We _STRONGLY_ encourage the use of `pyenv` for managing Python versions.
-Debugging and the server will run on a minimum python version of 3.10.
-
-To setup the python environment, you can run:
-
-```shell
-virtualenv venv
-source venv/bin/activate
-pip install -r requirements.txt
-pip install -r requirements-dev.txt
-pip install --editable .
-```
-
-### Extra software
-
-You'll need to install the following software to develop metamist:
-
-- Node / NPM (recommend using nvm)
-- MariaDB (using MariaDB in docker is also good)
-- Java (for liquibase / openapi-generator)
-- Liquibase
-- OpenAPI generator
-- wget (optional)
-
-Our recommendation is in the following code block:
-
-```shell
-brew install wget
-brew install java
-brew install liquibase
-```
-
-Add the following to your `.zshrc` file:
-
-```shell
-
-# homebrew should export this on an M1 Mac
-# the intel default is /usr/local
-export HB_PREFIX=${HOMEBREW_PREFIX-/usr/local}
-
-# installing Java through brew recommendation
-export CPPFLAGS="-I$HB_PREFIX/opt/openjdk/include"
-
-# installing liquibase through brew recommendation
-export LIQUIBASE_HOME=$(brew --prefix)/opt/liquibase/libexec
-
-export PATH="$HB_PREFIX/bin:$PATH:$HB_PREFIX/opt/openjdk/bin"
-```
-
-#### Node through node-version manager (nvm)
-
-We aren't using too many node-specific features, anything from 16 should work fine,
-this will install the LTS version:
-
-```shell
-brew install nvm
-
-# you may need to add the the following to your .zshrc
-# export NVM_DIR="$HOME/.nvm"
-# [ -s "$HB_PREFIX/opt/nvm/nvm.sh" ] && \. "$HB_PREFIX/opt/nvm/nvm.sh"  # This loads nvm
-# [ -s "$HB_PREFIX/opt/nvm/etc/bash_completion.d/nvm" ] && \. "$HB_PREFIX/opt/nvm/etc/bash_completion.d/nvm"  # This loads nvm bash_completion
-
-# install latest version of node + npm
-nvm install --lts
-```
-
-#### OpenAPI generator
-
-You'll need this to generate the Python and Typescript API.
-
-```shell
-npm install @openapitools/openapi-generator-cli -g
-openapi-generator-cli version-manager set 5.3.0
-
-# put these in your .zshrc
-export OPENAPI_COMMAND="npx @openapitools/openapi-generator-cli"
-alias openapi-generator="npx @openapitools/openapi-generator-cli"
-```
-
-#### MariaDB install
-
-If you're planning to install MariaDB locally, brew is the easiest:
-
-```shell
-
-brew install mariadb@10.8
-# start mariadb on computer start
-brew services start mariadb@10.8
-
-# make mariadb command available on path
-export PATH="$HB_PREFIX/opt/mariadb@10.8/bin:$PATH"
-```
-
-#### Your .zshrc file
-
-If you installed all the software through brew and npm
-like this guide suggests, your `.zshrc` may look like this:
-
-
-```shell
-alias openapi-generator="npx @openapitools/openapi-generator-cli"
-
-# homebrew should export this on an M1 Mac
-# the intel default is /usr/local
-export HB_PREFIX=${HOMEBREW_PREFIX-/usr/local}
-
-# metamist
-export SM_ENVIRONMENT=LOCAL # good default to have
-export SM_DEV_DB_USER=sm_api # makes it easier to copy liquibase update command
-export OPENAPI_COMMAND="npx @openapitools/openapi-generator-cli"
-
-export PATH="$HB_PREFIX/bin:$HB_PREFIX/opt/mariadb@10.8/bin:$PATH:$HB_PREFIX/opt/openjdk/bin"
-
-export CPPFLAGS="-I$HB_PREFIX/opt/openjdk/include"
-export LIQUIBASE_HOME=$(brew --prefix)/opt/liquibase/libexec
-
-# node
-export NVM_DIR="$HOME/.nvm"
-[ -s "$HB_PREFIX/opt/nvm/nvm.sh" ] && \. "$HB_PREFIX/opt/nvm/nvm.sh"  # This loads nvm
-[ -s "$HB_PREFIX/opt/nvm/etc/bash_completion.d/nvm" ] && \. "$HB_PREFIX/opt/nvm/etc/bash_completion.d/nvm"  # This loads nvm bash_completion
-```
-
-### Database setup
-
-These are the default values for the SM database connection.
-Please alter them if you use any different values when setting up the database.
-
-```shell
-export SM_DEV_DB_USER=root    # this is the default, but we now recommend sm_api
-export SM_DEV_DB_PASSWORD= # empty password
-export SM_DEV_DB_HOST=127.0.0.1
-export SM_DEV_DB_PORT=3306 # default mariadb port
-export SM_DEV_DB_NAME=sm_dev;
-```
-
-Create the database in MariaDB (by default, we call it `sm_dev`):
-
-> In newer installs of MariaDB, the root user is protected by default.
-
-We'll setup a user called `sm_api`, and setup permissions
-
-```shell
-sudo mysql -u root --execute "
-  CREATE DATABASE sm_dev;
-  CREATE USER sm_api@'%';
-  CREATE USER sm_api@localhost;
-  CREATE ROLE sm_api_role;
-  GRANT sm_api_role TO sm_api@'%';
-  GRANT sm_api_role TO sm_api@localhost;
-  SET DEFAULT ROLE sm_api_role FOR sm_api@'%';
-  SET DEFAULT ROLE sm_api_role FOR sm_api@localhost;
-  GRANT ALL PRIVILEGES ON sm_dev.* TO sm_api_role;
-"
-```
-
-Then, before you run you'll need to export the varied:
-
-```shell
-# also put this in your .zshrc
-export SM_DEV_DB_USER=sm_api
-```
-
-Download the `mariadb-java-client` and create the schema using liquibase:
-
-```shell
-pushd db/
-wget https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
-liquibase \
-    --changeLogFile project.xml \
-    --url jdbc:mariadb://localhost/sm_dev \
-    --driver org.mariadb.jdbc.Driver \
-    --classpath mariadb-java-client-3.0.3.jar \
-    --username ${SM_DEV_DB_USER:-root} \
-    update
-popd
-```
-
-#### Using Maria DB docker image
-
-Pull mariadb image
-
-```bash
-docker pull mariadb:10.8.3
-```
-
-Run a mariadb container that will server your database. `-p 3307:3306` remaps the port to 3307 in case if you local MySQL is already using 3306
-
-```bash
-docker stop mysql-p3307  # stop and remove if the container already exists
-docker rm mysql-p3307
-# run with an empty root password
-docker run -p 3307:3306 --name mysql-p3307 -e MYSQL_ALLOW_EMPTY_PASSWORD=true -d mariadb:10.8.3
-```
-
-```bash
-mysql --host=127.0.0.1 --port=3307 -u root -e 'CREATE DATABASE sm_dev;'
-mysql --host=127.0.0.1 --port=3307 -u root -e 'show databases;'
-```
-
-Go into the `db/` subdirectory, download the `mariadb-java-client` and create the schema using liquibase:
-
-```bash
-
-pushd db/
-wget https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
-liquibase \
-    --changeLogFile project.xml \
-    --url jdbc:mariadb://127.0.0.1:3307/sm_dev \
-    --driver org.mariadb.jdbc.Driver \
-    --classpath mariadb-java-client-3.0.3.jar \
-    --username root \
-    update
-popd
-```
-
-Finally, make sure you configure the server (making use of the environment variables) to point it to your local Maria DB server
-
-```bash
-export SM_DEV_DB_PORT=3307
-```
-
-### Running the server
-
-You'll want to set the following environment variables (permanently) in your
-local development environment.
-
-The `SM_LOCALONLY_DEFAULTUSER` environment variable along with `ALLOWALLACCESS` to allow access to a local metamist server without providing a bearer token. This will allow you to test the front-end components that access data. This happens automatically on the production instance through the Google identity-aware-proxy.
-
-```shell
-export SM_ALLOWALLACCESS=1
-export SM_LOCALONLY_DEFAULTUSER=$(whoami)
-```
-
-```shell
-# ensures the SWAGGER page points to your local: (localhost:8000/docs)
-# and ensures if you use the PythonAPI, it also points to your local
-export SM_ENVIRONMENT=LOCAL
-# skips permission checks in your local environment
-export SM_ALLOWALLACCESS=true
-# uses your username as the "author" in requests
-export SM_LOCALONLY_DEFAULTUSER=$(whoami)
-
-# probably need this
-
-
-# start the server
-python3 -m api.server
-# OR
-# uvicorn --port 8000 --host 0.0.0.0 api.server:app
-```
-
-#### Running + debugging in VSCode
-
-The following `launch.json` is a good base to debug the web server in VSCode:
-
-```json
-{
-    "version": "0.2.0",
-    "configurations": [
-        {
-            "name": "Run API",
-            "type": "python",
-            "request": "launch",
-            "module": "api.server",
-            "justMyCode": false,
-            "env": {
-                "SM_ALLOWALLACCESS": "true",
-                "SM_LOCALONLY_DEFAULTUSER": "<user>-local",
-                "SM_ENVIRONMENT": "local",
-                "SM_DEV_DB_USER": "sm_api",
-            }
-        }
-    ]
-}
-```
-
-We could now place breakpoints on the sample route (ie: `api/routes/sample.py`), and debug requests as they come in.
-
-Then in VSCode under the _Run and Debug_ tab (⌘⇧D), you can "Run API":
-
-![Run API](resources/debug-api.png)
-
-#### Quickstart: Generate and install the python installable API
-
-Generating the installable APIs (Python + Typescript) involves running
-the server, getting the `/openapi.json`, and running `openapi-generator`.
-
-The `regenerate_api.py` script does this in a few ways:
-
-1. Uses a running server on `localhost:8000`
-2. Runs a docker container from the `SM_DOCKER` environment variable
-3. Spins up the server itself
-
-Most of the time, you'll use 1 or 3:
-
-```bash
-# this will start the api.server, so make sure you have the dependencies installed,
-python regenerate_api.py \
-    && pip install .
-```
-
-If you'd prefer to use the Docker approach (eg: on CI), this command
-will build the docker container and supply it to regenerate_api.py.
-
-```bash
-# SM_DOCKER is a known env variable to regenerate_api.py
-export SM_DOCKER="cpg/metamist-server:dev"
-docker build --build-arg SM_ENVIRONMENT=local -t $SM_DOCKER -f deploy/api/Dockerfile .
-python regenerate_api.py
-```
-
-#### Generating example data
-
-> You'll need to generate the installable API before running this step
-
-You can run the `generate_data.py` script to generate some
-random data to look at.
-
-```shell
-export SM_ENVIRONMENT=local # important
-python test/data/generate_data.py
-```
-
-#### Developing the UI
-
-```shell
-# Ensure you have started sm locally on your computer already, then in another tab open the UI.
-# This will automatically proxy request to the server.
-cd web
-npm install
-npm run compile
-npm start
-```
-
-This will start a web server using Vite, running on [localhost:5173](http://localhost:5173).
-
-
-### OpenAPI and Swagger
-
-The Web API uses `apispec` with OpenAPI3 annotations on each route to describe interactions with the server. We can generate a swagger UI and an installable
-python module based on these annotations.
-
-Some handy links:
-
-- [OpenAPI specification](https://swagger.io/specification/)
-- [Describing parameters](https://swagger.io/docs/specification/describing-parameters/)
-- [Describing request body](https://swagger.io/docs/specification/describing-request-body/)
-- [Media types](https://swagger.io/docs/specification/media-types/)
-
-The web API exposes this schema in two ways:
-
-- Swagger UI: `http://localhost:8000/docs`
-  - You can use this to construct requests to the server
-  - Make sure you fill in the Bearer token (at the top right )
-- OpenAPI schema: `http://localhost:8000/schema.json`
-  - Returns a JSON with the full OpenAPI 3 compliant schema.
-  - You could put this into the [Swagger editor](https://editor.swagger.io/) to see the same "Swagger UI" that `/api/docs` exposes.
-  - We generate the metamist installable Python API based on this schema.
-
-## Deployment
-
-The CPG deploy is managed through Cloud Run on the Google Cloud Platform.
-The deploy github action builds the container, and is deployed.
-
-Additionally you can access metamist through the identity-aware proxy (IAP),
-which handles the authentication through OAuth, allowing you to access the
-front-end.
+This project is licensed under the MIT License. You can see it in the [LICENSE](LICENSE) file in the root directory of this source tree.
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 000000000..89d35b38b
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,421 @@
+# Installation
+
+This document provides detailed instructions on how to install the project. Follow these steps to set up the project on your local system.
+
+## Prerequisites
+
+[Homebrew](https://brew.sh) is the simplest way to install system dependencies needed for this project.
+
+[Chocolatey](https://community.chocolatey.org/) is a good equivalent to Homebrew for package management in Windows.
+
+## System Requirements
+
+
+- **Node/NPM** (recommend using [pnpm](https://pnpm.io/motivation) for this, but you can also use [nvm](https://github.com/nvm-sh/nvm))
+- **MariaDB** (using MariaDB in docker is also good)
+- **Java** (for Liquibase / OpenAPI  Generator)
+- **Liquibase**
+- **OpenAPI Generator**
+- **pyenv**
+- **wget** *(optional)*
+
+### Mac
+
+```bash
+brew install pnpm # recommended over nvm
+# OR
+brew install nvm
+
+brew install java
+brew install liquibase
+brew install pyenv
+brew install wget
+
+# skip if you wish to install via docker
+brew install mariadb@10.8
+
+```
+
+### Windows
+
+Instructions for Windows should theoretically work but we have only tested this project to work on -nix systems. As such, we are unable to verify any discrepancies on Windows, and there could be slight variations in your setup.
+
+```bash
+# Assuming you have Chocolatey
+choco install pnpm # Recommended
+# OR
+choco install nvm
+
+choco install jdk8
+choco install liquibase
+choco install pyenv-win
+choco install wget
+
+# skip if you wish to install via docker
+choco install mariadb --version=10.8.3
+```
+
+## Installation Steps
+
+### Creating the environment
+
+- Python dependencies for the `metamist` API package are listed in `setup.py`.
+- Additional dev requirements are listed in `requirements-dev.txt`.
+- Packages for the sever-side code are listed in `requirements.txt`.
+
+We *STRONGLY* encourage the use of `pyenv` for managing Python versions. Debugging and the server will run on a minimum python version of 3.10. Refer to the [team-docs](https://github.com/populationgenomics/team-docs/blob/main/python.md) for more instructions on how to set this up.
+
+Use of a virtual environment to contain all requirements is highly recommended:
+
+```bash
+virtualenv venv
+source venv/bin/activate
+pip install -r requirements.txt -r requirements-dev.txt
+
+# Installs metamist as a package
+pip install --editable .
+```
+
+You will also need to set the following environment variables. Adjust the paths if you installed the dependencies using an alternate means:
+
+```bash
+# homebrew should export this on an M1 Mac
+# the intel default is /usr/local
+export HB_PREFIX=${HOMEBREW_PREFIX-/usr/local}
+
+# installing Java through brew recommendation
+export CPPFLAGS="-I$HB_PREFIX/opt/openjdk/include"
+export PATH="$HB_PREFIX/bin:$PATH:$HB_PREFIX/opt/openjdk/bin"
+
+# installing liquibase through brew recommendation
+export LIQUIBASE_HOME=$(brew --prefix)/opt/liquibase/libexec
+
+# mariadb
+export PATH="$HB_PREFIX/opt/mariadb@10.8/bin:$PATH"
+
+# metamist config
+export SM_ENVIRONMENT=LOCAL # good default to have
+export SM_DEV_DB_USER=sm_api # makes it easier to copy liquibase update command
+```
+
+You can also add these to your shell config file e.g `.zshrc` or `.bashrc` for persistence to new sessions.
+
+#### PNPM/NVM Config
+
+Depending on your choice for using `pnpm` or `nvm` you will have to configure your shell for it.
+
+If you installed `pnpm`, you should have a similar snippet from the brew installation output:
+
+```shell
+export PNPM_HOME="/Users/$(whoami)/Library/pnpm"
+case ":$PATH:" in
+  *":$PNPM_HOME:"*) ;;
+  *) export PATH="$PNPM_HOME:$PATH" ;;
+esac
+```
+
+Add this to your `.zshrc` to auto-load on next shell session.
+
+If you installed `nvm`, you will need to add lazy load since `nvm` has high load penalties.
+
+- For Oh-My-Zsh users, you can just add the `nvm` plugin to your `.zshrc` via these [instructions](https://github.com/ohmyzsh/ohmyzsh/blob/master/plugins/nvm/README.md)
+
+- If you do NOT have Oh-My-Zsh, you can use this [plugin](https://github.com/undg/zsh-nvm-lazy-load):
+
+```shell
+git clone https://github.com/undg/zsh-nvm-lazy-load $ZSH/custom/plugins/zsh-nvm
+
+#Add this to your plugins variable in the `.zshrc` file and then source the file.
+plugins=(... zsh-nvm-lazy-load)
+```
+
+Once set up, install the OpenAPI Generator:
+
+- For `pnpm`:
+
+```shell
+# Install npm via pnpm
+# This also activates the env for you, replace `use` with `add` to only install it.
+pnpm env use --global lts
+pnpm install @openapitools/openapi-generator-cli -g
+```
+
+Add this to your `.zshrc`:
+
+```shell
+# openapi
+export OPENAPI_COMMAND="pnpm dlx @openapitools/openapi-generator-cli"
+alias openapi-generator="pnpm dlx @openapitools/openapi-generator-cli"
+```
+
+- For `nvm`:
+
+```shell
+# Install npm via nvm
+nvm install --lts
+npm install @openapitools/openapi-generator-cli -g
+```
+
+Add this to your `.zshrc`:
+
+```shell
+# openapi
+export OPENAPI_COMMAND="npx @openapitools/openapi-generator-cli"
+alias openapi-generator="npx @openapitools/openapi-generator-cli"
+```
+
+Finally, set the version:
+
+```shell
+openapi-generator-cli version-manager set 5.3.0
+```
+
+### Database Setup - Native Installation
+
+Set the following environment variables:
+
+```bash
+export SM_DEV_DB_USER=sm_api
+export SM_DEV_DB_PASSWORD= # empty password
+export SM_DEV_DB_HOST=127.0.0.1
+export SM_DEV_DB_PORT=3306 # default mariadb port
+export SM_DEV_DB_NAME=sm_dev;
+```
+
+Next, create the database `sm_dev` in MariaDB.
+
+> In newer versions of MariaDB, the root user is protected.
+
+Create a new user `sm_api` and provide permissions:
+
+```bash
+sudo mysql -u root --execute "
+  CREATE DATABASE sm_dev;
+  CREATE USER sm_api@'%';
+  CREATE USER sm_api@localhost;
+  CREATE ROLE sm_api_role;
+  GRANT sm_api_role TO sm_api@'%';
+  GRANT sm_api_role TO sm_api@localhost;
+  SET DEFAULT ROLE sm_api_role FOR sm_api@'%';
+  SET DEFAULT ROLE sm_api_role FOR sm_api@localhost;
+  GRANT ALL PRIVILEGES ON sm_dev.* TO sm_api_role;
+"
+```
+
+Using `liquibase` we can now set up the tables as per the schema in `db/project.xml`:
+
+```bash
+pushd db/
+wget https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
+liquibase \
+    --changeLogFile project.xml \
+    --url jdbc:mariadb://localhost/sm_dev \
+    --driver org.mariadb.jdbc.Driver \
+    --classpath mariadb-java-client-3.0.3.jar \
+    --username ${SM_DEV_DB_USER:-root} \
+    update
+popd
+```
+
+### Database Setup - Docker Installation
+
+Ensure you have Docker installed or follow [this guide](https://docs.docker.com/engine/install/) to setup.
+
+Pull the image:
+
+```bash
+docker pull mariadb:10.8.3
+```
+
+Run the container on port 3306:
+
+```bash
+docker run --name mariadb-p3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -p 3306:3306 -d docker.io/library/mariadb:10.8.3
+```
+
+If you have a local MySQL instance already running on port 3306, you can map the docker container to run on 3307:
+
+```bash
+docker run --name mariadb-p3307 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -p 3307:3306 -d docker.io/library/mariadb:10.8.3
+```
+
+You can now execute bash commands inside a shell:
+
+```bash
+docker exec -it mariadb-p3306 bash
+```
+
+Set up the database with the `sm_api` user and appropriate permissions:
+
+```bash
+mysql -u root --execute "
+  CREATE DATABASE sm_dev;
+  CREATE USER sm_api@'%';
+  CREATE USER sm_api@localhost;
+  CREATE ROLE sm_api_role;
+  GRANT sm_api_role TO sm_api@'%';
+  GRANT sm_api_role TO sm_api@localhost;
+  SET DEFAULT ROLE sm_api_role FOR sm_api@'%';
+  SET DEFAULT ROLE sm_api_role FOR sm_api@localhost;
+  GRANT ALL PRIVILEGES ON sm_dev.* TO sm_api_role;
+"
+```
+
+Exit the container bash shell once done and on the host, run liquibase with the correct port mapping to set up the tables:
+
+```bash
+pushd db/
+wget https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/3.0.3/mariadb-java-client-3.0.3.jar
+liquibase \
+    --changeLogFile project.xml \
+    --url jdbc:mariadb://127.0.0.1:3306/sm_dev \
+    --driver org.mariadb.jdbc.Driver \
+    --classpath mariadb-java-client-3.0.3.jar \
+    --username root \
+    update
+popd
+```
+
+Ensure the database port environment variable matches the mapping above:
+
+```bash
+export SM_DEV_DB_PORT=3306 # or 3307
+```
+
+## Running the server
+
+You'll want to set the following environment variables (permanently) in your local development environment.
+
+The `SM_ENVIRONMENT`, `SM_LOCALONLY_DEFAULTUSER` and `SM_ALLOWALLACCESS` environment variables allow access to a local metamist server without providing a bearer token.
+
+This will allow you to test the front-end components that access data. This happens automatically on the production instance through the Google identity-aware-proxy.
+
+```bash
+# ensures the SWAGGER page points to your local: (localhost:8000/docs)
+# and ensures if you use the PythonAPI, it also points to your local
+export SM_ENVIRONMENT=LOCAL
+# skips permission checks in your local environment
+export SM_ALLOWALLACCESS=true
+# uses your username as the "author" in requests
+export SM_LOCALONLY_DEFAULTUSER=$(whoami)
+```
+
+With those variables set, it is a good time to populate some test data if this is your first time running this server:
+
+```bash
+python3 test/data/generate_data.py
+```
+
+You can now run the server:
+
+```bash
+# start the server
+python3 -m api.server
+# OR
+# uvicorn --port 8000 --host 0.0.0.0 api.server:app
+```
+
+
+## Running Locally for Dev
+
+### Running and Debugging in VS Code
+
+The following `launch.json` is a good base to debug the web server in VS Code:
+
+```json
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Run API",
+            "type": "python",
+            "request": "launch",
+            "module": "api.server",
+            "justMyCode": false,
+            "env": {
+                "SM_ALLOWALLACCESS": "true",
+                "SM_LOCALONLY_DEFAULTUSER": "<user>-local",
+                "SM_ENVIRONMENT": "local",
+                "SM_DEV_DB_USER": "sm_api",
+            }
+        }
+    ]
+}
+```
+
+You can now place breakpoints anywhere and debug the API with "Run API" under the *Run and Debug* tab (⌘⇧D) or (Ctrl+Shift+D):
+
+![Run and Debug](../resources/debug-api.png)
+
+### Generate and install the python installable API
+
+After making any changes to the logic, it is worth regenerating the API with the OpenAPI Generator.
+
+Generating the installable APIs (Python + Typescript) involves running the server, getting the `/openapi.json`, and running `openapi-generator`.
+
+The `regenerate_api.py` script does this for us in a few ways:
+
+1. Uses a running server on `localhost:8000`
+2. Runs a docker container from the `SM_DOCKER` environment variable.
+3. Spins up the server itself.
+
+You can simply run:
+
+```bash
+# this will start the api.server, so make sure you have the dependencies installed,
+python regenerate_api.py \
+    && pip install .
+```
+
+or if you prefer the Docker approach (eg: for CI), this command will build the docker container and supply it to `regenerate_api.py`:
+
+```bash
+# SM_DOCKER is a known env variable to regenerate_api.py
+export SM_DOCKER="cpg/metamist-server:dev"
+docker build --build-arg SM_ENVIRONMENT=local -t $SM_DOCKER -f deploy/api/Dockerfile .
+python regenerate_api.py
+```
+
+### Developing the UI
+
+```bash
+# Ensure you have started metamist server locally on your computer already, then in another tab open the UI.
+# This will automatically proxy request to the server.
+cd web
+npm install
+npm run compile
+npm start
+```
+
+This will start a web server using Vite, running on `localhost:5173`.
+
+### OpenAPI and Swagger
+
+The Web API uses `apispec` with OpenAPI3 annotations on each route to describe interactions with the server. We can generate a swagger UI and an installable
+python module based on these annotations.
+
+Some handy links:
+
+- [OpenAPI specification](https://swagger.io/specification/)
+- [Describing parameters](https://swagger.io/docs/specification/describing-parameters/)
+- [Describing request body](https://swagger.io/docs/specification/describing-request-body/)
+- [Media types](https://swagger.io/docs/specification/media-types/)
+
+The web API exposes this schema in two ways:
+
+- Swagger UI: `http://localhost:8000/docs`
+  - You can use this to construct requests to the server
+  - Make sure you fill in the Bearer token (at the top right )
+- OpenAPI schema: `http://localhost:8000/schema.json`
+  - Returns a JSON with the full OpenAPI 3 compliant schema.
+  - You could put this into the [Swagger editor](https://editor.swagger.io/) to see the same "Swagger UI" that `/api/docs` exposes.
+  - We generate the metamist installable Python API based on this schema.
+
+## Deployment
+
+The CPG deploy is managed through Cloud Run on the Google Cloud Platform.
+The deploy github action builds the container, and is deployed.
+
+Additionally you can access metamist through the identity-aware proxy (IAP),
+which handles the authentication through OAuth, allowing you to access the
+front-end.
diff --git a/resources/2024-01-15_db-diagram.png b/resources/2024-01-15_db-diagram.png
new file mode 100644
index 000000000..8a34e0032
Binary files /dev/null and b/resources/2024-01-15_db-diagram.png differ