Implement database infra utilities and tests for api (#118, #122)

- Adds unit and functional tests for api
- Adds docker compose with db init script to seed from dump
- Updates READMEs with relevant instructions

.github/workflows/validate-n-build-api.yml

@@ -0,0 +1,66 @@
+name: Validate and Build API
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    branches: ["main"]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  validate-n-build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Check for changes
+        uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            api:
+              - 'api/**'
+            shared:
+              - 'shared/**'
+
+      - name: Restore node_modules cache
+        id: deps-cache
+        uses: martijnhols/actions-cache/restore@v3
+        with:
+          path: api/node_modules
+          key: ${{ runner.os }}-build-api-deps-cache-${{ hashFiles('api/package-lock.json') }}
+
+      - name: Install Dependencies
+        if: steps.deps-cache.outputs.cache-hit != 'true'
+        run: |
+          cd api
+          npm install
+
+      - name: Cache node modules
+        if: steps.deps-cache.outputs.cache-hit != 'true'
+        uses: martijnhols/actions-cache/save@v3
+        with:
+          path: api/node_modules
+          key: ${{ runner.os }}-build-api-deps-cache-${{ hashFiles('api/package-lock.json') }}
+
+      - name: Run static code analysis
+        if: steps.filter.outputs.api == 'true' || steps.filter.outputs.shared == 'true'
+        run: |
+          cd api
+          npm run lint
+
+      - name: Run unit tests
+        if: steps.filter.outputs.api == 'true' || steps.filter.outputs.shared == 'true'
+        run: |
+          cd api
+          npm run test:unit
+
+
+      - name: Build the Docker image for API
+        if: steps.filter.outputs.api == 'true' || steps.filter.outputs.shared == 'true'
+        run: docker build . --file ./api/Dockerfile

.gitignore

@@ -89,6 +89,7 @@ dist
 
 # IDE files
 .idea
+.vscode
 
 # yarn v2
 .yarn/cache

.postgres.local.env

@@ -0,0 +1,4 @@
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=password
+POSTGRES_DBS_FOR_IMPORT=cloudmos-akash-sandbox
+POSTGRES_USERS_DB=cloudmos-users

README.md

@@ -82,17 +82,47 @@ Directory: [config](/config/)
 
 See [Example_Queries.md](./doc/Example_Queries.md)
 
-# How to run
+# Running the Application
 
-Indexing akash from block #1 take a long time so it is recommended to start from an existing database backup instead.
+This document provides instructions on how to set up and run the application, including steps for manual database restoration and using Docker Compose for ease of setup.
 
-[cloudmos-akash-20230611.sql](https://mega.nz/file/heEj1YKI#aC--PVLpKCrgmlHBKwoZ0VD1lZ6XEGvSThMP44rH3gk) (18.88Gb)
+## Manual Database Restoration
 
-First create a postgresql database and then restore from the backup using `pg_restore`.
+Due to the extensive time required to index Akash from block #1, it's recommended to initialize your database using an existing backup for efficiency. This approach is particularly beneficial for development purposes.
 
-`pg_restore --host "localhost" --port "5432" --username "postgres" --password "password" --dbname "cloudmos-akash" --verbose "/path/to/cloudmos-akash-20230611.sql"`
+### Available Backups
 
-Once your database is ready, follow the instructions in the README of the project you want to run.
+- **Mainnet Database (32.1 GB):** [cloudmos-akash-mainnet.sql.gz](https://storage.googleapis.com/cloudmos-postgresql-backups/cloudmos-akash-mainnet.sql)
+  - Suitable for scenarios requiring complete data.
+- **Sandbox Database (258.5 MB):** [cloudmos-akash-sandbox.sql.gz](https://storage.googleapis.com/cloudmos-postgresql-backups/cloudmos-akash-sandbox.sql.gz)
+  - Ideal for most development needs, although it may lack recent chain updates.
+
+### Restoration Steps
+
+1. Create a PostgreSQL database.
+2. Restore the database using `psql`. Ensure PostgreSQL tools are installed on your system.
+
+For a .sql.gz file:
+```sh
+gunzip -c /path/to/cloudmos-akash-sandbox.sql.gz | psql --host "localhost" --port "5432" --username "postgres" --dbname "cloudmos-akash"
+```
+After restoring the database, you can proceed with the specific project's README instructions for further setup and running the application.
+
+### Using Docker Compose
+For convenience, a Docker Compose configuration is provided to automate the database setup, download, and data import process.
+
+#### Configuration
+1. Ensure environment variables are set in .postgres.local.env. This file configures PostgreSQL and import settings.
+2. The `POSTGRES_DBS_FOR_IMPORT` variable should be updated with the databases you wish to import. This is a comma-separated list with potential values including `cloudmos-akash-sandbox` (default) and `cloudmos-akash-2` (mainnet). Leave this variable empty if no import is desired.
+
+#### Running Docker Compose
+```sh
+docker-compose up
+```
+This command spins up the database service and automatically handles the downloading and importing of the specified data.
+
+
+```bash
 
 # Database Structure
 

api/.env.functional.test

@@ -0,0 +1,3 @@
+Network=sandbox
+AkashSandboxDatabaseCS=postgres://postgres:password@localhost:5432/cloudmos-akash-sandbox
+UserDatabaseCS=postgres://postgres:password@localhost:5432/cloudmos-users

api/.eslintrc.json

@@ -1,6 +1,6 @@
 {
   "env": {
-    "browser": true,
+    "node": true,
     "es2021": true
   },
   "ignorePatterns": ["node_modules/", "dist/", "webpack.*.js"],

api/.gitignore

@@ -36,11 +36,12 @@ node_modules/
 
 #dotenv
 .env
+.env.local
 
 # akash
 /api/data
 
 .DS_Store
 .vim
 
-dist
+dist

api/README.md

@@ -34,6 +34,61 @@ Auth0Issuer|ex: `'https://auth.cloudmos.io/'`
 StripeSecretKey|ex: `sk_test_12aw315wdawd3...293d12d32df8jf`
 WebsiteUrl|`http://localhost:3001`
 
+
+## Testing
+Project is configured to use [Jest](https://jestjs.io/) for testing. It is intended to be covered with unit and functional tests where applicable.
+
+### Running tests
+
+To execute both **unit and functional** tests, run:
+
+```shell
+npm test
+````
+
+To run **unit** tests exclusively, use:
+
+```shell
+npm run test:unit
+```
+
+To run only **functional** tests, use:
+
+```shell
+npm run test:functional
+```
+
+#### Watch Mode
+To automatically re-run tests upon any changes, use the following watch mode commands:
+
+```shell
+npm test:watch
+npm run test:unit:watch
+npm run test:functional:watch
+```
+#### Collecting Coverage
+To collect and view test coverage, use the following commands:
+```shell
+npm test:coverage
+npm run test:unit:coverage
+npm run test:functional:coverage
+```
+
+### Contributing to Tests
+**Unit Tests**: Focus on testing individual functions and components in isolation, without external dependencies.
+
+**Test File Structure**: For consistency, each component tested with unit tests should reside in its own directory, named after the component. Place the test file alongside the component, suffixed with .spec. For example:
+
+**Functional Tests**: Aim to evaluate the system's behavior as a whole, including endpoints, workers, and workflows. Place functional tests in the `test/functional` directory.
+
+```
+src/
+  components/
+    myComponent/
+      myComponent.ts
+      myComponent.spec.ts
+```
+
 ## Changes from **beta** to **v1** (February 2024)
 
 ### Api Versioning
@@ -71,4 +126,4 @@ Here is a list of endpoints that have changed in this release. Old endpoints wil
 |`/marketData`|`/market-data`
 |`/predicted-block-date/<height>/<blockWindow>` | `/predicted-block-date/<height>?blockWindow=<blockWindow>`
 |`/predicted-date-height/<timestamp>/<blockWindow>` | `/predicted-date-height/<timestamp>?blockWindow=<blockWindow>`
-|`/providers/<provider>/deployments/<skip>/<take>/<status>`|`/providers/<provider>/deployments/<skip>/<take>?status=<status>`
+|`/providers/<provider>/deployments/<skip>/<take>/<status>`|`/providers/<provider>/deployments/<skip>/<take>?status=<status>`

api/jest.config.js

@@ -0,0 +1,37 @@
+const MAP_ALIASES = {
+  '^@src(.*)$': "<rootDir>/src$1",
+  '^@shared/(.*)$': '<rootDir>/../shared/$1',
+};
+const MAP_TO_SAME_SEQUELIZE_PACKAGE = {
+  '^sequelize(.*)$': '<rootDir>/node_modules/sequelize$1',
+}
+
+const common = {
+  transform: {
+    '^.+\\.(t|j)s$': ['ts-jest', { tsconfig: './tsconfig.json' }],
+  },
+  rootDir: '.',
+  moduleNameMapper: {
+    ...MAP_ALIASES,
+    ...MAP_TO_SAME_SEQUELIZE_PACKAGE,
+  },
+  setupFiles: ['./test/setup.ts'],
+};
+
+module.exports = {
+  collectCoverageFrom: ['./src/**/*.{js,ts}'],
+  projects: [
+    {
+      displayName: 'unit',
+      ...common,
+      testMatch: ['<rootDir>/src/**/*.spec.ts'],
+      setupFilesAfterEnv: ['./test/setup-unit-tests.ts'],
+    },
+    {
+      displayName: 'functional',
+      ...common,
+      testMatch: ['<rootDir>/test/functional/**/*.spec.ts'],
+      setupFilesAfterEnv: ['./test/setup-functional-tests.ts'],
+    },
+  ],
+};