feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

redaphid · 2022-05-03T20:10:04Z

This PR intends to simplify the development ergonomics of nft.storage. It does so by simplifying the environment via docker-compose and uses the environment variables passed into docker-compose as the only source of truth for configuration.

Any developer on the project can simply copy .env.tpl to .env, then run yarn dev:api to start the api system up in an immutable state.
For development work, the .env file defines the constants for playwright, the cloudflare worker, and Github Actions CI.
Github Actions now runs the same command as a developer starting the system. (e.g. copying .env.tpl to .env and running yarn test)
yarn test can be run at the same time as the development cluster without port conflicts or data cross-contamination.
yarn test:watch in api watches for code changes, then re-runs the playwright tests in an isolated environment on save.
yarn dev watches the code, and on save rebuilds
yarn dev:persist persists the database in a docker volume between runs.
yarn dev:clean will clean the database state for yarn dev:persist
yarn test will run playwright locally against the docker-compose cluster
yarn test:docker will run playwright itself inside a docker container`

Closes #1732

…ember what they are

… dance I'm trying to automate

…sh as a way to pass env vars in

…out the constants.js file

…long :(

…exit code, as this was causing problems in the 'persist' mode. Added a `yarn dev:clean` to clean the volumes for the persist mode. Managed to remove one docker-compose file :)

…ocker to control this environment.

redaphid · 2022-06-02T23:51:55Z

@hugomrdias We incorporated your feedback :). Mind looking this over again when you have the chance?

vasco-santos

See that we had a lot of back and forth here. Apologies if I added some questions in comments that were already discussed :)

I like the overall direction here, I like to have both options of using docker and local environment for testing. Left some minor suggestions so that we can get this merged in

vasco-santos · 2022-06-08T08:46:52Z

.env.tpl

@@ -1,38 +1,40 @@
-# needs to be real so create a personal magic.link account or use the staging key
-MAGIC_SECRET_KEY=secret



Could we keep same comments as before?

## PostgREST # API Secrets

I find them useful when we need to tweak the environment, for instance if we need to do. debug session in staging environment

vasco-santos · 2022-06-08T08:49:30Z

.github/workflows/api.yml

-          apiToken: ${{secrets.CF_API_TOKEN }}
-          workingDirectory: 'packages/api'
+      - run: 'cp .env.tpl .env'
+      - run: yarn test:api:docker


I like the cp approach here to better simulate what we do.

Not sure I understand why we are getting rid of deploy-dev? In web3.storage we were feeling we should have it too, as this is useful to test something without merging to main, as other things might need to be shipped and get blocked on a staging test for a large feature.

It's possible that @redaphid and I were confused about deploy-dev. We both thought it was a remnant of the local-tunnel days, where you had to deploy to cloudflare in order to run the dev environment at all.

If it's still useful as an ad-hoc "not quite staging" env we can bring it back.

If it's still useful as an ad-hoc "not quite staging" env we can bring it back.

It is, but it will be re-written with new commits in other PRs which is annoying. Would love to be able to trigger it in CI for a specific branch, but I think we won't be able to provide a branch on a manual workflow trigger

yes please keep the deploy-dev job and also this yaml is not valid at all i think.

vasco-santos · 2022-06-08T08:58:35Z

packages/api/docker/docker-compose.yml

 services:
-  rest:
+  post-rest:


Suggested change

post-rest:

postgrest:

vasco-santos · 2022-06-08T09:04:13Z

packages/api/docker/docker-compose.yml

+    build:
+      dockerfile: './docker/Dockerfile'
+      context: '../'
+    entrypoint: 'psql $DATABASE_CONNECTION -v "ON_ERROR_STOP=1" -f ./db/config.sql -f ./db/tables.sql -f ./db/cargo.testing.sql -f ./db/functions.sql'


I think we should have a init.sql script here. If we add a new sql file that needs to run, it will be difficult for the developer to understand the multiple places where it should be added. So, having a init.sql that would execute the other files would be rad. Not super sure how to do it, but found this

That's a good idea, especially if we start having fancier migrations - we can just hook them into the init script.

vasco-santos · 2022-06-08T09:07:13Z

packages/api/docker/postgres/Dockerfile

@@ -1,13 +1,7 @@
-FROM supabase/postgres:13.3.0
+FROM supabase/postgres:12.5.0


Why are we downgrading?

packages/api/pw-test.config.cjs

vasco-santos · 2022-06-08T09:24:01Z

packages/api/package.json

+    "test": "./scripts/start-test-local.sh",
+    "test:docker": "./scripts/start-test.sh ",


Suggested change

"test": "./scripts/start-test-local.sh",

"test:docker": "./scripts/start-test.sh ",

"test": "./scripts/start-test.sh",

"test:docker": "./scripts/start-test-docker.sh ",

Can we keep same namings to avoid confusion? We would need shell script names renamed

…ge/nft.storage into feat/make-api-running-immutable

redaphid · 2022-06-14T21:16:10Z

@vasco-santos I've incorporated all your suggestions! Lmk if this works for you.

hugomrdias

Can we please remove all the playwright and miniflare out of docker and focus on the DB.
Also please avoid any formatting not enforced by the repo config to make PRs simpler to review.

hugomrdias · 2022-06-15T14:05:47Z

packages/api/docker/postgres/Dockerfile

@@ -1,14 +1,8 @@
-FROM supabase/postgres:13.3.0
+FROM  supabase/postgres:13.3.0

 COPY 00-initial-schema.sql /docker-entrypoint-initdb.d/00-initial-schema.sql


we should add the other sql files here

hugomrdias · 2022-06-15T14:10:43Z

.github/workflows/api.yml

-          apiToken: ${{secrets.CF_API_TOKEN }}
-          workingDirectory: 'packages/api'
+      - run: 'cp .env.tpl .env'
+      - run: yarn test:api:docker


yes please keep the deploy-dev job and also this yaml is not valid at all i think.

vasco-santos

We should update this PR to:

Scope only development setup, let's have test setup from chore: convert tests to use miniflare + ava #2065
In the API package we are building the API, so let's not dockerize the API (either Miniflare or Playwright) and dockerize only external services, i.e Cluster, DB, ...

vasco-santos · 2022-07-20T12:34:23Z

packages/api/README.md

@@ -26,7 +26,7 @@ eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkaWQ6ZXRocjoweDY1MDA3QTczOWFiN0F
 In the case you need to clean up docker after failed tests or debugging session you can just run the command below.

 ```bash
-yarn clean
+yarn dev:clean


We don't have a dev:clean script in api package

vasco-santos · 2022-07-20T12:37:38Z

packages/api/docker/docker-compose.yml

+  playwright:
+    profiles:
+      - test
+    depends_on:
+      api:
+        condition: service_healthy
+    build:
+      dockerfile: ./docker/test.Dockerfile
+      context: ../
+    environment:
+      - BRANCH
+      - CLUSTER_API_URL
+      - CLUSTER_BASIC_AUTH_TOKEN
+      - CLUSTER_SERVICE
+      - COMMITHASH
+      - DATABASE_CONNECTION
+      - DATABASE_TOKEN
+      - DATABASE_URL
+      - DEBUG
+      - ENV
+      - LOGTAIL_TOKEN
+      - MAGIC_SECRET_KEY
+      - MAILCHIMP_API_KEY
+      - MAINTENANCE_MODE
+      - METAPLEX_AUTH_TOKEN
+      - PRIVATE_KEY
+      - S3_ACCESS_KEY_ID
+      - S3_BUCKET_NAME
+      - S3_ENDPOINT
+      - S3_REGION
+      - S3_SECRET_ACCESS_KEY
+      - SALT
+      - SENTRY_DSN
+      - VERSION


Suggested change

playwright:

profiles:

- test

depends_on:

api:

condition: service_healthy

build:

dockerfile: ./docker/test.Dockerfile

context: ../

environment:

- BRANCH

- CLUSTER_API_URL

- CLUSTER_BASIC_AUTH_TOKEN

- CLUSTER_SERVICE

- COMMITHASH

- DATABASE_CONNECTION

- DATABASE_TOKEN

- DATABASE_URL

- DEBUG

- ENV

- LOGTAIL_TOKEN

- MAGIC_SECRET_KEY

- MAILCHIMP_API_KEY

- MAINTENANCE_MODE

- METAPLEX_AUTH_TOKEN

- PRIVATE_KEY

- S3_ACCESS_KEY_ID

- S3_BUCKET_NAME

- S3_ENDPOINT

- S3_REGION

- S3_SECRET_ACCESS_KEY

- SALT

- SENTRY_DSN

- VERSION

vasco-santos · 2022-07-20T12:45:26Z

package.json

    "dev:website": "cd packages/website && yarn dev",
    "test": "run-s test:*",
    "test:client": "yarn --cwd packages/client test",
    "test:api": "yarn --cwd packages/api test",
+    "test:api:docker": "yarn --cwd packages/api test:docker",


Suggested change

"test:api:docker": "yarn --cwd packages/api test:docker",

vasco-santos · 2022-07-20T12:47:10Z

packages/api/docker/test.Dockerfile

+FROM mcr.microsoft.com/playwright:v1.20.0-focal
+
+RUN npm i -g nodemon typescript
+RUN mkdir -p /app
+
+WORKDIR /app
+
+# Make your docker builds 100x faster with this one trick :)
+COPY ./package.json .
+COPY  ./tsconfig.json .
+RUN yarn install
+
+COPY ./pw-test.config.cjs .
+COPY ./docker/scripts ./docker-scripts
+COPY ./scripts ./scripts
+COPY ./db ./db
+COPY ./src/ ./src
+COPY ./test/ ./test
+
+ENTRYPOINT ["./docker-scripts/run-playwright.sh" ]


Suggested change

FROM mcr.microsoft.com/playwright:v1.20.0-focal

RUN npm i -g nodemon typescript

RUN mkdir -p /app

WORKDIR /app

# Make your docker builds 100x faster with this one trick :)

COPY ./package.json .

COPY ./tsconfig.json .

RUN yarn install

COPY ./pw-test.config.cjs .

COPY ./docker/scripts ./docker-scripts

COPY ./scripts ./scripts

COPY ./db ./db

COPY ./src/ ./src

COPY ./test/ ./test

ENTRYPOINT ["./docker-scripts/run-playwright.sh" ]

vasco-santos · 2022-07-20T12:48:11Z

packages/api/package.json

+    "test": "./scripts/start-test.sh",
+    "test:docker": "./scripts/start-test-docker.sh ",


Let's keep test as it was and move forward with tests like #2065

vasco-santos · 2022-07-20T12:49:13Z

packages/api/scripts/start-test.sh

+#!/usr/bin/env bash
+set -eX
+
+# We're moving this script around a lot, and it's pretty cwd-dependent.
+ENV_FILE=../../.env
+COMPOSE_FILES="--file ./docker/docker-compose.yml --file ./docker/docker-compose.local-ports.yml"
+
+cleanup() {
+  docker compose --project-name "nft-storage-api-local-tests" down --remove-orphans
+}
+trap cleanup EXIT
+
+docker compose \
+  --project-name="nft-storage-api-local-tests" \
+  $COMPOSE_FILES --env-file="$ENV_FILE" up \
+  --detach \
+  --always-recreate-deps \
+  --remove-orphans \
+  --force-recreate \
+  --renew-anon-volumes \
+  --build \
+;
+
+THIS_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+API_DIR="$THIS_DIR/../"
+
+pushd $API_DIR
+
+export DATABASE_URL="http://localhost:3000"
+export CLUSTER_API_URL="http://localhost:9094"
+
+tsc \
+&& npx playwright-test "./test/"'**/*.spec.js' --sw src/index.js \
+;
+
+
+popd


Suggested change

#!/usr/bin/env bash

set -eX

# We're moving this script around a lot, and it's pretty cwd-dependent.

ENV_FILE=../../.env

COMPOSE_FILES="--file ./docker/docker-compose.yml --file ./docker/docker-compose.local-ports.yml"

cleanup() {

docker compose --project-name "nft-storage-api-local-tests" down --remove-orphans

}

trap cleanup EXIT

docker compose \

--project-name="nft-storage-api-local-tests" \

$COMPOSE_FILES --env-file="$ENV_FILE" up \

--detach \

--always-recreate-deps \

--remove-orphans \

--force-recreate \

--renew-anon-volumes \

--build \

;

THIS_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"

API_DIR="$THIS_DIR/../"

pushd $API_DIR

export DATABASE_URL="http://localhost:3000"

export CLUSTER_API_URL="http://localhost:9094"

tsc \

&& npx playwright-test "./test/"'**/*.spec.js' --sw src/index.js \

;

popd

vasco-santos · 2022-07-20T12:49:29Z

packages/api/scripts/start-test-docker.sh

+#!/usr/bin/env sh
+
+ENV_FILE=../../.env
+docker compose --profile="test" --file ./docker/docker-compose.yml --env-file="$ENV_FILE" config
+
+docker compose \
+  --project-name="nft-storage-api-test" \
+  --profile="test" --file ./docker/docker-compose.yml --env-file="$ENV_FILE" up \
+  --always-recreate-deps \
+  --force-recreate \
+  --renew-anon-volumes \
+  --remove-orphans \
+  --build \
+  --attach "playwright" \
+  --no-log-prefix \
+;
+


Suggested change

#!/usr/bin/env sh

ENV_FILE=../../.env

docker compose --profile="test" --file ./docker/docker-compose.yml --env-file="$ENV_FILE" config

docker compose \

--project-name="nft-storage-api-test" \

--profile="test" --file ./docker/docker-compose.yml --env-file="$ENV_FILE" up \

--always-recreate-deps \

--force-recreate \

--renew-anon-volumes \

--remove-orphans \

--build \

--attach "playwright" \

--no-log-prefix \

;

vasco-santos · 2022-07-20T12:50:59Z

packages/api/docker/scripts/run-miniflare.sh

+#!/usr/bin/env bash
+
+npx miniflare --debug \
+  --build-command "npm run build" \
+  --binding ENV="$ENV" \
+  --binding PRIVATE_KEY="$PRIVATE_KEY" \
+  --binding DATABASE_CONNECTION="$DATABASE_CONNECTION" \
+  --binding DATABASE_TOKEN="$DATABASE_TOKEN" \
+  --binding DATABASE_URL="$DATABASE_URL" \
+  --binding LOGTAIL_TOKEN="$LOGTAIL_TOKEN" \
+  --binding MAGIC_SECRET_KEY="$MAGIC_SECRET_KEY" \
+  --binding SALT="$SALT" \
+  --binding SENTRY_DSN="$SENTRY_DSN" \
+  --binding MAINTENANCE_MODE="$MAINTENANCE_MODE" \
+  --binding MAILCHIMP_API_KEY="$MAILCHIMP_API_KEY" \
+  --binding CLUSTER_API_URL="$CLUSTER_API_URL" \
+  --binding CLUSTER_BASIC_AUTH_TOKEN="$CLUSTER_BASIC_AUTH_TOKEN" \
+  --binding DEBUG="$DEBUG" \
+  --binding METAPLEX_AUTH_TOKEN="$METAPLEX_AUTH_TOKEN" \
+  --binding S3_ACCESS_KEY_ID="$S3_ACCESS_KEY_ID" \
+  --binding S3_BUCKET_NAME="$S3_BUCKET_NAME" \
+  --binding S3_ENDPOINT="$S3_ENDPOINT" \
+  --binding S3_REGION="$S3_REGION" \
+  --binding S3_SECRET_ACCESS_KEY="$S3_SECRET_ACCESS_KEY" \
+;
+


Suggested change

#!/usr/bin/env bash

npx miniflare --debug \

--build-command "npm run build" \

--binding ENV="$ENV" \

--binding PRIVATE_KEY="$PRIVATE_KEY" \

--binding DATABASE_CONNECTION="$DATABASE_CONNECTION" \

--binding DATABASE_TOKEN="$DATABASE_TOKEN" \

--binding DATABASE_URL="$DATABASE_URL" \

--binding LOGTAIL_TOKEN="$LOGTAIL_TOKEN" \

--binding MAGIC_SECRET_KEY="$MAGIC_SECRET_KEY" \

--binding SALT="$SALT" \

--binding SENTRY_DSN="$SENTRY_DSN" \

--binding MAINTENANCE_MODE="$MAINTENANCE_MODE" \

--binding MAILCHIMP_API_KEY="$MAILCHIMP_API_KEY" \

--binding CLUSTER_API_URL="$CLUSTER_API_URL" \

--binding CLUSTER_BASIC_AUTH_TOKEN="$CLUSTER_BASIC_AUTH_TOKEN" \

--binding DEBUG="$DEBUG" \

--binding METAPLEX_AUTH_TOKEN="$METAPLEX_AUTH_TOKEN" \

--binding S3_ACCESS_KEY_ID="$S3_ACCESS_KEY_ID" \

--binding S3_BUCKET_NAME="$S3_BUCKET_NAME" \

--binding S3_ENDPOINT="$S3_ENDPOINT" \

--binding S3_REGION="$S3_REGION" \

--binding S3_SECRET_ACCESS_KEY="$S3_SECRET_ACCESS_KEY" \

;

vasco-santos · 2022-07-20T12:51:15Z

packages/api/docker/scripts/run-playwright.sh

+#!/usr/bin/env bash
+
+tsc \
+&& npx playwright-test 'test/**/*.spec.js' --sw src/index.js \
+;


Suggested change

#!/usr/bin/env bash

tsc \

&& npx playwright-test 'test/**/*.spec.js' --sw src/index.js \

;

redaphid added 30 commits April 11, 2022 11:41

feat: Make the test environment data immutable so tests are repeatable

2acfbf4

Closes #1732

fix: add a tmp/ folder to .gitignore

6583f5c

token state manual testing for now

4f31e5f

continuing to write manual tests for state

cce6f7a

add simple, manual tests to see if state is being preserved between runs

c4991ec

Docker can run db-init script. It doesn't work, but it can run it

56e44ae

fix docker image to be the same as the sql we're using

3e25945

volume mount db_init script to test things out

88eae2e

put scripts I've been running to a script file so I don't have to rem…

025c8bf

…ember what they are

Miniflare listens in the docker container if you do this secret magic…

d56ed3e

… dance I'm trying to automate

remove tcp from host port exposure

4c63f56

move the helper scripts I use for testing into a scripts/ directory

c99b45a

fancier, more correct shebangs

6d06599

working on getting environment vars into miniflare

cff7a67

make script that uses env vars in miniflare cli

3dc4cb4

debugging env config; get env vars from root .env file

9d2d6f4

clean start print docker-compose config for debugging. run-miniflare.…

24c0e47

…sh as a way to pass env vars in

send env vars from .env->compose->miniflare. lol

2ec2e5a

cat env vars from docker container to .env file

28e44ca

enough env vars for miniflare to work!

a8a8cca

.env.tpl file updated to use docker-compose dns

1505c47

remove unneccessary port exposure as per @dashcraft's request :)

8c25067

prepping to execute db-sql.js directly

8152e18

revert to our hacked cli.js script so this is left in a working state

05f6fe3

no volume mounts; newer esbuild

2ee6d63

add .js to an import

ba5f9c2

just run the sql files directly for db init

1e484ec

working poc of constants injection via esbuild

59c0686

working poc of constants injection via esbuild that *only* templates …

0460f1a

…out the constants.js file

merge w/main so we don't diverge too far. #this branch is living too …

9d9adf1

…long :(

yusefnapora and others added 6 commits June 2, 2022 17:51

rm console logs

dffc744

rm commented out code

82c73d4

now with 100% less constants injection :)

4737976

add a dev:clean script to clean persistence

9929366

make a healthcheck for the database, so we don't rely on db-init's …

3d272d8

…exit code, as this was causing problems in the 'persist' mode. Added a `yarn dev:clean` to clean the volumes for the persist mode. Managed to remove one docker-compose file :)

move default yarn:test -> local. Add yarn:test:docker for using d…

0a979ee

…ocker to control this environment.

rm debug log

9998341

redaphid requested a review from vasco-santos June 7, 2022 18:01

vasco-santos reviewed Jun 8, 2022

View reviewed changes

redaphid added 7 commits June 13, 2022 14:14

Merge branch 'main' into feat/make-api-running-immutable

b043d10

Fix postres Dockerfile downgrade

669e3e9

rename post-rest->postgrest

8a46de3

post-rest->postgrest, add cross-fetch stuff back into build

06eadfe

rename scripts to reflect their respective yarn run commands

887a944

re-add deploy-dev to github ci

d1da8c4

Merge branch 'feat/make-api-running-immutable' of github.com:nftstora…

0970904

…ge/nft.storage into feat/make-api-running-immutable

redaphid requested review from vasco-santos and yusefnapora June 14, 2022 21:15

hugomrdias suggested changes Jun 15, 2022

View reviewed changes

redaphid added 6 commits June 16, 2022 13:26

re-add deploy-dev. I missed the actual key of the test name

a090356

line up env vars so the change is more obvious in this file

bd82712

line up env vars so the change is more obvious in this file

ec8e6ce

use main's yarn.lock

69c748d

minor whitespace change

9b0584c

add slack constant in to the env vars approach

6708db9

JeffLowe added the pi/support-1 label Jul 20, 2022

vasco-santos suggested changes Jul 20, 2022

View reviewed changes

yusefnapora mentioned this pull request Aug 10, 2022

chore: replace docker shell script with js CLI commands and use for dev and test environments #2130

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

redaphid commented May 3, 2022 •

edited

Loading

redaphid commented Jun 2, 2022

vasco-santos left a comment

vasco-santos Jun 8, 2022

vasco-santos Jun 8, 2022

yusefnapora Jun 10, 2022

vasco-santos Jun 14, 2022

hugomrdias Jun 15, 2022

vasco-santos Jun 8, 2022

vasco-santos Jun 8, 2022

yusefnapora Jun 10, 2022

vasco-santos Jun 8, 2022

vasco-santos Jun 8, 2022

redaphid commented Jun 14, 2022

hugomrdias left a comment

hugomrdias Jun 15, 2022

hugomrdias Jun 15, 2022

vasco-santos left a comment

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

vasco-santos Jul 20, 2022

		@@ -1,38 +1,40 @@
		# needs to be real so create a personal magic.link account or use the staging key
		MAGIC_SECRET_KEY=secret

		@@ -1,13 +1,7 @@
		FROM supabase/postgres:13.3.0
		FROM supabase/postgres:12.5.0

		"test": "./scripts/start-test-local.sh",
		"test:docker": "./scripts/start-test.sh ",

		"test": "./scripts/start-test.sh",
		"test:docker": "./scripts/start-test-docker.sh ",

feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

Are you sure you want to change the base?

feat: Allow developers to run API in docker, and select between mutable and immutable modes #1888

Conversation

redaphid commented May 3, 2022 • edited Loading

redaphid commented Jun 2, 2022

vasco-santos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redaphid commented Jun 14, 2022

hugomrdias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redaphid commented May 3, 2022 •

edited

Loading