Skip to content

Commit

Permalink
Dev to Staging: Agora 3.5.0 and Data mv71 (#1343)
Browse files Browse the repository at this point in the history 
* AG-1471: install synapseclient, download release data from Synpase

* AG-1473: create mongo image containing release data

* AG-1474: run e2e tests

* AG-1471: do not run job on pull_request trigger since forks do not have access to secrets

* AG-1472: check if image exists in GHCR, then build/push image if needed

* AG-1472: split workflow into three jobs

* AG-1472: handle case where package has not yet been created

* AG-1472: remove forward slash from package name so gh call works

* AG-1472: ensure e2e tests are run when image has already been built

* AG-1472: improvements from code review

* Bump braces from 3.0.2 to 3.0.3

Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3.
- [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md)
- [Commits](micromatch/braces@3.0.2...3.0.3)

---
updated-dependencies:
- dependency-name: braces
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

* AG-1472: ensure org package is lowercase

* AG-1498: support use of containerized data image in local development [skip ci]

* AG-1330: combine workflows into one workflow with multiple jobs

* AG-1518: rename data release image to distinguish from sage-monorepo image

* Bump fast-xml-parser and @aws-sdk/credential-providers

Bumps [fast-xml-parser](https://github.com/NaturalIntelligence/fast-xml-parser) and [@aws-sdk/credential-providers](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/packages/credential-providers). These dependencies needed to be updated together.

Updates `fast-xml-parser` from 4.2.5 to 4.4.1
- [Release notes](https://github.com/NaturalIntelligence/fast-xml-parser/releases)
- [Changelog](https://github.com/NaturalIntelligence/fast-xml-parser/blob/master/CHANGELOG.md)
- [Commits](NaturalIntelligence/fast-xml-parser@v4.2.5...v4.4.1)

Updates `@aws-sdk/credential-providers` from 3.398.0 to 3.624.0
- [Release notes](https://github.com/aws/aws-sdk-js-v3/releases)
- [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/packages/credential-providers/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.624.0/packages/credential-providers)

---
updated-dependencies:
- dependency-name: fast-xml-parser
  dependency-type: indirect
- dependency-name: "@aws-sdk/credential-providers"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump to mv70 (display) in dev

* AG-1395: Fix typo in manifest version value

* removed druggability section in gene details resources tab

* added new resource cards in Resources tab

* adding pharos class to hero

* updated section title from PHAROS to PHAROS CLASS

* Adjust ordering of resource sections

* Update resource strings and URLs

* Update mongoDB install URLs in README

* adding uniprotkb_accessions links

* applied a margin under subheadings biodomains and pharos class

* adding margin to subitems in hero

* added new uniprotkb_accessions property to mocks

* Update ci.yml

updating docker-compose to docker compose

* AG-1565 remove druggability columns

* AG-1599 remove pharos class

* fix pharos class schema bug

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Hallie Swan <[email protected]>
Co-authored-by: sagely1 <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jessica Britton <[email protected]>
Co-authored-by: JessterB <[email protected]>
  • Loading branch information
@beatrizsaldana @hallieswan
@sagely1 @dependabot @JessterB
6 people authored Dec 18, 2024
1 parent 5f1a19a commit 9a45720
Show file tree
Hide file tree
Showing 30 changed files with 2,719 additions and 2,074 deletions.
5 changes: 5 additions & 0 deletions .env.example
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
DB_USER="root"
DB_PASS="${DB_PASS}"
DB_PORT="27015"
DB_NAME="agora"
DATA_IMAGE_PATH="ghcr.io/sage-bionetworks/agora-data-nonmonorepo:${DATA_IMAGE_TAG}"
247 changes: 247 additions & 0 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
name: Build and run unit tests, build data image and run e2e tests

on: push

jobs:
build-and-run-unit-tests:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 16
cache: 'npm'
- run: npm ci
- run: APP_ENV=develop npm run ci:build:develop
- run: npm test

check-for-image:
# Run in Sage repo on develop or main branches
# and on all branches in user-owned forks
if: ${{ github.ref_name == 'develop' || github.ref_name == 'main' || github.actor == github.repository_owner }}
timeout-minutes: 5
runs-on: ubuntu-20.04
permissions:
packages: read
environment: e2e
env:
ORG_NAME: 'Sage-Bionetworks'
NAMESPACE: ${{ github.repository_owner }}
DATA_IMAGE_NAME: '${{ vars.DB_NAME }}-data-nonmonorepo'
outputs:
DATA_MANIFEST_ID: ${{ steps.get-ids-from-package-json.outputs.DATA_MANIFEST_ID }}
DATA_VERSION: ${{ steps.get-ids-from-package-json.outputs.DATA_VERSION }}
DATA_IMAGE_PATH: ${{ steps.get-ids-from-package-json.outputs.DATA_IMAGE_PATH }}
IMAGE_EXISTS: ${{ steps.check-container-repo.outputs.IMAGE_EXISTS }}
steps:
- uses: actions/checkout@v4

- name: get data image tag, data manifest id, and data version
id: get-ids-from-package-json
env:
PACKAGE_JSON_PATH: 'package.json'
run: |
DATA_MANIFEST_ID=$(jq -r '."data-file"' "${{ env.PACKAGE_JSON_PATH }}")
echo "DATA_MANIFEST_ID=${DATA_MANIFEST_ID}" >> "$GITHUB_OUTPUT"
DATA_VERSION=$(jq -r '."data-version"' "${{ env.PACKAGE_JSON_PATH }}")
echo "DATA_VERSION=${DATA_VERSION}" >> "$GITHUB_OUTPUT"
DATA_IMAGE_TAG="${DATA_MANIFEST_ID}.${DATA_VERSION}"
echo "DATA_IMAGE_TAG=${DATA_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
# GHCR requires image names to be lowercase: https://github.com/orgs/community/discussions/27086
NAMESPACE_LOWERCASE=$(echo "${{ env.NAMESPACE }}" | tr '[:upper:]' '[:lower:]')
DATA_IMAGE_PATH="ghcr.io/${NAMESPACE_LOWERCASE}/${{ env.DATA_IMAGE_NAME }}:${DATA_IMAGE_TAG}"
echo "DATA_IMAGE_PATH=${DATA_IMAGE_PATH}" >> "$GITHUB_OUTPUT"
- name: login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: check whether image exists
id: check-container-repo
env:
# https://docs.github.com/en/rest/packages/packages?apiVersion=2022-11-28#list-package-versions-for-a-package-owned-by-an-organization
ORG_PATH: '/orgs/${{ env.ORG_NAME }}/packages/container/${{ env.DATA_IMAGE_NAME }}/versions'
# https://docs.github.com/en/rest/packages/packages?apiVersion=2022-11-28#list-package-versions-for-a-package-owned-by-the-authenticated-user
USER_PATH: '/user/packages/container/${{ env.DATA_IMAGE_NAME }}/versions'
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DATA_IMAGE_TAG: ${{ steps.get-ids-from-package-json.outputs.DATA_IMAGE_TAG }}
run: |
GH_PATH=$( [[ "${{ env.NAMESPACE }}" == "${{ env.ORG_NAME }}" ]] && echo "${{ env.ORG_PATH }}" || echo "${{ env.USER_PATH }}" )
# allow error to handle case where gh throws an error when package has not yet been created
set +e
IMAGE_VERSIONS=$(gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" "${GH_PATH}")
set -e
PACKAGE_NOT_FOUND=$(echo "${IMAGE_VERSIONS}" | jq '.message? == "Package not found."')
if [[ "${PACKAGE_NOT_FOUND}" == "true" ]]; then
IMAGE_EXISTS="false"
else
IMAGE_EXISTS=$(echo "${IMAGE_VERSIONS}" | jq '[.[] | select(any(.metadata.container.tags[]; . == "${{ env.DATA_IMAGE_TAG }}"))] | length > 0')
fi
echo "IMAGE_EXISTS=${IMAGE_EXISTS}" >> "$GITHUB_OUTPUT"
build-image:
needs: check-for-image
if: ${{ needs.check-for-image.result == 'success' && needs.check-for-image.outputs.IMAGE_EXISTS == 'false' }}
timeout-minutes: 60
runs-on: ubuntu-20.04
permissions:
packages: write
environment: e2e
env:
LOCAL_DATA_PATH: './local/data'
LOCAL_TEAM_IMAGES_DIR: './local/data/team_images'
LOCAL_COLLECTIONS_PATH: './scripts/collections.csv'
LOCAL_IMPORT_SCRIPT_PATH: './docker/import-data.sh'
LOCAL_CREATE_INDEXES_PATH: './scripts/mongo-create-Indexes.js'
DATA_VERSION: ${{ needs.check-for-image.outputs.DATA_VERSION }}
DATA_MANIFEST_ID: ${{ needs.check-for-image.outputs.DATA_MANIFEST_ID }}
DATA_IMAGE_PATH: ${{ needs.check-for-image.outputs.DATA_IMAGE_PATH }}
steps:
- uses: actions/checkout@v4

# https://github.com/actions/setup-python/issues/807
- name: create placeholder requirements.txt file
run: touch ./requirements.txt

- uses: actions/setup-python@v5
with:
python-version: '3.9'
cache: 'pip'

- name: install synapseclient
run: |
pip install --upgrade synapseclient
- name: download data from synapse
timeout-minutes: 10
env:
SYNAPSE_AUTH_TOKEN: ${{ secrets.SYNAPSE_AUTH_TOKEN }}
run: |
echo "==> make directory for team images"
mkdir -p "${{ env.LOCAL_TEAM_IMAGES_DIR }}"
echo "==> download manifest file from Synapse"
synapse get --downloadLocation "${{ env.LOCAL_DATA_PATH }}" -v "${{ env.DATA_VERSION }}" "${{ env.DATA_MANIFEST_ID }}"
echo "==> download all files referenced in manifest from Synapse"
cat "${{ env.LOCAL_DATA_PATH }}"/data_manifest.csv | tail -n +2 | while IFS=, read -r id version; do
echo "===> Downloading ${id}.${version}"
synapse get --downloadLocation "${{ env.LOCAL_DATA_PATH }}" -v "${version}" "${id}"
done
echo "==> download team images"
synapse get -r --downloadLocation "${{ env.LOCAL_TEAM_IMAGES_DIR }}/" "${{ vars.TEAM_IMAGES_ID }}"
- name: build image
env:
DOCKERFILE_PATH: './docker/Dockerfile'
run: |
docker build \
-t "${{ env.DATA_IMAGE_PATH }}" \
-f "${{ env.DOCKERFILE_PATH }}" \
--build-arg SOURCE_REPO="https://github.com/${{ github.repository }}" \
--build-arg DESCRIPTION="${GITHUB_SHA::7}" \
--build-arg LOCAL_DATA_PATH="${{ env.LOCAL_DATA_PATH }}" \
--build-arg LOCAL_COLLECTIONS_PATH="${{ env.LOCAL_COLLECTIONS_PATH }}" \
--build-arg LOCAL_IMPORT_SCRIPT_PATH="${{ env.LOCAL_IMPORT_SCRIPT_PATH }}" \
--build-arg LOCAL_CREATE_INDEXES_PATH="${{ env.LOCAL_CREATE_INDEXES_PATH }}" \
.
- name: login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

# Note: if a new package is created using this workflow, then the package
# visibility will default to the source repository visibility. For example,
# a package created by a workflow in a public repo will also be public.
# Publicly visible container packages are anonymously downloadable.
- name: push to GitHub Container Registry
run: docker push "${{ env.DATA_IMAGE_PATH }}"

run-e2e-tests:
needs:
- check-for-image
- build-image
# always() prevents skipping this job when build-image was skipped
# see https://github.com/actions/runner/issues/491
if: ${{ always() && needs.check-for-image.result == 'success' && (needs.build-image.result == 'success' || needs.build-image.result == 'skipped')}}
timeout-minutes: 60
runs-on: ubuntu-20.04
permissions:
packages: read
environment: e2e
env:
CI: true
DOCKER_COMPOSE_PATH: './docker/docker-compose.yml'
DATA_IMAGE_PATH: ${{ needs.check-for-image.outputs.DATA_IMAGE_PATH }}
DB_USER: ${{ secrets.DB_USER }}
DB_PASS: ${{ secrets.DB_PASS }}
DB_PORT: ${{ vars.DB_PORT }}
DB_NAME: ${{ vars.DB_NAME }}
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: 16
cache: 'npm'

- name: install dependencies
run: npm ci

- name: install Playwright Browsers
run: npx playwright install --with-deps

- name: login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: start database, server, and app
run: |
echo "==> start database and data containers"
docker compose -f "${{ env.DOCKER_COMPOSE_PATH }}" up -d
echo "==> wait for data container to finish seeding database container"
DATA_CONTAINER=$(docker compose -f "${{ env.DOCKER_COMPOSE_PATH }}" ps -a --format '{{.Name}}' mongo-seed)
docker wait "${DATA_CONTAINER}"
echo "==> start server and app in background and wait for server to grab port"
MONGODB_HOST="localhost" MONGODB_PORT="${{ env.DB_PORT }}" APP_ENV="e2e" npm run dev >/dev/null 2>&1 &
sleep 5
- name: run e2e tests
run: npx playwright test

- name: clean up database, server, and app
run: |
echo "==> stop app"
pid=$(lsof -i :3000 -t) && kill ${pid}
echo "==> stop server"
pid=$(lsof -i :8080 -t) && kill ${pid}
echo "==> stop database"
docker compose -f "${{ env.DOCKER_COMPOSE_PATH }}" down
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 5
55 changes: 0 additions & 55 deletions .github/workflows/node.js.yml
View file

This file was deleted.

4 changes: 4 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,7 @@ testem.log
# System files
.DS_Store
Thumbs.db

# Environment file
.env

Loading

0 comments on commit 9a45720

Please sign in to comment.