Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate photo storage to GCloud storage #273

Merged
merged 16 commits into from
May 9, 2023
Merged

Migrate photo storage to GCloud storage #273

merged 16 commits into from
May 9, 2023

Conversation

vnugent
Copy link
Contributor

@vnugent vnugent commented Apr 26, 2023

Issues:

TBD in a follow up PR:

Migration steps (I'll test this in staging first)


New data model

  1. Media object collection is a 1-1 mapping with objects on GCloud bucket
  2. rawTags is an embedded array of Tags eliminating the need to do a lookup (SQL join)
  3. Still need to join raw tags with both Climbs and Areas collection to get the tagged climb/area name

new-data-model

@vnugent
Copy link
Contributor Author

vnugent commented May 3, 2023

@zichongkao @musoke @CocoisBuggy
I wanted you to be aware of the new data model for media & tags. This is a large PR because media & tags touch a lot of the code. I decide to "rip the bandaid off" and simplify/unify multiple TS types of tags/media into one 'MediaWithTags'.

All frontend breaking changes are fixed in this PR: OpenBeta/open-tacos#800

@vnugent vnugent marked this pull request as draft May 3, 2023 09:49
@vnugent vnugent requested a review from CocoisBuggy May 3, 2023 12:43
Copy link
Contributor

@musoke musoke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a huge set of changes! Looks like a more sustainable base for the future 😃

@vnugent
Copy link
Contributor Author

vnugent commented May 7, 2023

I did a dry-run import of production data locally and can confirm that existing tags were migrated to the new collection.

Tests: https://github.com/OpenBeta/openbeta-graphql/blob/f8ea4bfd9e514d97607b4135b7d6604b4c0408f4/src/db/utils/jobs/migration/Tests.ts

# node  build/db/utils/jobs/migration/Tests.js
...
{"level":30,"oldTagCount":619,"newTagCount":619,"result":"PASS","msg":"Old vs new tag count"}
{"level":30,"found":[],"result":"PASS","msg":"Orphane IDs check"}

chore: rename media GQL types & queries

bug: fix latlng bug
@vnugent vnugent marked this pull request as ready for review May 9, 2023 00:25
@vnugent vnugent merged commit c68ada1 into develop May 9, 2023
1 check passed
l4u532 pushed a commit to l4u532/openbeta-graphql that referenced this pull request Dec 10, 2023
* chore: extract & save image width/height during migration
* chore: resolve username in the backend instead of frontend
* chore: return image metadata to tag queries
* chore: climb/area/user profile media share the same type
* refactor: embed tags in media object collection
* refactor: update find-tags-by-area-id to work with embedded tags
* refactor: move leaderboard api to new embedded tags model
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants