Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace legacy custom db migration system with alembic #639

Draft
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

cmyui
Copy link
Member

@cmyui cmyui commented Feb 26, 2024

Describe your changes

Related Issues / Projects

Checklist

  • I've manually tested my code

app/settings.py Outdated
@@ -18,7 +18,7 @@
DB_USER = os.environ["DB_USER"]
DB_PASS = os.environ["DB_PASS"]
DB_NAME = os.environ["DB_NAME"]
DB_DSN = f"mysql://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
DB_DSN = f"mysql+pymysql://{DB_USER}:{DB_PASS}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to undo this/move it to be alembic-specific; we want aiomysql at runtime

@cmyui
Copy link
Member Author

cmyui commented Feb 26, 2024

Diffs observed from generating the base schema

INFO  [alembic.autogenerate.compare] Detected removed table 'relationships'
INFO  [alembic.autogenerate.compare] Detected removed table 'startups'
INFO  [alembic.autogenerate.compare] Detected removed index 'nmapsets_id_uindex' on 'mapsets'
INFO  [alembic.autogenerate.compare] Detected removed table 'mapsets'
INFO  [alembic.autogenerate.compare] Detected removed table 'performance_reports'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'achievements.name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=256) to String(length=256, collation='utf8') on 'achievements.desc'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=16) to String(length=16, collation='utf8') on 'clans.name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=6) to String(length=6, collation='utf8') on 'clans.tag'
INFO  [alembic.autogenerate.compare] Detected changed index 'clans_name_uindex' on 'clans': unique=True to unique=False
INFO  [alembic.autogenerate.compare] Column 'comments.target_id' has no type within the model; can't compare
INFO  [alembic.autogenerate.compare] Detected column comment 'comments.target_id'
INFO  [alembic.autogenerate.compare] Detected type change from ENUM('replay', 'map', 'song') to Enum('REPLAY', 'BEATMAP', 'SONG', name='target_type') on 'comments.target_type'
INFO  [alembic.autogenerate.compare] Detected type change from INTEGER() to FLOAT(precision=6, scale=3) on 'comments.time'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=80) to String(length=80, collation='utf8') on 'comments.comment'
INFO  [alembic.autogenerate.compare] Detected column comment 'comments.colour'
INFO  [alembic.autogenerate.compare] Detected column comment 'ingame_logins.ip'
INFO  [alembic.autogenerate.compare] Detected column comment 'logs.from'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'logs.msg'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'mail.msg'
INFO  [alembic.autogenerate.compare] Detected type change from ENUM('osu!', 'private') to Enum('OSU', 'PRIVATE', name='server') on 'maps.server'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'maps.md5'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.artist'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.title'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=128) to String(length=128, collation='utf8') on 'maps.version'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=19) to String(length=19, collation='utf8') on 'maps.creator'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=256) to String(length=256, collation='utf8') on 'maps.filename'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'ratings.map_md5'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'scores.id'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'scores.map_md5'
INFO  [alembic.autogenerate.compare] Detected type change from FLOAT(precision=7, scale=3) to FLOAT(precision=6, scale=3) on 'scores.pp'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'scores.status'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'scores.mode'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=32) to String(length=32) on 'scores.online_checksum'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'stats.tscore'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT(unsigned=True) to Integer() on 'stats.rscore'
INFO  [alembic.autogenerate.compare] Detected type change from BIGINT() to Integer() on 'tourney_pool_maps.mods'
INFO  [alembic.autogenerate.compare] Detected type change from TINYINT() to Integer() on 'tourney_pool_maps.slot'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(length=64) to String(length=16) on 'tourney_pools.name'
INFO  [alembic.autogenerate.compare] Detected type change from MEDIUMTEXT() to String(length=32, collation='utf8') on 'users.name'
INFO  [alembic.autogenerate.compare] Detected type change from MEDIUMTEXT() to String(length=32, collation='utf8') on 'users.safe_name'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=60) to String(length=60) on 'users.pw_bcrypt'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=2) to String(length=2) on 'users.country'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=16) to String(length=16, collation='utf8') on 'users.custom_badge_name'
INFO  [alembic.autogenerate.compare] Detected type change from VARCHAR(charset='utf8mb3', collation='utf8mb3_general_ci', length=2048) to String(length=2048, collation='utf8') on 'users.userpage_content'
INFO  [alembic.autogenerate.compare] Detected type change from CHAR(length=36) to String(length=36) on 'users.api_key'
INFO  [alembic.autogenerate.compare] Detected removed index 'users_irc_key_uindex' on 'users'
INFO  [alembic.autogenerate.compare] Detected added index ''users_name_uindex'' on '('name',)'
INFO  [alembic.autogenerate.compare] Detected added index ''users_safe_name_uindex'' on '('safe_name',)'
INFO  [alembic.autogenerate.compare] Detected removed column 'users.irc_key'

(This is basically the todo list for this pr)

@cmyui cmyui changed the title Base alembic working Replace legacy custom migration system with alembic Feb 26, 2024
@cmyui cmyui changed the title Replace legacy custom migration system with alembic Replace legacy custom db migration system with alembic Feb 26, 2024
alembic.ini Outdated Show resolved Hide resolved

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can set this up to just defer to logging.yaml

# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could use pre-commit hooks here? We could probably call into quite a few of them directly if we really had to. Maybe we'll find it's not worth using this & pre-commit after the fact is ok

Copy link
Member Author

@cmyui cmyui Feb 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: this is still missing the existing seed data, as well as a couple of the tables that already exist (we should really have a 1:1 representation of the existing db in this PR, so that it gives us the ability to use alembic to remove those existing parts of the schema in future PRs -- otherwise we'll have a desync between user's db states & where alembic thinks they are).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also now missing index from #642

@NiceAesth
Copy link
Contributor

this would be epic to get in

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants