Programmatic upgrade using async db driver

[alejoar]

I want my app to auto upgrade the db every time it starts up. For that I followed instructions provided by @zzzeek here: #805 (although to be fair I'm not sure I followed properly)

I created a minimal example with my issue here: https://github.com/alejoar/fastapi-async-sqlalchemy-alembic

These work (from the cli):

alembic revision --autogenerate
alembic upgrade head

This doesn't work (from the code):

In database.py

def run_upgrade(connection, cfg):
    cfg.attributes["connection"] = connection
    command.upgrade(cfg, "head")


async def run_async_upgrade():
    async with async_engine.begin() as conn:
        await conn.run_sync(run_upgrade, config.Config("alembic.ini"))

In main.py:

(this runs on app start up)

@app.on_event("startup")
async def on_startup():
    await run_async_upgrade()

[zzzeek]

Place this function in your env.py script:

def run_migrations_online():
    connectable = config.attributes.get('connection', None)

    if connectable is None:
        # only create Engine if we don't have a Connection
        # from the outside
        connectable = engine_from_config(
            config.get_section(config.config_ini_section),
            prefix='sqlalchemy.',
            poolclass=pool.NullPool)

        with connectable.connect() as connection:
            context.configure(
                connection=connection, target_metadata=target_metadata
            )

            with context.begin_transaction():
                context.run_migrations()
    else:
        context.configure(
            connection=connectable,
            target_metadata=target_metadata
        )

        with context.begin_transaction():
            context.run_migrations()

then run this script as follows:

import asyncio
from sqlalchemy.ext.asyncio import create_async_engine

from alembic import command, config


def run_upgrade(connection, cfg):
    cfg.attributes["connection"] = connection
    command.upgrade(cfg, "head")


async def run_async_upgrade():
    async_engine = create_async_engine("sqlite+aiosqlite://", echo=True)
    async with async_engine.begin() as conn:
        await conn.run_sync(run_upgrade, config.Config("alembic.ini"))


asyncio.run(run_async_upgrade())

for the flask part, you'd need to ask flask people for that.

[zzzeek]

oh and you want to use the async database within env.py as well? OK then youi have to combine that recipe with the other one at https://alembic.sqlalchemy.org/en/latest/cookbook.html#using-asyncio-with-alembic , that's a little tricky (tricky. yeah). we need to put the asyncio.run() inside of run_migrations_online(). This works both ways:

# env.py

import asyncio
from logging.config import fileConfig

from sqlalchemy import pool
from sqlalchemy.ext.asyncio import async_engine_from_config
from alembic import context

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)

target_metadata = None

def run_migrations_offline():
    url = config.get_main_option("sqlalchemy.url")
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def do_run_migrations(connection):
    context.configure(connection=connection, target_metadata=target_metadata)

    with context.begin_transaction():
        context.run_migrations()


async def run_async_migrations():
    connectable = async_engine_from_config(
        config.get_section(config.config_ini_section),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    async with connectable.connect() as connection:
        await connection.run_sync(do_run_migrations)

    await connectable.dispose()


def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """

    connectable = config.attributes.get("connection", None)

    if connectable is None:
        asyncio.run(run_async_migrations())
    else:
        do_run_migrations(connectable)


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

then test.py as before still works:

import asyncio
from sqlalchemy.ext.asyncio import create_async_engine

from alembic import command, config


def run_upgrade(connection, cfg):
    cfg.attributes["connection"] = connection
    command.upgrade(cfg, "head")


async def run_async_upgrade():
    async_engine = create_async_engine("sqlite+aiosqlite://", echo=True)
    async with async_engine.begin() as conn:
        await conn.run_sync(run_upgrade, config.Config("alembic.ini"))


asyncio.run(run_async_upgrade())

as does "alembic upgrade"

[zzzeek]

i've pushed this recipe up to the cookbook section in the docs

[alejoar]

That works perfectly! Thank you @zzzeek

[lawalAfeez820]

asyncio.run(run_async_upgrade()) is return error

untimeError: asyncio.run() cannot be called from a running event loop
sys:1: RuntimeWarning: coroutine 'main' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

please help with this

[RafRaf]

It works for me.

@lawalAfeez820 Use the code snippet above. Updated run_migrations_online func is the key. Thank you @zzzeek.

I guess this is your run_migrations_online (in env.py)?

def run_migrations_online() -> None:
    """Run migrations in 'online' mode."""

    asyncio.run(run_async_migrations())

Use this one instead:

def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """

    connectable = config.attributes.get("connection", None)

    if connectable is None:
        asyncio.run(run_async_migrations())
    else:
        do_run_migrations(connectable)