Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres error "SAVEPOINT can only be used in transaction blocks" when used with live_server #17

Open
citizen-stig opened this issue Apr 15, 2019 · 7 comments

Comments

@citizen-stig
Copy link

I have tests, that I would like to run with live_server from pytest-flask package.

I thought it works ok, but only for single request. Full repo with working example, failing test

Here are some pieces of the code:

# models.py
class Category(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String, unique=True, nullable=False)

# view
@blog.route('/categories')
def list_categories():
    return '<br/>'.join(str(x.id) + ': ' + x.name for x in models.Category.query)

factories from conftest.py

@pytest.fixture(scope='function')
def base_factory(db_session):
    class BaseFactory(factory.alchemy.SQLAlchemyModelFactory):
        class Meta:
            abstract = True
            sqlalchemy_session = db_session
            sqlalchemy_session_persistence = 'flush'
    return BaseFactory

@pytest.fixture(scope='function')
def category_factory(base_factory):
    class CategoryFactory(base_factory):
        class Meta:
            model = models.Category
        name = factory.Sequence(lambda n: u'Category %d' % n)
    return CategoryFactory

Test itself:

from flask import url_for
import requests

from demo_app import models

def test_list_categories(live_server, category_factory):
    categories = category_factory.create_batch(5)

    assert models.Category.query.count() == 5

    url = url_for('blog.list_categories', _external=True)

    response_1 = requests.get(url)

    assert response_1.status_code == 200
    data_1 = response_1.content.decode('utf-8')
    for category in categories:
        assert category.name in data_1

    assert models.Category.query.count() == 5

    response_2 = requests.get(url)

    assert response_2.status_code == 200
    data_2 = response_2.content.decode('utf-8')
    print(data_2)
    for category in categories:
        assert category.name in data_2

I'm getting error on line assert models.Category.query.count() == 5, or if I comment it out, data_2 data returned by the server is empty.

The error:

self = <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x104135780>
cursor = <cursor object at 0x1040ea900; closed: -1>
statement = 'SAVEPOINT sa_savepoint_23', parameters = {}
context = <sqlalchemy.dialects.postgresql.psycopg2.PGExecutionContext_psycopg2 object at 0x104321048>

    def do_execute(self, cursor, statement, parameters, context=None):
>       cursor.execute(statement, parameters)
E       sqlalchemy.exc.InternalError: (psycopg2.errors.NoActiveSqlTransaction) SAVEPOINT can only be used in transaction blocks
E       
E       [SQL: SAVEPOINT sa_savepoint_23]
E       (Background on this error at: http://sqlalche.me/e/2j85)

Full stack trace

Could you please help me, I don't know where to start to investigate this.

@jeancochrane
Copy link
Owner

jeancochrane commented Apr 19, 2019

This is a pretty interesting problem. I don't have much experience with live_server so I don't really know how it works, but one thing I'm curious about is how it handles SQLAlchemy connections in the process that runs the server (which I'm guessing gets forked from the pytest process). The SQLAlchemy docs are pretty clear that connections shouldn't be shared across processes:

The SQLAlchemy Engine object refers to a connection pool of existing database connections. So when this object is replicated to a child process, the goal is to ensure that no database connections are carried over.

This makes me wonder whether live_server is doing some work under the hood to dispose of the connection, which could result in the confusion here where a transaction context is expected but not found. Alternatively, perhaps live_server isn't doing anything at all to handle the shared connection, and we need to be on the lookout for potential multiprocessing on our end and proactively create a new transactional context. Either way, the pytest-flask docs don't say much about this, so we'd probably need to take a look at the internals to see what's going on.

Another interesting lead is that I can't actually replicate the error you're seeing. Instead, I see an error that gets raised when the live_server thread tries to run model.Category.query:

*** sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac

This SO thread seems to see a similar error in a multiprocessing context.

Sorry if that isn't a very helpful answer at this stage. I should have some more time next week to dig into this more deeply if you're still having problems.

@citizen-stig
Copy link
Author

@jeancochrane Thanks for such detailed explanation, and any help is appreciated!

That is strange, that error is different. I used pipenv, so all exact versions should be presented in Pipenv.lock I run Postgresl on docker, maybe I should build docker-compose environment, so it will be easier to reproduce.

@jeancochrane
Copy link
Owner

jeancochrane commented Apr 24, 2019 via email

@samuelhwilliams
Copy link

samuelhwilliams commented Jun 6, 2019

I'm trialling out this plugin to see if I can get tests isolated in transactions in a repo I work with. After a few hours of fiddling with the main unit tests, and getting them all passing, I'm now onto the functional tests using the live_server. I'm getting the same error as OP. :(

@citizen-stig
Copy link
Author

@jeancochrane , I've finally managed to create docker environment, repository is updated:
https://github.com/citizen-stig/flask-pytest-samples

just use docker-compose up and error will be printed in console

@jeancochrane
Copy link
Owner

Awesome, thanks @citizen-stig! I'll have time to do some testing this weekend.

@msharaf315
Copy link

any upadtes on this im facing a similar issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants