-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding MySQL support for db fixtures #10
base: master
Are you sure you want to change the base?
Changes from 6 commits
0adba51
e179192
aeb15c8
90a37c9
cbe1306
db4de7a
7608cf4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,10 +10,17 @@ python: | |
- '3.6' | ||
|
||
env: | ||
- TEST_DATABASE_URL=postgresql://postgres@localhost:5432/pytest_test | ||
- DB=pgsql TEST_DATABASE_URL=postgresql://postgres@localhost:5432/pytest_test | ||
- DB=mysql TEST_DATABASE_URL=mysql+mysqldb://[email protected]/pytest_test | ||
|
||
addons: | ||
postgresql: '9.6' | ||
mysql: '5.7' | ||
|
||
before_script: | ||
- sh -c "if [ '$DB' = 'postgres' ]; then psql -c 'DROP DATABASE IF EXISTS pytest_test;' -U postgres; fi" | ||
- sh -c "if [ '$DB' = 'postgres' ]; then psql -c 'CREATE DATABASE pytest_test;' -U postgres; fi" | ||
- sh -c "if [ '$DB' = 'mysql' ]; then mysql -e 'CREATE DATABASE IF NOT EXISTS pytest_test;'; fi" | ||
|
||
install: | ||
- pip install --upgrade pip | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -102,6 +102,9 @@ def _engine(pytestconfig, request, _transaction, mocker): | |
# the Engine dialect to reflect tables) | ||
engine.dialect = connection.dialect | ||
|
||
# Necessary for branching db test code | ||
engine.name = connection.engine.name | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice, this is a clever way to introspect the database engine. |
||
|
||
@contextlib.contextmanager | ||
def begin(): | ||
''' | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -6,6 +6,7 @@ | |||||
import sqlalchemy as sa | ||||||
from flask import Flask | ||||||
from flask_sqlalchemy import SQLAlchemy | ||||||
from pytest_mysql.factories import mysql | ||||||
from pytest_postgresql.factories import (init_postgresql_database, | ||||||
drop_postgresql_database) | ||||||
|
||||||
|
@@ -17,7 +18,8 @@ | |||||
'connection string to the environmental variable ' + | ||||||
'TEST_DATABASE_URL in order to run tests.') | ||||||
else: | ||||||
DB_OPTS = sa.engine.url.make_url(DB_CONN).translate_connect_args() | ||||||
DB_URL = sa.engine.url.make_url(DB_CONN) | ||||||
DB_OPTS = DB_URL.translate_connect_args() | ||||||
|
||||||
pytest_plugins = ['pytest-flask-sqlalchemy'] | ||||||
|
||||||
|
@@ -27,16 +29,28 @@ def database(request): | |||||
''' | ||||||
Create a Postgres database for the tests, and drop it when the tests are done. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
''' | ||||||
pg_host = DB_OPTS.get("host") | ||||||
pg_port = DB_OPTS.get("port") | ||||||
pg_user = DB_OPTS.get("username") | ||||||
pg_db = DB_OPTS["database"] | ||||||
db_host = DB_OPTS.get("host") | ||||||
db_port = DB_OPTS.get("port") | ||||||
db_user = DB_OPTS.get("username") | ||||||
db_name = DB_OPTS["database"] | ||||||
|
||||||
init_postgresql_database(pg_user, pg_host, pg_port, pg_db) | ||||||
BACKEND = DB_URL.get_backend_name() | ||||||
|
||||||
@request.addfinalizer | ||||||
def drop_database(): | ||||||
drop_postgresql_database(pg_user, pg_host, pg_port, pg_db, 9.6) | ||||||
if BACKEND == 'mysql': | ||||||
mysql('dummy_mysql_fixture', db=db_name) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems like the The docs suggest that the It might be simpler to just use a Python MySQL client (like Let me know if you're having trouble with this and I'd be glad to give it a try! I don't have a lot of MySQL experience so I'll leave it up to you whether or not you want to take a stab at it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great notes, I'll confess I am very new to the pytest framework so I was overwhelmed with all the different interactions taking place. Having you clear up what needs to be done is helping me focus. Let me work on it a little and then we can have a look to refine it further. |
||||||
|
||||||
elif BACKEND == 'postgresql': | ||||||
init_postgresql_database(db_user, db_host, db_port, db_name) | ||||||
|
||||||
@request.addfinalizer | ||||||
def drop_database(): | ||||||
drop_postgresql_database(db_user, db_host, db_port, db_name, 9.6) | ||||||
|
||||||
else: | ||||||
raise ValueError( | ||||||
'Unsupported database type ({}) requested in ' | ||||||
'TEST_DATABASE_URL: {}'.format(BACKEND, DB_URL) | ||||||
) | ||||||
|
||||||
|
||||||
@pytest.fixture(scope='session') | ||||||
|
@@ -47,6 +61,7 @@ def app(database): | |||||
app = Flask(__name__) | ||||||
|
||||||
app.config['SQLALCHEMY_DATABASE_URI'] = DB_CONN | ||||||
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's your reasoning behind adding this config? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh yes, essentially SQLAlchemy pollutes the log with deprecation warning messages because default behaviour is changing. I noticed it when I was running As to why I set it to True, just because it preserved existing behaviour. That being said, it is not required by the library, and disabling it/leaving it alone works fine. So I'll remove this statement as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great, thanks for the context! I'm glad you flagged this, we should address deprecations in a separate issue. |
||||||
|
||||||
return app | ||||||
|
||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -212,33 +212,57 @@ def test_drop_table(db_testdir): | |
''' | ||
Make sure that we can drop tables and verify they do not exist in the context | ||
of a test. | ||
|
||
NOTE: For MySQL `DROP TABLE ...` statements "cause an implicit commit after | ||
executing. The intent is to handle each such statement in its own special | ||
transaction because it cannot be rolled back anyway." | ||
https://dev.mysql.com/doc/refman/5.7/en/implicit-commit.html | ||
|
||
So this test is skipped for 'mysql' engines | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What happens if you don't skip this test in MySQL? I would expect an autocommit not to cause a problem here, since the outermost transaction is supposed to listen for nested inner commits and restart transactions when they occur. Maybe there's something about "implicit commits" in MySQL that overrides nested transactions? I'd like to get to the bottom of this behavior, since it seems pretty restricting to not be able to run any of the statements that are listed on the MySQL docs for implicit commits ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The reason I started looking into it was because the tests were failing to begin with, and I was really unaware as to why. I just thought to check that there might be a Transaction violation for DROP TABLE, and the MySQL docs is what I found in my search results. Quoting from the docs (https://dev.mysql.com/doc/refman/5.7/en/implicit-commit.html):
And the way MySQL does nested transactions in the InnoDB engine is to use Just to be sure, because I thought I might have misunderstood, I decided to just run a nested transaction manually. And they confirmed the behaviour observed. That being said, it is reasonable to seek a more confident answer. These are the statements I ran:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I found a much more concrete reference in the MySQL docs:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for doing this research! I'll spend some time with your references to make sure that we're on the same page. If it's true that there's no way to get |
||
''' | ||
|
||
db_testdir.makepyfile(""" | ||
def test_drop_table(person, db_engine): | ||
if db_engine.name == 'mysql': | ||
return | ||
|
||
# Drop the raw table | ||
db_engine.execute(''' | ||
DROP TABLE "person" | ||
DROP TABLE person | ||
''') | ||
|
||
# Check if the raw table exists | ||
existing_tables = db_engine.execute(''' | ||
SELECT relname | ||
FROM pg_catalog.pg_class | ||
WHERE relkind in ('r', 'm') | ||
AND relname = 'person' | ||
''').first() | ||
if db_engine.name == 'postgresql': | ||
existing_tables = db_engine.execute(''' | ||
SELECT relname | ||
FROM pg_catalog.pg_class | ||
WHERE relkind in ('r', 'm') | ||
AND relname = 'person' | ||
''').first() | ||
|
||
else: | ||
raise ValueError( | ||
'unsupported database engine type: ' + db_engine.name | ||
) | ||
|
||
assert not existing_tables | ||
|
||
def test_drop_table_changes_dont_persist(person, db_engine): | ||
|
||
existing_tables = db_engine.execute(''' | ||
SELECT relname | ||
FROM pg_catalog.pg_class | ||
WHERE relkind in ('r', 'm') | ||
AND relname = 'person' | ||
''').first() | ||
if db_engine.name == 'mysql': | ||
return | ||
|
||
if db_engine.name == 'postgresql': | ||
existing_tables = db_engine.execute(''' | ||
SELECT relname | ||
FROM pg_catalog.pg_class | ||
WHERE relkind in ('r', 'm') | ||
AND relname = 'person' | ||
''').first() | ||
|
||
else: | ||
raise ValueError( | ||
'unsupported database engine type: ' + db_engine.name | ||
) | ||
|
||
assert existing_tables | ||
""") | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are these lines necessary? I thought that the
database
fixture intests/_conftest.py
was setting up the database for us.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I was learning a bit about travis when I copied these statements from the tutorial docs. Indeed they are being created in
_conftest.py
. I will remove them.