Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: move tpc-h testing setup into ibis #7190

Merged
merged 3 commits into from
Sep 26, 2023

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Sep 21, 2023

This PR moves our TPC-H testing into ibis.

Up to now, we've been doing tests with https://github.com/ibis-project/tpc-queries, which was usually fine.

However, during some development of #6996 the tpc-queries job started failing
and I moved TPC-H into ibis for faster debugging cycles.

This PR is the result of that work.

It turns out that for the other backend I added here (Snowflake) we're failing
three tpc-h tests that involve correlated subqueries (surprise!).

We can incrementally add support for more backends by:

  1. Setting the appropriate supports_tpch flag in the backend's TestConf class.

  2. Defining a few methods on the backend's TestConf class:

    class TestConf:
        ...
        def load_tpch(self) -> None:
            """Load TPC-H data into the current database
    
            Used in the DuckDB backend to run CALL dbgen(...)
            """
    
        def _tpch_table(self, name: str) -> ir.Table:
             """Return a table expression for the given TPC-H table
    
             Used in the Snowflake backend to access TPC-H tables, which
             reside in a different database than the one we're testing in
             """
    
        def _transform_tpch_sql(self, parsed: sg.exp.Expression):
             """Apply any semantics-preserving transformations to the parsed SQL.
    
             Used in the Snowflake backend to attach the database and schema to
             every leaf table
             """

@cpcloud cpcloud added this to the 7.0 milestone Sep 21, 2023
@cpcloud cpcloud added bug Incorrect behavior inside of ibis tests Issues or PRs related to tests duckdb The DuckDB backend snowflake The Snowflake backend labels Sep 21, 2023
@cpcloud cpcloud force-pushed the tpch-tests branch 7 times, most recently from c324e8d to 710a8a8 Compare September 26, 2023 09:21
@cpcloud cpcloud added the trino The Trino backend label Sep 26, 2023
@cpcloud cpcloud force-pushed the tpch-tests branch 3 times, most recently from d5a29be to 7e0e43f Compare September 26, 2023 12:06
@cpcloud
Copy link
Member Author

cpcloud commented Sep 26, 2023

Backend tests are passing:

…/ibis on  tpch-tests is 📦 v6.1.0 via 🐍 v3.10.12 via ❄️  impure (ibis-3.10.12-env)
❯ pytest -m 'bigquery or snowflake' --snapshot-update --allow-snapshot-deletion -n auto -q
bringing up nodes...
....xxxx...x.......xx.....x...x.....sxxxx.x..xxxxx......x...xx.xx.xssssx.x.x.xx.xx.xxsss.......x.x...xxx.sssss.x.x..x...x...x....ss.x..xx...x.....x....x.x....x...x...................x...x.x. [  6%]
..x.x......x......xxx......x....x.........xxxxxxxxxxxx.....xxxxx.x.xxxxx.xxxxxxx.xx..xx.xx.xs.sssx....xx.x..x.x.x.......x....x.xx.x.x.x.ss........x.x.......x.x.....x.x.........ss............ [ 13%]
.x.......x....sxxxxxxxx.xxxx..xxxxx.xx.xx...xxx....x...........x..x.........ss......x.........x.........x...x...x.....x..x......................s..x.x..............x..............x.......... [ 20%]
..x............x.....xx.......x...x................x.x...........x.............x.x..x..........xxx..........x..................x....x....x...........x......x...............x........x.....x.. [ 27%]
......................xx.....................................x........x...x.........xxx.....x....x.s.x......................x.............xx..xx.xxxxs.........x.x......x...x.x.............x. [ 33%]
.....x.........x...........................................x.....x........x..........x......x..x...x....x....xx......x...x........x.xx.........x...................x...x...xx......x.x..xx.... [ 40%]
..........................x..................................x......x.........xx........xx.....x....x.........x.........x...............x.x......x.x..........x....................x.......... [ 47%]
x..xx...x..............x...s......x.xx.....xx.....x................x...x....x.......................x.x.x.x..x..x....xxx...x.x......x...........x...x............x...x.....x....x..x.....x..x. [ 54%]
x....x.x...x...x............x.xxx..x......x...xx...x......x......x......x.x...x....x........xx.xxx............x.x...x.....x..x.......x........x..x...x.....xxx.x.....xx...x.....x..xx......... [ 60%]
.....x....x...x.......xx......x..x......x..............x.x.x...xx.x.x.....x.........xx........xxx.......x.x...........ssss.....s.x............x.......x...x......x........xx.....x......xx..x. [ 67%]
.xx........x...xx...x..x..........s............x......xx.......x.......x.............................x.x.....................x.............x....xx.x.xx..x.x.x.x.............................. [ 74%]
................................................x........................................................x..........xx............................................................s.........x. [ 81%]
.........x....x....x.........x.............x.x.......x......x...x...........................x..........x.........................x.x..........................xx.x.........x.....x............ [ 88%]
..x........................x..x.x...x..x..x....x...x.x........................................................................................................................................ [ 94%]
...................................................x..x..............................................................xxx.x.....s...........x.x....                                             [100%]
2335 passed, 38 skipped, 433 xfailed in 471.84s (0:07:51)

Copy link
Member

@gforsyth gforsyth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gforsyth gforsyth merged commit 7648cfc into ibis-project:master Sep 26, 2023
85 checks passed
@cpcloud cpcloud deleted the tpch-tests branch September 26, 2023 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis duckdb The DuckDB backend snowflake The Snowflake backend tests Issues or PRs related to tests trino The Trino backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants