Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation tests are super slow #146

Open
tlento opened this issue Dec 1, 2022 · 1 comment
Open

Validation tests are super slow #146

tlento opened this issue Dec 1, 2022 · 1 comment
Assignees
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@tlento
Copy link
Collaborator

tlento commented Dec 1, 2022

Describe the bug
When we run our unit tests everything grinds to a halt on the semantic validator tests

This is potentially caused by our test construction - most of these tests do something like:

with pytest.raises(...):
    # run all validations

That means we run every validation on every test input model on every test case, even though most, if not all, of these tests cases are targeted at highly specific validation rules.

A good start here would be to migrate these test cases to only run the specific rule we care to test.

Another possible optimization - both for runtime and readability - is to use smaller, more targeted models defined local to the test case, so converting these cases away from model fixtures onto the local model shims would be great as well. See https://github.com/transform-data/metricflow/blob/main/metricflow/test/model/validations/test_validity_param_definitions.py#L51-L71 for an example of a locally defined model with a specific failure state written into it.

Steps To Reproduce
Steps to reproduce the behavior:

  1. Run make test
  2. Observe slowness for tests in the validations path

Expected behavior
These should be much faster

@tlento tlento added the bug Something isn't working label Dec 1, 2022
@tlento tlento self-assigned this Dec 1, 2022
@tlento tlento assigned jzhu13 and unassigned tlento Dec 15, 2022
@tlento tlento added the good first issue Good for newcomers label Dec 15, 2022
@QMalcolm
Copy link
Collaborator

We should probably move this issue to dbt-semantic-interfaces

@tlento tlento transferred this issue from dbt-labs/metricflow Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants