feat: add dbt 1.9 support by gaurpulkit · Pull Request #46 · AltimateAI/datapilot-cli

gaurpulkit · 2024-12-27T07:22:27Z

Tested with tuva manifest and catalog files of dbt versions 1.6, 1.7, 1.8 and 1.9

Important

Add support for dbt 1.9 by updating dependencies, schema handling, and wrapper functions for compatibility.

Dependencies:
- Update dbt-artifacts-parser to version 0.8.1 in setup.py.
Schema Changes:
- Set default values for nodes and macros in AltimateDependsOn in manifest.py.
- Use ConfigDict for model_config in catalog.py and manifest.py.
Wrapper Updates:
- Modify _get_node() and _get_macro() in v10/wrapper.py, v11/wrapper.py, and v12/wrapper.py to handle resource_type directly and convert docs to dict if present.
- Adjust get_tests() and get_seeds() in v11/wrapper.py and v12/wrapper.py to handle resource_type without .value.
Schema Renames:
- Rename Node to Nodes, Node1 to Nodes1, etc., in v12/schemas.py.

^{This description was created by}^{for afbaafb. It will automatically update as commits are pushed.}

ellipsis-dev

👍 Looks good to me! Reviewed everything up to 8a0ece7 in 24 seconds

More details

Looked at 421 lines of code in 6 files
Skipped 0 files when reviewing.
Skipped posting 2 drafted comments based on config settings.

1. src/datapilot/core/platforms/dbt/wrappers/manifest/v10/wrapper.py:116

Draft comment:
Ensure consistency in accessing resource_type. The change from node.resource_type.value to node.resource_type is correct and should be applied consistently across all instances. This change is also applicable in other files like v11/wrapper.py and v12/wrapper.py.
Reason this comment was not posted:
Comment did not seem useful.

2. src/datapilot/core/platforms/dbt/wrappers/manifest/v10/wrapper.py:178

Draft comment:
Ensure docs is converted to a dictionary if it exists. The change from macro.docs to macro.docs.dict() is correct and should be applied consistently across all instances. This change is also applicable in other files like v11/wrapper.py and v12/wrapper.py.
Reason this comment was not posted:
Comment did not seem useful.

Workflow ID: wflow_uhO3IAMiq8YyCSIX

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

suryaiyer95 · 2024-12-27T07:31:06Z

src/datapilot/core/platforms/dbt/wrappers/manifest/v10/wrapper.py

            contract=contract,
            meta=node.meta,
            patch_path=node.patch_path,
+            access=node.access.value,


this is in v1.7? Is this because of artifact parser update?

Yes, it's because of that

suryaiyer95 · 2024-12-27T07:31:45Z

src/datapilot/core/platforms/dbt/wrappers/manifest/v11/wrapper.py

            schema_name=node.schema_,
            name=node.name,
-            resource_type=AltimateResourceType(node.resource_type.value),
+            resource_type=AltimateResourceType(node.resource_type),


is this right?

How is it mapping between the enums. Please check

node.resource_type is a string now. It is right. I had to do a similar change when we had upgraded to a new version of dbt-artifacts-parser in the past as well, but only for the newest manifest version. Seems like they introduced the same thing for the older manifest versions this time.

mdesmet · 2025-01-08T09:21:29Z

setup.py

    install_requires=[
        "click==8.1.7",
-        "dbt-artifacts-parser==0.6.0",
+        "dbt-artifacts-parser==0.8.1",


Can you make this

Suggested change

"dbt-artifacts-parser==0.8.1",

"dbt-artifacts-parser>=0.8.1",

ellipsis-dev

👍 Looks good to me! Incremental review on 861a571 in 43 seconds

More details

Looked at 23 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 1 drafted comments based on config settings.

1. setup.py:65

Draft comment:
Consider verifying if the change from == to ~= for version specifiers is compatible with the project's requirements. This change allows for more flexibility but may introduce compatibility issues if not tested thoroughly.
Reason this comment was not posted:
Confidence changes required: 50%
The change from '==' to '~=' in the version specifier allows for more flexibility in dependency versions, which is generally a good practice unless specific versions are required. However, this should be verified against the project's compatibility requirements.

Workflow ID: wflow_FPLgl597IcqoGUce

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

…bt-1.9-support

mdesmet

Please check why tests are falling

gaurpulkit · 2025-01-08T10:51:42Z

pydantic.warnings.PydanticDeprecatedSince20: pydantic.config.Extra is deprecated, use literal values instead (e.g. extra='allow'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at

Pydantic warnings are being treated as errors

ellipsis-dev

👍 Looks good to me! Incremental review on afbaafb in 35 seconds

More details

Looked at 764 lines of code in 11 files
Skipped 0 files when reviewing.
Skipped posting 5 drafted comments based on config settings.

1. src/datapilot/core/platforms/dbt/insights/checks/check_model_has_labels_keys.py:79

Draft comment:
The use of model_dump() is correct for Pydantic v1.9+ models. Ensure consistency across the codebase.
Reason this comment was not posted:
Confidence changes required: 20%
The use of model_dump() is consistent across the codebase for converting Pydantic models to dictionaries. This is a valid change from the previous dict() method, aligning with Pydantic v1.9+ changes.

2. src/datapilot/core/platforms/dbt/insights/checks/check_source_has_freshness.py:64

Draft comment:
The use of model_dump() is correct for Pydantic v1.9+ models. Ensure consistency across the codebase.
Reason this comment was not posted:
Confidence changes required: 20%
The use of model_dump() is consistent across the codebase for converting Pydantic models to dictionaries. This is a valid change from the previous dict() method, aligning with Pydantic v1.9+ changes.

3. src/datapilot/core/platforms/dbt/wrappers/manifest/v10/wrapper.py:137

Draft comment:
The use of model_dump() is correct for Pydantic v1.9+ models. Ensure consistency across the codebase.
Reason this comment was not posted:
Confidence changes required: 20%
The use of model_dump() is consistent across the codebase for converting Pydantic models to dictionaries. This is a valid change from the previous dict() method, aligning with Pydantic v1.9+ changes.

4. src/datapilot/core/platforms/dbt/wrappers/manifest/v11/wrapper.py:137

Draft comment:
The use of model_dump() is correct for Pydantic v1.9+ models. Ensure consistency across the codebase.
Reason this comment was not posted:
Confidence changes required: 20%
The use of model_dump() is consistent across the codebase for converting Pydantic models to dictionaries. This is a valid change from the previous dict() method, aligning with Pydantic v1.9+ changes.

5. src/datapilot/core/platforms/dbt/wrappers/manifest/v12/wrapper.py:137

Draft comment:
The use of model_dump() is correct for Pydantic v1.9+ models. Ensure consistency across the codebase.
Reason this comment was not posted:
Confidence changes required: 20%
The use of model_dump() is consistent across the codebase for converting Pydantic models to dictionaries. This is a valid change from the previous dict() method, aligning with Pydantic v1.9+ changes.

Workflow ID: wflow_vFd2QmfpGk5gwVUp

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

add dbt 1.9 support

8a0ece7

gaurpulkit requested a review from suryaiyer95 December 27, 2024 07:22

ellipsis-dev bot reviewed Dec 27, 2024

View reviewed changes

suryaiyer95 reviewed Dec 27, 2024

View reviewed changes

gaurpulkit requested a review from suryaiyer95 January 8, 2025 09:11

gaurpulkit changed the title ~~fear: add dbt 1.9 support~~ feat: add dbt 1.9 support Jan 8, 2025

mdesmet reviewed Jan 8, 2025

View reviewed changes

change requirements

861a571

ellipsis-dev bot reviewed Jan 8, 2025

View reviewed changes

Merge branch 'main' of https://github.com/AltimateAI/datapilot into d…

2664714

…bt-1.9-support

mdesmet approved these changes Jan 8, 2025

View reviewed changes

mdesmet self-requested a review January 8, 2025 10:17

mdesmet reviewed Jan 8, 2025

View reviewed changes

fix pydantic issues

afbaafb

ellipsis-dev bot reviewed Jan 8, 2025

View reviewed changes

mdesmet merged commit 492a2da into main Jan 9, 2025
90 of 102 checks passed

mdesmet deleted the dbt-1.9-support branch January 9, 2025 02:00

	"dbt-artifacts-parser==0.8.1",
	"dbt-artifacts-parser>=0.8.1",

Comments

Conversation

gaurpulkit commented Dec 27, 2024 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

suryaiyer95 Dec 27, 2024

Choose a reason for hiding this comment

Uh oh!

gaurpulkit Jan 8, 2025

Choose a reason for hiding this comment

Uh oh!

suryaiyer95 Dec 27, 2024

Choose a reason for hiding this comment

Uh oh!

suryaiyer95 Dec 27, 2024

Choose a reason for hiding this comment

Uh oh!

gaurpulkit Jan 8, 2025

Choose a reason for hiding this comment

Uh oh!

mdesmet Jan 8, 2025

Choose a reason for hiding this comment

Uh oh!

gaurpulkit Jan 8, 2025

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

mdesmet left a comment

Choose a reason for hiding this comment

Uh oh!

gaurpulkit commented Jan 8, 2025

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gaurpulkit commented Dec 27, 2024 •

edited by ellipsis-dev bot

Loading