Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tags to SavedQuery #366

Merged
merged 2 commits into from
Nov 14, 2024
Merged

Add tags to SavedQuery #366

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
10794b9
InProgress: Add tags to SavedQuery
theyostalservice Nov 8, 2024
019d648
Change parse_obj to use deepcopy rather than editing in place
theyostalservice Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20241113-110648.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Add "tags" to SavedQuery nodes, similar to existing nodes' tags.
time: 2024-11-13T11:06:48.562566-08:00
custom:
Author: theyostalservice
Issue: "369"
22 changes: 19 additions & 3 deletions dbt_semantic_interfaces/implementations/saved_query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
from __future__ import annotations

from typing import List, Optional
from typing import Any, List, Optional, Union

from typing_extensions import override
from typing_extensions import Self, override

from dbt_semantic_interfaces.implementations.base import (
HashableBaseModel,
Expand Down Expand Up @@ -35,7 +35,11 @@ def _implements_protocol(self) -> SavedQueryQueryParams:
where: Optional[PydanticWhereFilterIntersection] = None


class PydanticSavedQuery(HashableBaseModel, ModelWithMetadataParsing, ProtocolHint[SavedQuery]):
class PydanticSavedQuery(
HashableBaseModel,
ModelWithMetadataParsing,
ProtocolHint[SavedQuery],
):
"""Pydantic implementation of SavedQuery."""

@override
Expand All @@ -48,3 +52,15 @@ def _implements_protocol(self) -> SavedQuery:
metadata: Optional[PydanticMetadata] = None
label: Optional[str] = None
exports: List[PydanticExport] = Field(default_factory=list)
tags: Union[str, List[str]] = Field(
default_factory=list,
)

@classmethod
def parse_obj(cls, input: Any) -> Self: # noqa
if isinstance(input, dict):
if isinstance(input.get("tags"), str):
input["tags"] = [input["tags"]]
if isinstance(input.get("tags"), list):
input["tags"].sort()
return super(HashableBaseModel, cls).parse_obj(input)
13 changes: 13 additions & 0 deletions dbt_semantic_interfaces/parsing/generated_json_schemas/default_explicit_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,19 @@
},
"query_params": {
"$ref": "#/definitions/saved_query_query_params_schema"
},
"tags": {
"oneOf": [
{
"type": "string"
},
{
"items": {
"type": "string"
},
"type": "array"
}
]
}
},
"required": [
Expand Down
9 changes: 9 additions & 0 deletions dbt_semantic_interfaces/parsing/schemas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,15 @@
"query_params": {"$ref": "saved_query_query_params_schema"},
"label": {"type": "string"},
"exports": {"type": "array", "items": {"$ref": "export_schema"}},
"tags": {
"oneOf": [
{"type": "string"},
{
"type": "array",
"items": {"type": "string"},
},
],
},
},
"required": ["name", "query_params"],
"additionalProperties": False,
Expand Down
6 changes: 6 additions & 0 deletions dbt_semantic_interfaces/protocols/saved_query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,9 @@ def label(self) -> Optional[str]:
def exports(self) -> Sequence[Export]:
"""Exports that can run using this saved query."""
pass

@property
@abstractmethod
def tags(self) -> Sequence[str]:
"""List of tags to be used as part of resource selection in dbt."""
pass
112 changes: 112 additions & 0 deletions tests/parsing/test_saved_query_parsing.py
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theyostalservice Could you add some test cases that verify that tags are "additive" for semantic models when relevant tag configs appear both within dbt_project.yml and stuff.yml files?

I'll explain more below 😎

"Clobbering" for enabled

Saved queries can have configuration added in dbt_project.yml as well as properties / "schema" YAML files.

Using the enabled config as an example:

dbt_project.yml

name: dbt_project_name_here

saved-queries:
  dbt_project_name_here:
    +enabled: false

models/_saved_queries.yml

saved_queries:
  - name: saved_query_name
    config:
      enabled: true

The above would make all saved queries not enabled by default ("disabled"). But then it would enable saved_query_name. This is the "clobbering" behavior that most dbt configs have.

"Additive" for tags

But for tags in semantic models, we might have config like this:

dbt_project.yml

saved-queries:
  dbt_project_name_here:
    +tags: ["tag_1", "tag_2"]

models/_saved_queries.yml

saved_queries:
  - name: saved_query_name
    config:
      tags: ["tag_3"]

The above would make the default tags for saved queries be the following by default:
["tag_1", "tag_2"]

But then saved_query_name would have:
["tag_1", "tag_2", "tag_3"]

This is the "additive" behavior of tags.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that dbt_project.yml behavior is driven here in dsi - i think it's handled in dbt-core. I can't even find "enabled" in this repo, tbh.

@courtneyholcomb - do you know if this is correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I can answer this now.

@dbeatty10 - that sort of behavior is not managed in the dsi repo; the extra project-level config stuff is all handled in dbt-core. dbt-labs/dbt-core#10987 isn't ready for review, but in some manually run tests, the updated code there works as you described. :-)

(Specifically, the changes in core/dbt/parser/schema_yaml_readers.py produce the change you're looking for. :-) )

Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,118 @@ def test_saved_query_where() -> None:
assert where == saved_query.query_params.where.where_filters[0].where_sql_template


def test_saved_query_with_single_tag_string() -> None:
"""Test for parsing a single string (not a list) tag in a saved query."""
yaml_contents = textwrap.dedent(
"""\
saved_query:
name: test_saved_query_group_bys
tags: "tag_1"
query_params:
metrics:
- test_metric_a
"""
)
file = YamlConfigFile(filepath="test_dir/inline_for_test", contents=yaml_contents)

build_result = parse_yaml_files_to_semantic_manifest(files=[file, EXAMPLE_PROJECT_CONFIGURATION_YAML_CONFIG_FILE])
assert len(build_result.semantic_manifest.saved_queries) == 1
saved_query = build_result.semantic_manifest.saved_queries[0]
assert saved_query.tags is not None
assert len(saved_query.tags) == 1
assert saved_query.tags == ["tag_1"]


def test_saved_query_with_multiline_list_of_tags() -> None:
"""Test for parsing a multiline list of tags in a saved query."""
yaml_contents = textwrap.dedent(
"""\
saved_query:
name: test_saved_query_group_bys
tags: ["tag_1", "tag_2"]
query_params:
metrics:
- test_metric_a
"""
)
file = YamlConfigFile(filepath="test_dir/inline_for_test", contents=yaml_contents)

build_result = parse_yaml_files_to_semantic_manifest(files=[file, EXAMPLE_PROJECT_CONFIGURATION_YAML_CONFIG_FILE])
assert len(build_result.semantic_manifest.saved_queries) == 1
saved_query = build_result.semantic_manifest.saved_queries[0]
assert saved_query.tags is not None
assert len(saved_query.tags) == 2
assert saved_query.tags == ["tag_1", "tag_2"]


def test_saved_query_with_single_line_list_of_tags() -> None:
"""Test for parsing a single-line list of tags in a saved query."""
yaml_contents = textwrap.dedent(
"""\
saved_query:
name: test_saved_query_group_bys
tags:
- "tag_1"
- "tag_2"
query_params:
metrics:
- test_metric_a
"""
)
file = YamlConfigFile(filepath="test_dir/inline_for_test", contents=yaml_contents)

build_result = parse_yaml_files_to_semantic_manifest(files=[file, EXAMPLE_PROJECT_CONFIGURATION_YAML_CONFIG_FILE])
assert len(build_result.semantic_manifest.saved_queries) == 1
saved_query = build_result.semantic_manifest.saved_queries[0]
assert saved_query.tags is not None
assert len(saved_query.tags) == 2
assert saved_query.tags == ["tag_1", "tag_2"]


def test_saved_query_tags_are_sorted() -> None:
"""Test tags in a saved query are SORTED after parsing."""
yaml_contents = textwrap.dedent(
"""\
saved_query:
name: test_saved_query_group_bys
tags:
- "tag_2"
- "tag_1"
query_params:
metrics:
- test_metric_a
"""
)
file = YamlConfigFile(filepath="test_dir/inline_for_test", contents=yaml_contents)

build_result = parse_yaml_files_to_semantic_manifest(files=[file, EXAMPLE_PROJECT_CONFIGURATION_YAML_CONFIG_FILE])
assert len(build_result.semantic_manifest.saved_queries) == 1
saved_query = build_result.semantic_manifest.saved_queries[0]
assert saved_query.tags is not None
assert len(saved_query.tags) == 2
assert saved_query.tags == ["tag_1", "tag_2"]


def test_saved_query_with_no_tags_defaults_to_empty_list() -> None:
"""Test tags in a saved query will default to empty list if missing."""
yaml_contents = textwrap.dedent(
"""\
saved_query:
name: test_saved_query_group_bys
query_params:
metrics:
- test_metric_a
"""
)
file = YamlConfigFile(filepath="test_dir/inline_for_test", contents=yaml_contents)

build_result = parse_yaml_files_to_semantic_manifest(files=[file, EXAMPLE_PROJECT_CONFIGURATION_YAML_CONFIG_FILE])
assert len(build_result.semantic_manifest.saved_queries) == 1
saved_query = build_result.semantic_manifest.saved_queries[0]
assert saved_query.tags is not None
assert saved_query.tags == []


def test_saved_query_exports() -> None:
"""Test for parsing exports referenced in a saved query."""
yaml_contents = textwrap.dedent(
Expand Down
Loading