Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logic to enable iceberg incremental tables. #1190

Merged
merged 18 commits into from
Sep 26, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20240923-203204.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Add Iceberg format Incremental Models
time: 2024-09-23T20:32:04.783741-07:00
custom:
Author: versusfacit
Issue: "321"
23 changes: 21 additions & 2 deletions dbt/include/snowflake/macros/materializations/incremental.sql
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,20 @@

{% materialization incremental, adapter='snowflake', supported_languages=['sql', 'python'] -%}

{% set original_query_tag = set_query_tag() %}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @VersusFacit, I noticed that the query tags for incremental models are not applied anymore.
Why did you remove this line?


{#-- Set vars --#}
{%- set full_refresh_mode = (should_full_refresh()) -%}
{%- set identifier = model['alias'] -%}
VersusFacit marked this conversation as resolved.
Show resolved Hide resolved
{%- set language = model['language'] -%}
{% set target_relation = this %}

{%- set target_relation = api.Relation.create(
VersusFacit marked this conversation as resolved.
Show resolved Hide resolved
identifier=identifier,
schema=schema,
database=database,
type='table',
table_format=config.get('table_format', 'default')
) -%}

{% set existing_relation = load_relation(this) %}

{#-- The temp relation will be a view (faster) or temp table, depending on upsert/merge strategy --#}
Expand All @@ -90,11 +98,22 @@
{%- call statement('main', language=language) -%}
{{ create_table_as(False, target_relation, compiled_code, language) }}
{%- endcall -%}

{% elif full_refresh_mode %}
{% if target_relation.needs_to_drop(existing_relation) %}
{{ drop_relation_if_exists(existing_relation) }}
{% endif %}
{%- call statement('main', language=language) -%}
{{ create_table_as(False, target_relation, compiled_code, language) }}
{%- endcall -%}

{% elif target_relation.table_format != existing_relation.table_format %}
VersusFacit marked this conversation as resolved.
Show resolved Hide resolved
{% do exceptions.raise_compiler_error(
"Unable to convert a " ~ existing_relation.table_format ~ " format table to " ~
target_relation.table_format ~ " without specifying --full-refresh."
)
%}

{% else %}
{#-- Create the temp relation, either as a view or as a temp table --#}
{% if tmp_relation_type == 'view' %}
Expand Down
Loading