implication of changes to env_vars and vars for state:modified

[graciegoheen]

We currently call out a lot of caveats to state comparison in our docs.

There are many scenarios where executing dbt list --select state:modified over or under selects the appropriate resources. This leads to:

• confusion "why are these models selected as state:modified, I haven't changed them"
• performance concerns "why am I running 60 models when I only changed 2"
• potentially bad code sneaking it's way into production "i checked my changes using a Slim CI job, but these models weren't run, now production is broken"

While it's unlikely we can cover ever single edge case here, we want to improve state:modified so that it ideally actually only selects the modified resources - we started an EPIC to track that work.

Investigating a few of the top issues has lead us to some specific opinions on how we think state:modified should work for vars and env_vars. Since these changes fall into the category of “not quite bug, but not quite a feature”, we want to run them by y’all first and hear if you have any reactions / concerns / comments before we implement.

Environment Variables

Current Behavior

Currently, state:modified includes all changes to the value of an environment variable.

Let’s say you want to point to a different location for your sources depending on the environment you are in (e.g. using a different database for prod vs. dev). To do so, you configure your source to set database to an environment variable like so:

# models/staging/jaffle_shop/sources.yml

sources:
  - name: jaffle_shop
    database: "{{ env_var('DBT_DATABASE'}}"
    tables:
      - name: customers

If you then run dbt in dev deferring to prod, this source will be marked as modified every time you use --select state:modified because the value of this environment variable (DBT_DATABASE) differs by environment.

More specifically, this environment-sensitivity leading to false positives only occurs when the env_var configuration is coming from a schema yml or definition (.sql, .py) file. Under the hood, this is because we are comparing the database values after the jinja in schema and definition files has been rendered, so dbt cannot tell whether it’s the unrendered or rendered configurations have changed.

Future State

We think this is wrong. Instead, we believe that changes to the value of an environment variable should be ignored when selecting state:modified:

• code changes to environment variables should mark a resource as modified (e.g. changing database from {{ env_var('DBT_DATABASE'}} to {{ env_var('DBT_SOURCE_DATABASE'}})
• value changes of environment variables should be ignored when evaluating if a resource is modified (e.g. the environment variable DBT_DATABASE compiling to raw_dev vs. raw_prod)

⚠️ This would mean that even if you were self-deferring (deferring to the environment that you’re running in) and updated the value of your environment variable, dbt would not detect the associated resources as modified.

Project Variables

Current Behavior

Currently, state:modified excludes all changes to the value of a project variable.

Let’s say you’re using a project variable to set a project-default “future date” to coalesce your NULL dates to:

# dbt_project.yml

...

vars:
  future_date: 'to_date(2999-01-01)'

And then you’re using that project variable in some of your models:

# models/staging/jaffle_shop/stg_orders.sql

select
	...
	coalesce(order_date, {{ var('future_date') }}) as order_date 
from {{ source('jaffle_shop', 'orders') }}

If you then update the value of this project variable (either in your dbt_project.yml or via a CLI override using —-vars) and run dbt using --select state:modified, this model will not be marked as modified. This is because we are not tracking which var values a given model relies on explicitly in dbt’s internal manifest.

Future State

We think this is wrong. Instead, we believe that changes to the value of a project variable should be included when selecting state:modified:

• code changes to project variables should mark a resource as modified (e.g. changing your logic from coalesce(order_date, {{ var('future_date') }}) to coalesce(order_date, {{ var('future_timestamp') }}))
• value changes of project variables should should mark a resource as modified (e.g. the project variable future_date compiling to to_date(2999-01-01) vs. to_date(3999-01-01))

Note: Changes to var values, when defined in a schema yml file in a config block, (e.g. database: "{{ var('dbt_database'}}") are already marked as modified today for the exact reason that dbt is overly-sensitive to environment variable value changes above — it is comparing rendered configurations to one another. We think this is correct for vars, and should continue to work this way.

Why would project variables and environment variables be treated differently?

• project variable default values are set as part of the code
• environment variables are intentionally designed to allow you to configure different behavior of you dbt project depending on where the project is running

Thoughts?

We’d love to hear your thoughts, use cases, etc. to help inform our decision here!

[joellabes]

Just checking that this summary table is correct, and we're proposing to flip which variables do/don't get picked up by state:modified?

Variable Type	Current state:modified Behaviour	Future state:modified Behaviour
Environment Variables	Changes are picked up	Changes are ignored
Project Variables	Changes are ignored	Changes are picked up

What happens if I do this?

export FUTURE_DATE_ENV_VAR='2999-01-01'

# dbt_project.yml
...
vars:
  future_date_var: {{ env_var('FUTURE_DATE_ENV_VAR') }}

--models/some_model.sql
select id, coalesce(order_date, {{ var('future_date_var') }})
...

I'm guessing that the env var will still have been passed into future_date_var during compilation/parsing so it will be picked up as a modification?

If so then that might also be a helpful escape hatch for anyone who is dependent on the current behaviour where env var changes are picked up as modifications.

[jaypeedevlin]

Would love when a macro is modified that its usages get marked as downstream modified, is this in the horizon?

[MichelleArk]

That is very much the intention! Here's a sneak peek of some initial testing a macros modification from a change to an underlying var value propagating to downstream usages: https://github.com/dbt-labs/dbt-core/pull/10502/files#diff-c6b4cdfaa959b364be404f07c71079147f1ac1b920d86f5f54dcdb5f94d43ed6R1197-R1233

[sp-tkerlavage]

Just commenting here, maybe this has already been considered in some other discussion elsewhere, if so I didn't see it.

Lets say we have a project variable that indicates the start of an incremental window. This is accomplished by calling a macro that returns todays date plus some (negative) offset.

  date_start_offset_days: -2
  date_start: '{{ today(offset_days = var("date_start_offset_days")) }}' 

When you determine what has been "modified" under the new logic will you be considering the rendered value or unrendered value? Unrendered meaning the macro call itself, rendered meaning the actual result of the macro call.

Meaning, if the macro calls themselves don't change change, the models that depend on the date_start will not be considered modified?

Or will you considered the value that the macro actually produces? If you consider this rendered value output by the macro, and defer to a job that ran yesterday, any job runs today that uses the date_start variable would be considered modified. I don't think that would be the intended behavior imo.

[noel]

Are there other features needed to address:

• confusion "why are these models selected as state:modified, I haven't changed them"
• performance concerns "why am I running 60 models when I only changed 2"

I know @joellabes created a tool that compares manifests, but I am wondering if there can be something on core that could summarize what was detected at a change.

This case is indeed very common and users don't know the implication of doing what to them seems "reasonable"

sources:
  - name: jaffle_shop
    database: "{{ env_var('DBT_DATABASE'}}"
    tables:
      - name: customers

When I debug these issues I use the state selectors to at least get me closer to the cause, maybe there can be a flag we can pass to dbt ls that would indicate what was different e.g. database in the case above.

[joellabes]

@noel the case above is no longer considered "modified" if you opt into the new behaviour change flag:

# dbt_project.yml
... 

flags: 
  state_modified_compare_more_unrendered_values: True

We're now persisting unrendered_config for models and unrendered_database/unrendered_schema for sources

[noel]

oh great. what version of dbt has that, 1.9?

[joellabes]

Yep! 1.9 or Versionless in Cloud