Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Source metadata freshness is using the wrong field to calculate last updated #899

Closed
2 tasks done
mikealfare opened this issue Feb 6, 2024 · 2 comments
Closed
2 tasks done
Assignees
Labels
bug Something isn't working dynamic_tables user docs [docs.getdbt.com] Needs better documentation

Comments

@mikealfare
Copy link
Contributor

Is this a new bug in dbt-snowflake?

  • I believe this is a new bug in dbt-snowflake
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Per @tjirab's #785 (comment):

Minor concern: LAST_ALTERED gets updated upon DML & DDL changes, and maintenance ops. It does not necessarily mean fresh data has become available.

Expected Behavior

We would expect to only reflect data updates, and not all object updates, when reporting data freshness.

Steps To Reproduce

  1. Make a change to the dynamic table config (refresh frequency, warehouse, etc.)
  2. Ensure the dependent data sources have not seen new data
  3. Observe that the data freshness is reporting newer data because it's picking up the update from 1.

Relevant log output

No response

Environment

- OS: any
- Python: any
- dbt-core: any
- dbt-snowflake: 1.6+ (dynamic table support)

Additional Context

No response

@mikealfare
Copy link
Contributor Author

After discussing internally, we determined that this was a known and acceptable risk. However, we also determined that the documentation should have been updated to reflect this scenario, but was not. We will resolve this by noting the scenario in our docs.

The docs issue will be attached here once I created it. In short, the assumption is that there should be minimal occasions where LAST_ALTERED reflects something other than a data update. And when it does reflect something other than a data update, that situation would resolve itself with the next data update. The impact is only material if the underlying data is rarely updated. It would seem in those scenarios that the data refresh timestamp is less useful anyway.

@mikealfare mikealfare added support user docs [docs.getdbt.com] Needs better documentation and removed triage bug Something isn't working labels Feb 8, 2024
@mikealfare mikealfare assigned mikealfare and unassigned Fleid Feb 8, 2024
@mikealfare mikealfare added bug Something isn't working and removed support labels Jul 17, 2024
@amychen1776
Copy link

I'm going to go ahead and close this issue out due to the docs being updated and this being an acceptable risk at this time. If anyone is interested in us addressing this, please open the issue again.

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dynamic_tables user docs [docs.getdbt.com] Needs better documentation
Projects
None yet
Development

No branches or pull requests

4 participants