Feat/1492 extend timestamp config #1669

donotpush · 2024-08-07T10:32:25Z

Description

dlt does not support timestamps without timezones and to be able to pass such timestamps form source to destination we need to extend the core library.

The scope of this PR is to implement a proof of concept (POC) for DuckDB and parquet. This includes updating the type mapper and adding tests where the timezone is passed via columns.

Additionally, a complementary notebook with various experiments is included in this PR.

https://github.com/donotpush/dlt-notebooks/blob/main/notebooks/duckdb-exploration.ipynb

@rudolfix requirements to solve #1492 :

We'd need a new hint in TColumnType that will say if data_type contains timezone or not (or define time zone as string, with default to "UTC"). See precision and scale hints already there. The idea is the same.
Each of our destinations has a type mapper which needs to be be able to interpret this new thing.
pyarrow.py contains functions that convert dlt schema into arrow schema and vice versa. those functions needs to be upgraded
we'll need plenty of tests: loading non-tz aware datetimes via json and parquet into all possible destinations
let's make it work for duckdb+postgres+snowflake first

Related Issues

Resolves Option to keep default date columns as TIMESTAMP_NTZ in oracle to snowflake migration #1492

netlify · 2024-08-07T10:32:39Z

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	`6ef862a`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/66cf3ab0fab24a00086b8862

rudolfix

LGTM! implementation idea and code structure is good. duckdb was a hard case because we had technical debt where we silently created columns without timezones if precision required it. now we can do it right

my take: try to implement postgres and snowflake next. if you need any credentials ping us but I think you have access to CI vault

thx for the tests

dlt/common/libs/pyarrow.py

dlt/destinations/impl/duckdb/duck.py

donotpush · 2024-08-13T10:54:29Z

LGTM! implementation idea and code structure is good. duckdb was a hard case because we had technical debt where we silently created columns without timezones if precision required it. now we can do it right

my take: try to implement postgres and snowflake next. if you need any credentials ping us but I think you have access to CI vault

thx for the tests

@rudolfix I've implemented your review comments, and added more tests.

I'll review & fix the CICD failing tests.

donotpush · 2024-08-20T13:13:36Z

Added postgresql dlt's implementation and tests.

Links to my notebooks:

Notebook	Link
DuckDB Timestamps Timezone	1492-duckdb-timestamps-timezone.ipynb
PostgreSQL Timestamps Timezone	1492-postgres-timestamps-timezone.ipynb

Working on Snowflake.

rudolfix

this is really good! I think we are really close to close it.

could we support BigQuery and Athena? those are popular destinations
for other destinations log a warning that we do not support ts without timezones

then we are good to go. I like your tests

donotpush · 2024-08-23T16:37:52Z

@rudolfix - Comment updated.

warning log added.

BigQuery - I attempted to implement the timezone flag for BigQuery using both TIMESTAMP and DATETIME data types, but I encountered issues. BigQuery does not automatically handle timezones in incoming data for the DATETIME type, leading to errors. Tansforming (or stripping) JSON or other data types before insertion doesn't seem to be a viable solution.

I have documented my findings here:
BigQuery Exploration Notebook

Athena - Athena only supports a single TIMESTAMP type, which does not allow for timezone or precision configuration. AWS documentation can be confusing; they list DDL and DML columns, with "Not Available" indicating a lack of support for other TIMESTAMP types. I ran a few tests on my AWS account with the following data:

File uploaded to s3://test-athena-julian/test/1.txt:

1 2024-07-30T10:00:00.12
2 2024-07-30T11:00:00.123456
3 2024-07-30T12:00:00.1234
4 2024-07-30T10:00:00.1234567
5 2024-07-30T11:00:00.123456789

I used the following DDL statement to create the table:

CREATE EXTERNAL TABLE IF NOT EXISTS EVENTS(
  id INT,
  t TIMESTAMP
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
STORED AS TEXTFILE
LOCATION 's3://test-athena-julian/test';

Amazon Athena Data Types Documentation

rudolfix

LGTM! I have suggestion to move docs to different sections. the code is good. thanks for investigating and cleaning up timestamp mess :)

rudolfix · 2024-08-28T10:13:36Z

docs/website/docs/dlt-ecosystem/destinations/duckdb.md

+
+`duckdb` can create unique indexes for columns with `unique` hints. However, **this feature is disabled by default** as it can significantly slow down data loading.
+
+## Supported column flags


those are still called hints. I'd move this section under

Data loading

and replace this header with

Data types

(it is used in several other destinations)

please apply this to other docs

Done!

Question: What about this section?

## Supported column hints `duckdb` can create unique indexes for columns with `unique` hints. However, **this feature is disabled by default** as it can significantly slow down data loading.

It was there before this PR. Should we keep it as it is, or move it under Data Loading as well?

rudolfix

LGTM!

* feat: add timezone flag to configure timestamp data * fix: delete timezone init * test: add duckdb timestamps with timezone * test: fix resource hints for timestamp * test: correct duckdb timestamps * test: timezone tests for parquet files * exp: add notebook with timestamp exploration * test: refactor timestamp tests * test: simplified tests and extended experiments * exp: timestamp exp for duckdb and parquet * fix: add pyarrow reflection for timezone flag * fix lint errors * fix: CI/CD move tests pyarrow module * fix: pyarrow timezone defaults true * refactor: typemapper signatures * fix: duckdb timestamp config * docs: updated duckdb.md timestamps * fix: revert duckdb timestamp defaults * fix: restore duckdb timestamp default * fix: duckdb timestamp mapper * fix: delete notebook * docs: added timestamp and timezone section * refactor: duckdb precision exception message * feat: postgres timestamp timezone config * fix: postgres timestamp precision * fix: postgres timezone false case * feat: add snowflake timezone and precision flag * test: postgres invalid timestamp precision * test: unified timestamp invalid precision * test: unified column flag timezone * chore: add warn log for unsupported timezone or precision flag * docs: timezone and precision flags for timestamps * fix: none case error * docs: add duckdb default precision * fix: typing errors * rebase: formatted files from upstream devel * fix: warning message and reference TODO * test: delete duplicated input_data array * docs: moved timestamp config to data types section * fix: lint and format * fix: lint local errors

rudolfix reviewed Aug 8, 2024

View reviewed changes

dlt/common/libs/pyarrow.py Outdated Show resolved Hide resolved

dlt/destinations/impl/duckdb/duck.py Show resolved Hide resolved

rudolfix requested changes Aug 22, 2024

View reviewed changes

donotpush added 24 commits August 23, 2024 23:32

feat: add timezone flag to configure timestamp data

f35d717

fix: delete timezone init

e030594

test: add duckdb timestamps with timezone

79cdbef

test: fix resource hints for timestamp

4aed5ef

test: correct duckdb timestamps

e2ee13b

test: timezone tests for parquet files

874f1bf

exp: add notebook with timestamp exploration

9ca83dc

test: refactor timestamp tests

f4ac0dc

test: simplified tests and extended experiments

3cd9694

exp: timestamp exp for duckdb and parquet

2723777

fix: add pyarrow reflection for timezone flag

c7c9f9b

fix lint errors

a2e40eb

fix: CI/CD move tests pyarrow module

6bc03d8

fix: pyarrow timezone defaults true

a19c01f

refactor: typemapper signatures

5e755c8

fix: duckdb timestamp config

0b32cb7

docs: updated duckdb.md timestamps

e0ff92a

fix: revert duckdb timestamp defaults

c4b6010

fix: restore duckdb timestamp default

c6d59b7

fix: duckdb timestamp mapper

43a17c8

fix: delete notebook

f3242be

docs: added timestamp and timezone section

380123c

refactor: duckdb precision exception message

91b55dd

feat: postgres timestamp timezone config

6fc4a0a

donotpush added 11 commits August 23, 2024 23:32

fix: postgres timestamp precision

7b0b14c

fix: postgres timezone false case

e537758

feat: add snowflake timezone and precision flag

386b74b

test: postgres invalid timestamp precision

7558f48

test: unified timestamp invalid precision

2256133

test: unified column flag timezone

628742b

chore: add warn log for unsupported timezone or precision flag

b0894c3

docs: timezone and precision flags for timestamps

a09f0da

fix: none case error

67b3340

docs: add duckdb default precision

464ba98

fix: typing errors

3a4613c

donotpush force-pushed the feat/1492-extend-timestamp-config branch from 58a2150 to 3a4613c Compare August 23, 2024 21:32

donotpush added 2 commits August 23, 2024 23:36

rebase: formatted files from upstream devel

2e856dc

fix: warning message and reference TODO

0a8e35e

donotpush requested a review from rudolfix August 26, 2024 14:41

rudolfix requested changes Aug 28, 2024

View reviewed changes

donotpush added 3 commits August 28, 2024 14:53

test: delete duplicated input_data array

35d4628

docs: moved timestamp config to data types section

3208c58

merge: resolve conflicts origin/devel

ed965c3

donotpush requested review from rudolfix August 28, 2024 13:41

donotpush added 2 commits August 28, 2024 15:46

fix: lint and format

7004a35

fix: lint local errors

6ef862a

rudolfix approved these changes Aug 30, 2024

View reviewed changes

rudolfix merged commit fbf0ef4 into devel Aug 30, 2024
56 checks passed

rudolfix deleted the feat/1492-extend-timestamp-config branch August 30, 2024 09:41

rudolfix mentioned this pull request Sep 3, 2024

Sqlalchemy destination #1734

Merged

7 tasks

rudolfix mentioned this pull request Sep 13, 2024

1.0.0 announcement and release notes #1778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/1492 extend timestamp config #1669

Feat/1492 extend timestamp config #1669

donotpush commented Aug 7, 2024 •

edited

Loading

netlify bot commented Aug 7, 2024 •

edited

Loading

rudolfix left a comment

donotpush commented Aug 13, 2024

donotpush commented Aug 20, 2024 •

edited

Loading

rudolfix left a comment

donotpush commented Aug 23, 2024 •

edited

Loading

rudolfix left a comment

rudolfix Aug 28, 2024

donotpush Aug 28, 2024

rudolfix left a comment


		`duckdb` can create unique indexes for columns with `unique` hints. However, this feature is disabled by default as it can significantly slow down data loading.

		## Supported column flags

Feat/1492 extend timestamp config #1669

Feat/1492 extend timestamp config #1669

Conversation

donotpush commented Aug 7, 2024 • edited Loading

Description

Related Issues

netlify bot commented Aug 7, 2024 • edited Loading

✅ Deploy Preview for dlt-hub-docs canceled.

rudolfix left a comment

Choose a reason for hiding this comment

donotpush commented Aug 13, 2024

donotpush commented Aug 20, 2024 • edited Loading

rudolfix left a comment

Choose a reason for hiding this comment

donotpush commented Aug 23, 2024 • edited Loading

rudolfix left a comment

Choose a reason for hiding this comment

rudolfix Aug 28, 2024

Choose a reason for hiding this comment

Data loading

Data types

donotpush Aug 28, 2024

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

donotpush commented Aug 7, 2024 •

edited

Loading

netlify bot commented Aug 7, 2024 •

edited

Loading

donotpush commented Aug 20, 2024 •

edited

Loading

donotpush commented Aug 23, 2024 •

edited

Loading