dlt.transformation implementation by sh-rp · Pull Request #2528 · dlt-hub/dlt

sh-rp · 2025-04-16T12:36:03Z

Description

See attached ticket including dependencies.

Currently based on branch #2512

Followup tasks in this ticket: #2540

Direct link to hidden docs page: https://deploy-preview-2528--dlt-hub-docs.netlify.app/docs/devel/general-usage/transformations/

Todos:

Run tests on all destinations
Add / fix suggestions from prototype PR in dlt+ repo
Review main transform code, check commented out code and see what to keep and what to adapt
Decide wether to keep python based transformations
Decide on which transformation tests to keep the essential decorator
add configuration
Add test with all datatypes

netlify · 2025-04-16T12:36:21Z

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	`a49311a`
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/6839cfaf3327de0008e60eba

…2684) * changes contrib and README (#2666) * changes contrib and README * Apply suggestions from code review Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> * raises if resolving dataclass without configspec * adds function type inspect that follows wrappers * removes make fun, uses wraps * adds conftest to transformations * (1) fixes tranformation overloads (2) passes TransformationConfiguration as base spec so buffer is always injected (3) wraps tranformation_function (4) makes str SQL a model (5) tests configurations and parametrized transformations * (1) removes resources returning resources (2) allows resources to be also functions (3) allows base spec to be passed to resource function (4) makes DltResource and SourceFactory to wrap decorated function and fixes signatures (5) allows inner resources to be injectable, warns for transformers (6) normalizes and tests how functions are wrapped and unwrapped so signatures and configs are available * normalizes config resolve behavior: default values can be overriden from providers but explicit cannot. if those were instances of base configurations, behavior was inconsistent (explicit values were treated like defaults). also if native value is found for a config and it does not accept native values, config resolution will fail, previously it was ignored * do not use config specs cached in module when creating autospecs * fixes venv tests when uv is present * if incremental parses from another incremental as native value, it copies origina type correctly * merges standalone resources with regular resources: (1) all are DltResources (2) we generate the correct types for __call__! (3) all resources can be configured including inner resources and including default params, previously only standalone could. that unifies behavior for resources and sources re. config injection (4) resources can return another resources if have DltResource in type annotation (5) resources can be renamed with lambda names also sections can be renamed * fixes transformation decorators so they generate correct typing * binds params to resource function instead of using defaults to avoid generating config injection in rest_api * removes remaining full_refresh flags * fixes Makefile commands to run common and local destination tests * fixes xdg home test * fixes venv tests for uv * linter and docsstring fixes --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>

…ive initial values are supported

….13)

# Conflicts: # dlt/extract/resource.py

sh-rp · 2025-05-30T06:59:50Z

-        finally:
-            Venv.PIP_TOOL = old_tool
+
+        @contextmanager


contextmanager on staticmethod does not work on python 3.9, this workaround does.

sh-rp · 2025-05-30T07:01:06Z

 def test_drop_first_run_and_pending_packages() -> None:
    """Attempts to drop before pipeline runs and when partial loads happen"""
-    pipeline = dlt.pipeline("drop_test_" + uniq_id(), destination="dummy")
+    pipeline = dlt.pipeline("drop_test_" + uniq_id(), destination="duckdb")


I changed this because this test was previously failing with a configuration error for the dummy destination. now another test is failing with the same error, so it must be something that happens only the first time the dummy is used? does not happen locally on my mac.

sh-rp · 2025-05-30T07:02:52Z

+
+                # make dict immutable: TODO: fix propery
+                # somehow default_factory does not work here..
+                if isinstance(att_value, dict):


we are creating dataclasses with a mutable dict value in some cases (when the default value is an assigned dict, this happens in one of the verified sources and will fail the tests where all verified sources are created). I tried to use the default_factory workaround from below, but somehow it did not work. Changing the value into something immutable fixes this, but I'm not sure this is right. Will also happen for lists.

sh-rp · 2025-05-30T07:14:01Z


        schema_update: TSchemaUpdate = {}
        schema = self.schema
-        dialect = self.config.destination_capabilities.sqlglot_dialect


for now I just use the dialect from the model file. We can also make sure that the injected capabilities take the configuration into account with a small change in pipeline, but imho we should probably raise here if model dialect and destination dialect are not the same..

imho we should probably raise here if model dialect and destination dialect are not the same..
I disagree here. It's a meaningful feature to be able to write all transformations the dialect you know best (e.g., BigQuery) and have them work everywhere.

Maybe we missed the transpile step in a few places. We should keep that for a follow-up ticket

We could rename dialect to a more meaningful destination_dialect and query_dialect depending on where it appears.

# Conflicts: # poetry.lock

* triggers devel tests * fixed malformed docstring * use native sqlglot type annotation * pass hints via SQLGlot metadata * fix linter errors and tests * fix a few more tests and edge cases * fix bug in lineage * enable columns schema for both ReadableRelation Types * add more tests and make lineage tests independent from loading * add lineage tests for all sql destinations * enable tests on ci and disable column schema for sqlalchemy for now * fix some more tests * add sqlalchemy hack * first fix for snowflake and some smaller chnages and clarifications * fix sqlglot schema creation, makes clickhouse work * re-add transformations tests folder * fix lineage datatype * disable databricks and synapse ibis backend tests * move transformation code from prototype excluding old lineage and including updates so that linter passes, no real code changes yet. * fix some of the python extractor based transformations * fix most tests * make basic transformation tests run on all destinations * enable all current transformation tests for all destinations run some duckdb transformations on all OSes * a little bit of cleanup * move common transactions and mark all destination transaction tests as essential for now * Add improvements from review in prototype PR and some cleanup * exclude dremio * fix some transformations tests * fix row_counts for snowflake and add some comments * converts SupportsReadableRelation to an ABC * add scalar access to SupportsReadableRelation * simplify transformation signature * add top level dlt objects and some small changes * second part of removing transformation extra args * add clickhouse tests * add config based transformation source * add better transformation examples * use fruitshop template for testing * remove custom row_counts method in favor of "global" test one * first draft of transformations doc * some work on the docs page * feat: 2540 lineage `allow_unknown_columns` and `allow_anonymous_columns` (dlt-hub#2577) * test compute_columns_schema() and exception handling * convert transformation code examples to snippets * finish first round of transformation docs * Quite a few PR fixes * fixes some tests * add support and docs for dataframe and arrow operations * add config and fallback if destination not reachable * fix scalar method fallback to models if pipeline destination is not available * hopefully fix one test * Docs: addition of normalizer behaviour to transformations docs (dlt-hub#2639) * Normalizer info added * Unnecessary paragraph removed, regular normalization linked * feat: 2540 - SQLGlot type mapping (dlt-hub#2587) * fixes some tests * post rebase cleanup * renamed kwarg * type handling done; WIP * sqlglot-dlt type mapping completed * added docstrings to tests * removed unused test file * attach metadata to DataType * refactored test to parameterized form * refactor function names * bug fix .to_py() * rename compute_columns_schema() kwargs * refactor type conversion branches * fixes some tests * add support and docs for dataframe and arrow operations * add config and fallback if destination not reachable * fix scalar method fallback to models if pipeline destination is not available * fix: update return type in athena_adapter docstring to reflect correct destination (dlt-hub#2599) * list secrets in vault config provider to avoid calls to backend (dlt-hub#2597) * fixes bug where configuration section was not propagated when embedded configuration is resolved * splits vault provider settings per vault type * adds option to list secrets to vault and google secrets provider * uses google secrets provider with global cache for tests * documents vault provider * test and docs fixes * slightly clarify clickhouse docs (dlt-hub#2594) * slightly clarify clickhouse docs * Update clickhouse.md * Extract dataset code snippets into tests snippets system (dlt-hub#2598) * extracts dataset code blocks into tested snippets and uses fruitshop pipeline as base dataset for demonstration purposes * add ibis group * Enabling 'model' loader_file_format for athena, synapse and dremio (dlt-hub#2556) * Athena model loader format initial support * test_verify_capabilities_data_types adjusted for athena * Synapse enabled * The offset logic for tsql made unreachable * Athena test config without iceberg removed, dremio added * Unnecessary synapse workaround removed * fix some typos in cursor-restapi docs (dlt-hub#2608) * fix some typos in cursor-restapi docs * fix typo * refactor init-command for use in dlt project (dlt-hub#2568) * refactor init-command for use in dlt project * remove config.toml from project docs * fix ibis mypy error --------- Co-authored-by: dave <shrps@posteo.net> * docs: Fix incorrect nesting in secrets.toml (dlt-hub#2614) * fixes parquet data writer settings docs & rewrites configuration docs (dlt-hub#2583) * fixes parquet data writer settings docs * adds section to dlt resource decorator * fixes and tests how config sections are created when single resource is extracted * fixes config sections for parallel doc example * exports postgres adapter * rewrites configuration docs, moves a few docs sections in sidebar * snippet fixes * accepts docs changes from review Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> * adds tip how to eject core source * linter fixes --------- Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> * enables fsspec per-thread instance cache and updates documentation (dlt-hub#2621) * bumps pendulum and docs (dlt-hub#2624) * fixes sql database docstrings and docs * bumps poetry to 3.0.1 and drop dlt poetry * Added dedup sort example (dlt-hub#2235) * Added dedup sort example * Updated formatting * Updated * Updated * Update docs/website/docs/general-usage/incremental-loading.md --------- Co-authored-by: Alena Astrakhantseva <alena@dlthub.com> Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org> * Docs: add advanced project tutorial (dlt-hub#2338) * hopefully fix one test * trigger ci * improve tests, lint --------- Co-authored-by: David Scharf <shrps@posteo.net> Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> Co-authored-by: rudolfix <rudolfix@rudolfix.org> Co-authored-by: anuunchin <88698977+anuunchin@users.noreply.github.com> Co-authored-by: hsm207 <hsm207@users.noreply.github.com> Co-authored-by: djudjuu <djudju@proton.me> Co-authored-by: Alexander Grueneberg <com.github@agrueneberg.info> Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> Co-authored-by: dat-a-man <98139823+dat-a-man@users.noreply.github.com> Co-authored-by: Alena Astrakhantseva <alena@dlthub.com> * qualify all queries that come into the transformations * fix lineage for snowflake and clickhouse lineage * apply schema fix for sqlglot and remove special treatment of snowflake * align datasets interfaces with ibis implementation ["col"] selects column and not table with one column * disable incremental on transformations decorator and warn if incremental args are discovered * fixes one more test * fixes snowflake tests after sqlglot schema fix * removes standalone resources, fixes transformation function wrapping (dlt-hub#2684) * changes contrib and README (dlt-hub#2666) * changes contrib and README * Apply suggestions from code review Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> * raises if resolving dataclass without configspec * adds function type inspect that follows wrappers * removes make fun, uses wraps * adds conftest to transformations * (1) fixes tranformation overloads (2) passes TransformationConfiguration as base spec so buffer is always injected (3) wraps tranformation_function (4) makes str SQL a model (5) tests configurations and parametrized transformations * (1) removes resources returning resources (2) allows resources to be also functions (3) allows base spec to be passed to resource function (4) makes DltResource and SourceFactory to wrap decorated function and fixes signatures (5) allows inner resources to be injectable, warns for transformers (6) normalizes and tests how functions are wrapped and unwrapped so signatures and configs are available * normalizes config resolve behavior: default values can be overriden from providers but explicit cannot. if those were instances of base configurations, behavior was inconsistent (explicit values were treated like defaults). also if native value is found for a config and it does not accept native values, config resolution will fail, previously it was ignored * do not use config specs cached in module when creating autospecs * fixes venv tests when uv is present * if incremental parses from another incremental as native value, it copies origina type correctly * merges standalone resources with regular resources: (1) all are DltResources (2) we generate the correct types for __call__! (3) all resources can be configured including inner resources and including default params, previously only standalone could. that unifies behavior for resources and sources re. config injection (4) resources can return another resources if have DltResource in type annotation (5) resources can be renamed with lambda names also sections can be renamed * fixes transformation decorators so they generate correct typing * binds params to resource function instead of using defaults to avoid generating config injection in rest_api * removes remaining full_refresh flags * fixes Makefile commands to run common and local destination tests * fixes xdg home test * fixes venv tests for uv * linter and docsstring fixes --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> * allows for initial values that are configurations also in case no native initial values are supported * fixes docs linting * Outer select quotes columns (dlt-hub#2694) * fix normalizer tests * fix a few small tests * remove dependency on ibis for common tests (not supported on python 3.13) * fixes for python 3.9 * fix sqlglot schema propagation and retrieval * fixes leaking sqlalchemy credentials into other test * skip not materialized columns in sqlglot schema generation --------- Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org> Co-authored-by: zilto <zilto@github.com> Co-authored-by: Thierry Jean <68975210+zilto@users.noreply.github.com> Co-authored-by: anuunchin <88698977+anuunchin@users.noreply.github.com> Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> Co-authored-by: hsm207 <hsm207@users.noreply.github.com> Co-authored-by: djudjuu <djudju@proton.me> Co-authored-by: Alexander Grueneberg <com.github@agrueneberg.info> Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> Co-authored-by: dat-a-man <98139823+dat-a-man@users.noreply.github.com> Co-authored-by: Alena Astrakhantseva <alena@dlthub.com>

* triggers devel tests * fixed malformed docstring * use native sqlglot type annotation * pass hints via SQLGlot metadata * fix linter errors and tests * fix a few more tests and edge cases * fix bug in lineage * enable columns schema for both ReadableRelation Types * add more tests and make lineage tests independent from loading * add lineage tests for all sql destinations * enable tests on ci and disable column schema for sqlalchemy for now * fix some more tests * add sqlalchemy hack * first fix for snowflake and some smaller chnages and clarifications * fix sqlglot schema creation, makes clickhouse work * re-add transformations tests folder * fix lineage datatype * disable databricks and synapse ibis backend tests * move transformation code from prototype excluding old lineage and including updates so that linter passes, no real code changes yet. * fix some of the python extractor based transformations * fix most tests * make basic transformation tests run on all destinations * enable all current transformation tests for all destinations run some duckdb transformations on all OSes * a little bit of cleanup * move common transactions and mark all destination transaction tests as essential for now * Add improvements from review in prototype PR and some cleanup * exclude dremio * fix some transformations tests * fix row_counts for snowflake and add some comments * converts SupportsReadableRelation to an ABC * add scalar access to SupportsReadableRelation * simplify transformation signature * add top level dlt objects and some small changes * second part of removing transformation extra args * add clickhouse tests * add config based transformation source * add better transformation examples * use fruitshop template for testing * remove custom row_counts method in favor of "global" test one * first draft of transformations doc * some work on the docs page * feat: 2540 lineage `allow_unknown_columns` and `allow_anonymous_columns` (#2577) * test compute_columns_schema() and exception handling * convert transformation code examples to snippets * finish first round of transformation docs * Quite a few PR fixes * fixes some tests * add support and docs for dataframe and arrow operations * add config and fallback if destination not reachable * fix scalar method fallback to models if pipeline destination is not available * hopefully fix one test * Docs: addition of normalizer behaviour to transformations docs (#2639) * Normalizer info added * Unnecessary paragraph removed, regular normalization linked * feat: 2540 - SQLGlot type mapping (#2587) * fixes some tests * post rebase cleanup * renamed kwarg * type handling done; WIP * sqlglot-dlt type mapping completed * added docstrings to tests * removed unused test file * attach metadata to DataType * refactored test to parameterized form * refactor function names * bug fix .to_py() * rename compute_columns_schema() kwargs * refactor type conversion branches * fixes some tests * add support and docs for dataframe and arrow operations * add config and fallback if destination not reachable * fix scalar method fallback to models if pipeline destination is not available * fix: update return type in athena_adapter docstring to reflect correct destination (#2599) * list secrets in vault config provider to avoid calls to backend (#2597) * fixes bug where configuration section was not propagated when embedded configuration is resolved * splits vault provider settings per vault type * adds option to list secrets to vault and google secrets provider * uses google secrets provider with global cache for tests * documents vault provider * test and docs fixes * slightly clarify clickhouse docs (#2594) * slightly clarify clickhouse docs * Update clickhouse.md * Extract dataset code snippets into tests snippets system (#2598) * extracts dataset code blocks into tested snippets and uses fruitshop pipeline as base dataset for demonstration purposes * add ibis group * Enabling 'model' loader_file_format for athena, synapse and dremio (#2556) * Athena model loader format initial support * test_verify_capabilities_data_types adjusted for athena * Synapse enabled * The offset logic for tsql made unreachable * Athena test config without iceberg removed, dremio added * Unnecessary synapse workaround removed * fix some typos in cursor-restapi docs (#2608) * fix some typos in cursor-restapi docs * fix typo * refactor init-command for use in dlt project (#2568) * refactor init-command for use in dlt project * remove config.toml from project docs * fix ibis mypy error --------- Co-authored-by: dave <shrps@posteo.net> * docs: Fix incorrect nesting in secrets.toml (#2614) * fixes parquet data writer settings docs & rewrites configuration docs (#2583) * fixes parquet data writer settings docs * adds section to dlt resource decorator * fixes and tests how config sections are created when single resource is extracted * fixes config sections for parallel doc example * exports postgres adapter * rewrites configuration docs, moves a few docs sections in sidebar * snippet fixes * accepts docs changes from review Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> * adds tip how to eject core source * linter fixes --------- Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> * enables fsspec per-thread instance cache and updates documentation (#2621) * bumps pendulum and docs (#2624) * fixes sql database docstrings and docs * bumps poetry to 3.0.1 and drop dlt poetry * Added dedup sort example (#2235) * Added dedup sort example * Updated formatting * Updated * Updated * Update docs/website/docs/general-usage/incremental-loading.md --------- Co-authored-by: Alena Astrakhantseva <alena@dlthub.com> Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org> * Docs: add advanced project tutorial (#2338) * hopefully fix one test * trigger ci * improve tests, lint --------- Co-authored-by: David Scharf <shrps@posteo.net> Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> Co-authored-by: rudolfix <rudolfix@rudolfix.org> Co-authored-by: anuunchin <88698977+anuunchin@users.noreply.github.com> Co-authored-by: hsm207 <hsm207@users.noreply.github.com> Co-authored-by: djudjuu <djudju@proton.me> Co-authored-by: Alexander Grueneberg <com.github@agrueneberg.info> Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> Co-authored-by: dat-a-man <98139823+dat-a-man@users.noreply.github.com> Co-authored-by: Alena Astrakhantseva <alena@dlthub.com> * qualify all queries that come into the transformations * fix lineage for snowflake and clickhouse lineage * apply schema fix for sqlglot and remove special treatment of snowflake * align datasets interfaces with ibis implementation ["col"] selects column and not table with one column * disable incremental on transformations decorator and warn if incremental args are discovered * fixes one more test * fixes snowflake tests after sqlglot schema fix * removes standalone resources, fixes transformation function wrapping (#2684) * changes contrib and README (#2666) * changes contrib and README * Apply suggestions from code review Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> * raises if resolving dataclass without configspec * adds function type inspect that follows wrappers * removes make fun, uses wraps * adds conftest to transformations * (1) fixes tranformation overloads (2) passes TransformationConfiguration as base spec so buffer is always injected (3) wraps tranformation_function (4) makes str SQL a model (5) tests configurations and parametrized transformations * (1) removes resources returning resources (2) allows resources to be also functions (3) allows base spec to be passed to resource function (4) makes DltResource and SourceFactory to wrap decorated function and fixes signatures (5) allows inner resources to be injectable, warns for transformers (6) normalizes and tests how functions are wrapped and unwrapped so signatures and configs are available * normalizes config resolve behavior: default values can be overriden from providers but explicit cannot. if those were instances of base configurations, behavior was inconsistent (explicit values were treated like defaults). also if native value is found for a config and it does not accept native values, config resolution will fail, previously it was ignored * do not use config specs cached in module when creating autospecs * fixes venv tests when uv is present * if incremental parses from another incremental as native value, it copies origina type correctly * merges standalone resources with regular resources: (1) all are DltResources (2) we generate the correct types for __call__! (3) all resources can be configured including inner resources and including default params, previously only standalone could. that unifies behavior for resources and sources re. config injection (4) resources can return another resources if have DltResource in type annotation (5) resources can be renamed with lambda names also sections can be renamed * fixes transformation decorators so they generate correct typing * binds params to resource function instead of using defaults to avoid generating config injection in rest_api * removes remaining full_refresh flags * fixes Makefile commands to run common and local destination tests * fixes xdg home test * fixes venv tests for uv * linter and docsstring fixes --------- Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> * allows for initial values that are configurations also in case no native initial values are supported * fixes docs linting * Outer select quotes columns (#2694) * fix normalizer tests * fix a few small tests * remove dependency on ibis for common tests (not supported on python 3.13) * fixes for python 3.9 * fix sqlglot schema propagation and retrieval * fixes leaking sqlalchemy credentials into other test * skip not materialized columns in sqlglot schema generation --------- Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org> Co-authored-by: zilto <zilto@github.com> Co-authored-by: Thierry Jean <68975210+zilto@users.noreply.github.com> Co-authored-by: anuunchin <88698977+anuunchin@users.noreply.github.com> Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com> Co-authored-by: hsm207 <hsm207@users.noreply.github.com> Co-authored-by: djudjuu <djudju@proton.me> Co-authored-by: Alexander Grueneberg <com.github@agrueneberg.info> Co-authored-by: Violetta Mishechkina <sansiositres@gmail.com> Co-authored-by: dat-a-man <98139823+dat-a-man@users.noreply.github.com> Co-authored-by: Alena Astrakhantseva <alena@dlthub.com>

sh-rp self-assigned this Apr 16, 2025

sh-rp added the enhancement New feature or request label Apr 16, 2025

sh-rp linked an issue Apr 16, 2025 that may be closed by this pull request

Transformations #2527

Closed

sh-rp force-pushed the feat/2527-transformations branch 8 times, most recently from edbe0e5 to 411f3da Compare April 22, 2025 14:47

triggers devel tests

fef9c7e

sh-rp force-pushed the feat/2527-transformations branch from 411f3da to cdb11a6 Compare April 22, 2025 16:02

zilto and others added 15 commits April 22, 2025 18:35

fixed malformed docstring

76c6849

use native sqlglot type annotation

11102a3

pass hints via SQLGlot metadata

fb13202

fix linter errors and tests

cc05ac4

fix a few more tests and edge cases

ab9753a

fix bug in lineage

74f2350

enable columns schema for both ReadableRelation Types

30b3336

add more tests and make lineage tests independent from loading

9042285

add lineage tests for all sql destinations

054bee7

enable tests on ci and disable column schema for sqlalchemy for now

06fe918

fix some more tests

cb3cd1f

add sqlalchemy hack

5b7b2c8

first fix for snowflake and some smaller chnages and clarifications

a225ee7

fix sqlglot schema creation, makes clickhouse work

8ac873b

re-add transformations tests folder

fe0699a

sh-rp force-pushed the feat/2527-transformations branch from cdb11a6 to 8d03514 Compare April 22, 2025 16:35

sh-rp and others added 7 commits May 28, 2025 11:24

fixes snowflake tests after sqlglot schema fix

c16b0d8

allows for initial values that are configurations also in case no nat…

3af080e

…ive initial values are supported

fixes docs linting

64c7828

Outer select quotes columns (#2694)

99c548d

fix normalizer tests

55aa146

fix a few small tests

1b3f800

sh-rp force-pushed the feat/2527-transformations branch from 23f0c2b to 1b3f800 Compare May 28, 2025 20:09

sh-rp added 2 commits May 29, 2025 08:29

remove dependency on ibis for common tests (not supported on python 3…

5224ec6

….13)

fixes for python 3.9

b2cbb96

sh-rp force-pushed the feat/2527-transformations branch from b6982c8 to b2cbb96 Compare May 29, 2025 07:05

fix sqlglot schema propagation and retrieval

4d12dd7

sh-rp force-pushed the feat/2527-transformations branch from 0d49c06 to 4d12dd7 Compare May 29, 2025 23:28

Merge branch 'devel' into feat/2527-transformations

db7628d

# Conflicts: # dlt/extract/resource.py

sh-rp commented May 30, 2025

View reviewed changes

rudolfix and others added 3 commits May 30, 2025 11:41

fixes leaking sqlalchemy credentials into other test

da2fde0

Merge branch 'devel' into feat/2527-transformations

5bb39b2

skip not materialized columns in sqlglot schema generation

bb32193

sh-rp force-pushed the feat/2527-transformations branch 3 times, most recently from e4235a9 to bb32193 Compare May 30, 2025 12:12

Merge branch 'devel' into feat/2527-transformations

a49311a

# Conflicts: # poetry.lock

sh-rp force-pushed the feat/2527-transformations branch from 4269b43 to a49311a Compare May 30, 2025 15:33

sh-rp merged commit 5ceba48 into devel May 30, 2025
45 checks passed

sh-rp deleted the feat/2527-transformations branch May 30, 2025 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dlt.transformation implementation#2528

dlt.transformation implementation#2528
sh-rp merged 77 commits intodevelfrom
feat/2527-transformations

sh-rp commented Apr 16, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Apr 16, 2025 •

edited

Loading

Uh oh!

sh-rp May 30, 2025

Uh oh!

sh-rp May 30, 2025

Uh oh!

sh-rp May 30, 2025

Uh oh!

sh-rp May 30, 2025

Uh oh!

zilto May 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sh-rp commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Todos:

Uh oh!

netlify Bot commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for dlt-hub-docs canceled.

Uh oh!

sh-rp May 30, 2025

Choose a reason for hiding this comment

Uh oh!

sh-rp May 30, 2025

Choose a reason for hiding this comment

Uh oh!

sh-rp May 30, 2025

Choose a reason for hiding this comment

Uh oh!

sh-rp May 30, 2025

Choose a reason for hiding this comment

Uh oh!

zilto May 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sh-rp commented Apr 16, 2025 •

edited

Loading

netlify Bot commented Apr 16, 2025 •

edited

Loading