feat(datasets): make table write mode configurable #1093

gitgud5000 · 2025-05-31T01:33:54Z

TODO(deepyaman): Move notes on credentials support to a new PR from @gitgud5000 with those changes, once it exists. I'm not deleting the detailed notes from here until they're moved elsewhere.

feat(datasets): ibis.TableDataset add mode and credentials support

feat: Introduces a mode parameter for save operations, allowing "append", "overwrite", "error", and "ignore" options to control write behavior.
- Supports legacy overwrite flag with backward compatibility and deprecation warning.
- Adds mode dispatching logic to handle different write scenarios.
- Prevents simultaneous use of both mode and legacy overwrite to avoid ambiguity.
- Updates examples and docs to reflect new parameter.
feat: Adds a credentials parameter to accept connection info (dict or string URI), superseding connection.
- Raises a warning if both credentials and deprecated connection are provided.
- Adds backend extraction helper and adjusts _describe() to use it.
- Improves documentation to reflect the preferred usage.

Semi-breaking change: overwrite and connection are deprecated; users should migrate to mode and credentials.

Description

This PR introduces two key enhancements to ibis.TableDataset:

Configurable Save Modes:
- Adds a mode parameter to save_args similar to Spark’s DataFrameWriter.mode and Pandas’ to_csv(mode=...), with support for:
  - "append": Insert data into an existing table (requires the backend to implement insert()).
  - "overwrite": Drop and recreate the table/view.
  - "error" or "errorifexists": Fail if the table already exists.
  - "ignore": Do nothing if the table exists; otherwise, create it.
- Backward compatible: legacy overwrite=True|False maps to mode="overwrite"|"error".
- Raises an error if both mode and overwrite are specified simultaneously.
credentials Parameter:
- Introduces credentials as the preferred method for specifying Ibis backend connection configurations.
- Accepts:
  - A string connection URI.
  - A dictionary of parameters (with optional con string).
- Supersedes the older connection parameter.
- Warns if both are provided.
- Updates both _connect() and _describe() to support new functionality.

Development notes

Closes Support for inserting data in ibis.TableDataset #834 (requested by @deepyaman on Sep 14, 2024, to add “insert” support via a mode argument).
- In the parent issue Enhance current integration between Kedro and Ibis #1174, the main ask was to allow an “append”/“insert” operation for Ibis datasets.
- This implementation covers:
  - Rework the API akin to Spark’s DataFrameWriter.mode and Pandas’s to_csv(mode=…).
  - Backward compatibility with overwrite behavior.
  - Clear, explicit behavior for each mode option.
  - Support for passing credentials in different formats supported by ibis.connect()

Expand for details:

Save Mode Handling

DEFAULT_SAVE_ARGS updated to use mode="overwrite" by default.
__init__:
- Validates mode/overwrite presence.
- Maps overwrite → mode internally.
save() dispatches based on mode:
- append → calls insert() if supported, else raises NotImplementedError.
- overwrite, error, ignore → handled via create_table/create_view with appropriate overwrite flag.
_describe() includes mode.

Credentials Support

Added credentials param to init signature and docstring.
Accepts either:
- A connection string (e.g. "duckdb:///my.db")
- A dict with connection params or a con string.
_connect() routes accordingly based on type.
_get_backend_name() extracts backend info for _describe().

All supported mode options were tested using both DuckDB and Postgres backends.
credentials parsing was tested for string, dict-with-con, and dict-with-backend formats.

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Updated jsonschema/kedro-catalog-X.XX.json if necessary
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes
Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

…ble materialization - feat: Introduces a `mode` parameter for save operations, allowing "append", "overwrite", "error", and "ignore" options to control write behavior. - Supports legacy `overwrite` flag with backward compatibility and deprecation warning. - Adds mode dispatching logic to handle different write scenarios. - Updates examples and docs to reflect new parameter. - Prevents simultaneous use of both `mode` and legacy `overwrite` to avoid ambiguity. - docs: Improves documentation and usage examples for new save modes. Semi-breaking change: Replaces `overwrite` save argument with `mode`; users should update configurations to use `mode`. Signed-off-by: gitgud5000 <[email protected]>

Signed-off-by: gitgud5000 <[email protected]>

deepyaman

Thanks for the PR!

Did a quick-pass review. I see you're still actively updating, so I'll revisit this later.

kedro-datasets/kedro_datasets/ibis/table_dataset.py

deepyaman · 2025-05-31T02:09:25Z

kedro-datasets/kedro_datasets/ibis/table_dataset.py


        self._materialized = self._save_args.pop("materialized")

+        # Handle mode / overwrite conflict


Wonder if insert + view needs to be disallowed; I haven't actually checked yet.

gitgud5000 · 2025-05-31T03:05:24Z

I've been activily working with this because I need the features now 😅.

I've also added support for passing credentials to the ibis.TableDataset to this PR I'll update the OP shortly to reflect this.
Is that ok or should I create a separate PR for this feature? @deepyaman

…precate connection - feat: introduce a new `credentials` parameter to accept connection info as a string or dict, preferred over the deprecated `connection` param - warn if both `credentials` and `connection` are provided, prioritizing `credentials` - support connection strings and dicts with connection strings for backend connections - feat: add backend name extraction to be used in `_describe()` method - docs: update docstrings to explain `credentials` and deprecation of `connection` !Semi- Breaking Change: deprecates the `connection` parameter in favor of `credentials` Signed-off-by: gitgud5000 <[email protected]>

…lacing the function dispatch dict with direct conditional handling, improving clarity and maintainability - docs: expands and clarifies docstring to explain available save modes and their effects, referencing Spark semantics for familiarity Signed-off-by: gitgud5000 <[email protected]>

Signed-off-by: gitgud5000 <[email protected]>

deepyaman · 2025-05-31T23:54:38Z

I've been activily working with this because I need the features now 😅.

Best reason. 😁

I've also added support for passing credentials to the ibis.TableDataset to this PR I'll update the OP shortly to reflect this. Is that ok or should I create a separate PR for this feature? @deepyaman

Sorry, I didn't check in time; ideal is a separate PR, but it's probably no big deal to have it in the same. If it ends up being a blocker to get something merged, can always split it out later.

- fix: add early return for empty DataFrame - fix: ensure table creation occurs before insert operations in append mode when table doesn't exist Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 · 2025-08-18T12:55:24Z

Could you please take a look at this pull request, @ankatiyar @ravi-kumar-pilla?

ankatiyar · 2025-08-18T12:59:49Z

@gitgud5000 I'll take a look, in the meanwhile could you add some unit tests to go along with the changes? :D

- fix: remove lingering 'mode' key when mapping legacy overwrite to prevent unexpected writer kwargs - fix: treat empty pandas DataFrame as a no-op in save, supporting both pandas and ibis tables Prevents accidental parameter leakage and avoids errors when saving empty data. Signed-off-by: gitgud5000 <[email protected]>

…and legacy overwrite behavior Signed-off-by: gitgud5000 <[email protected]>

… new save modes and credentials parameter Signed-off-by: gitgud5000 <[email protected]>

…-support

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 · 2025-08-18T15:32:56Z

@ankatiyar Done ✅

I wasn’t able to resolve a couple of things:

All the tests I added pass locally, except for test_connection_config with the mssql config.

=========================== short test summary info ============================
FAILED kedro-datasets/tests/ibis/test_table_dataset.py::TestTableDataset::test_connection_config[None-None-None-connection_config1-key1]

The lint check pipeline is still failing, and I’m not sure why or how to fix it. Would appreciate your guidance 🙂

Signed-off-by: Ankita Katiyar <[email protected]>

Signed-off-by: Deepyaman Datta <[email protected]>

ankatiyar · 2025-10-08T09:49:55Z

The code looks good to me! I'm wondering if we need to support overwrite and mode both and then remove overwrite later instead of just supporting mode and doing a major release for kedro-datasets? There's some new datasets in the works (though experimental) that could warrant a breaking release? (cc @rashidakanchwala)

rashidakanchwala · 2025-10-08T10:06:54Z

yes, the next release will likely be a major one (given the new GenAI datasets), so doing the breaking change now could make sense.

That said, I’m not sure how widely the Ibis dataset is used or whether existing users would expect backward compatibility / a deprecation phase before overwrite is removed. If usage is low, it might be fine to go straight to mode and make it part of the next major.

deepyaman · 2025-10-08T17:08:34Z

That said, I’m not sure how widely the Ibis dataset is used or whether existing users would expect backward compatibility / a deprecation phase before overwrite is removed. If usage is low, it might be fine to go straight to mode and make it part of the next major.

I believe there are a fair number of users, and I'd rather keep the deprecation warning given that (1) it's already implemented and (2) it's not some complicated functionality to maintain, and there are no significant tradeoffs/downsides to having the functionality.

yes, the next release will likely be a major one (given the new GenAI datasets), so doing the breaking change now could make sense.

What is the timeline for the next release? @gitgud5000 also implemented credentials support as part of this PR that we were split out/are moving to a follow up. I assume that would also be good to get in for the next release.

kedro-datasets/RELEASE.md

Signed-off-by: Deepyaman Datta <[email protected]>

ankatiyar

LGTM! Thanks @gitgud5000 and @deepyaman 💯

…de argument (PR kedro-org#1093) and credentials support in the upcoming changes section. Signed-off-by: gitgud5000 <[email protected]>

@deepyaman

* Add credentials parameter to TableDataset for authentication Signed-off-by: gitgud5000 <[email protected]> * Add tests for TableDataset credentials functionality Signed-off-by: gitgud5000 <[email protected]> * test(ibis): suppress false positive secret warning Signed-off-by: gitgud5000 <[email protected]> * style(tests): apply lint and formatting fixes Signed-off-by: gitgud5000 <[email protected]> * fix(ibis): add credentials type validation to TableDataset. Appease mypy and surface explicit errors for invalid credential inputs. Signed-off-by: gitgud5000 <[email protected]> * fix(ibis): Fix formatting and lint issues Signed-off-by: gitgud5000 <[email protected]> * fix: secret false positive Signed-off-by: gitgud5000 <[email protected]> * fix: formatting Signed-off-by: gitgud5000 <[email protected]> * fix(tests): remove commented-out references from upstream examples Signed-off-by: gitgud5000 <[email protected]> * fix(RELEASE.md): update documentation for `ibis.TableDataset` with mode argument (PR #1093) and credentials support in the upcoming changes section. Signed-off-by: gitgud5000 <[email protected]> * fix: docs Signed-off-by: gitgud5000 <[email protected]> * fix: docs... Signed-off-by: gitgud5000 <[email protected]> * Address comments by @deepyaman: simplify `TableDataset` credentials merging; update docs and tests Signed-off-by: gitgud5000 <[email protected]> * fix credentials type annotation to resolves a mypy false positive. Signed-off-by: gitgud5000 <[email protected]> * feat(schema): add optional credentials object to catalog JSON schema Signed-off-by: gitgud5000 <[email protected]> * fix test determine backend from dataset config and add credentials-override case Signed-off-by: gitgud5000 <[email protected]> --------- Signed-off-by: gitgud5000 <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 2e9e539 to a41bb19 Compare May 31, 2025 01:41

chore: reformat code with Ruff for consistency

1f609c5

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from a41bb19 to 1f609c5 Compare May 31, 2025 01:44

deepyaman reviewed May 31, 2025

View reviewed changes

gitgud5000 changed the title ~~feat(datasets): ibis.TableDataset add configurable save mode for table materialization~~ feat(datasets): ibis.TableDataset add configurable save mode for table materialization and credentials support May 31, 2025

gitgud5000 added 3 commits May 31, 2025 03:51

fix: added missing 'errorifexists' to conditions in save modes

6445239

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from d545871 to 6445239 Compare May 31, 2025 03:52

Merge branch 'main' into ibis-dataset-savemode-support

256795d

ankatiyar requested a review from deepyaman June 3, 2025 14:52

gitgud5000 marked this pull request as draft June 4, 2025 11:13

gitgud5000 marked this pull request as ready for review August 15, 2025 18:49

fix(datasets): handle empty DataFrames and improve append mode

6130581

- fix: add early return for empty DataFrame - fix: ensure table creation occurs before insert operations in append mode when table doesn't exist Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 40bbfe3 to 6130581 Compare August 15, 2025 20:15

gitgud5000 added 2 commits August 18, 2025 10:31

test(tabledataset): add tests for save modes: append, error, ignore, …

55b8666

…and legacy overwrite behavior Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from db200f3 to 55b8666 Compare August 18, 2025 14:31

gitgud5000 added 4 commits August 18, 2025 10:40

docs(RELEASE.md): update major features and improvements section with…

5a84f44

… new save modes and credentials parameter Signed-off-by: gitgud5000 <[email protected]>

Merge remote-tracking branch 'origin/main' into ibis-dataset-savemode…

7b04389

…-support

fix lint

6cfb8fe

Signed-off-by: gitgud5000 <[email protected]>

docs(table_dataset): fix indentation in docstring

2f50c6d

Signed-off-by: gitgud5000 <[email protected]>

Fix lint

4c6ddb9

Signed-off-by: Ankita Katiyar <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 63b95c1 to 4c6ddb9 Compare August 18, 2025 16:38

refactor(datasets): define save mode using an enum

513aab0

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from 1057b14 to 513aab0 Compare September 23, 2025 23:44

deepyaman changed the title ~~feat(datasets): ibis.TableDataset add configurable save mode for table materialization and credentials support~~ feat(datasets): make table write mode configurable Sep 23, 2025

deepyaman added 9 commits September 23, 2025 22:08

chore(datasets): remove notes unrelated to this PR

2629069

Signed-off-by: Deepyaman Datta <[email protected]>

chore(datasets): reword release note on save modes

c4534e5

Signed-off-by: Deepyaman Datta <[email protected]>

refactor(datasets): simplify branching in save()

48c5e85

Signed-off-by: Deepyaman Datta <[email protected]>

revert(datasets): remove extra blank added to file

c75643d

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): clean up any tables that we create

b4ba79a

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): condense multi-parametrizes to one

4299844

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): check raises a narrower error type

5f9fc08

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): test legacy overwrite with fixture

4946541

Signed-off-by: Deepyaman Datta <[email protected]>

chore(datasets): use <= instead of .issubset()

55ba8a0

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from 62d976a to 55ba8a0 Compare September 24, 2025 19:29

deepyaman requested review from ankatiyar and ravi-kumar-pilla September 26, 2025 17:15

ankatiyar reviewed Oct 8, 2025

View reviewed changes

kedro-datasets/RELEASE.md Outdated Show resolved Hide resolved

chore(datasets): warn if using deprecated argument

6912ec6

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from 586c24c to 6912ec6 Compare October 12, 2025 20:40

Merge branch 'main' into ibis-dataset-savemode-support

de43c76

deepyaman requested a review from ankatiyar October 12, 2025 20:41

deepyaman enabled auto-merge (squash) October 12, 2025 22:40

ankatiyar approved these changes Oct 13, 2025

View reviewed changes

Merge branch 'main' into ibis-dataset-savemode-support

83bf582

deepyaman merged commit 170cc68 into kedro-org:main Oct 13, 2025
13 of 14 checks passed

gitgud5000 mentioned this pull request Oct 19, 2025

feat(datasets): support TableDataset credentials #1218

Merged

6 tasks


		self._materialized = self._save_args.pop("materialized")

		# Handle mode / overwrite conflict

feat(datasets): make table write mode configurable #1093

feat(datasets): make table write mode configurable #1093

Uh oh!

Conversation

gitgud5000 commented May 31, 2025 • edited by deepyaman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Development notes

Save Mode Handling

Credentials Support

Checklist

Uh oh!

deepyaman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepyaman May 31, 2025

Choose a reason for hiding this comment

Uh oh!

gitgud5000 commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepyaman commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitgud5000 commented Aug 18, 2025

Uh oh!

ankatiyar commented Aug 18, 2025

Uh oh!

gitgud5000 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ankatiyar commented Oct 8, 2025

Uh oh!

rashidakanchwala commented Oct 8, 2025

Uh oh!

deepyaman commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ankatiyar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gitgud5000 commented May 31, 2025 •

edited by deepyaman

Loading

gitgud5000 commented May 31, 2025 •

edited

Loading

deepyaman commented May 31, 2025 •

edited

Loading

gitgud5000 commented Aug 18, 2025 •

edited

Loading

deepyaman commented Oct 8, 2025 •

edited

Loading