Update `transpile` to support `--overrides-path` and `--target-technology` arguments #2072

asnare · 2025-10-02T13:03:07Z

Changes

This PR updates our transpile command to support some additional arguments:

--overrides-file: for specifying an overrides (JSON) file with BB-based conversions.
--target-technology: for specifying the target technology when using BB to perform ETL conversions.

Relevant implementation details

Adding support for these options uncovered some related problems relating to the way dialect-specific options are configured and handled at runtime. This meant some yak-shaving, including:

During transpile, always checking the dialect-specific options even if the configuration file is missing or invalid.
Properly handling the <none> sentinel value used by BB at both configuration time and transpile-time.
Tidying up some of the unit test fixtures and tests that cover this area of the code.

Caveats/things to watch out for when reviewing:

Previously it looks like during install-transpile if the user hit enter on the overrides-file option they would end up with the string "None" as the value in the transpiler_options mapping. It's unclear to me where this was ultimately dealt with, or whether these configurations will just cause transpile to fail.

The integration tests here are flaky, and it's unclear why.
[Late addition: I've just added some additional plumbing to the integration tests to help with understanding why they're flaky. For some reason the underlying dbxconv process sometimes terminates with 255 as the exit code. Hopefully this will help understand why.]

Linked issues

Resolves #2070.

Functionality

updated relevant user documentation
modified existing command: databricks labs lakebridge install

Tests

manually tested
added unit tests
added integration tests

…nts during `transpile`.

github-actions · 2025-10-02T13:08:23Z

✅ 32/32 passed, 6 flaky, 2m29s total

Flaky tests:

🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (13.438s)
🤪 test_transpile_teradata_sql_non_interactive[False] (14.427s)
🤪 test_transpiles_informatica_to_sparksql (13.532s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.081s)
🤪 test_transpile_teradata_sql (18.502s)
🤪 test_transpile_teradata_sql_non_interactive[True] (7.398s)

_{Running from acceptance #2603}

Other changes include: - Always test against the current (published) version of the BB plugin. - Only install and set up the plugin once. - Include the config/loading (from the workspace) and verification. - Check more of the details from running the transpiler, including the JSON output summary. - Remove the async bits.

…ions attribute of the configuration.

…g dash ('-').

…and try to continue. Bailing out will lead to a crash.

… way.

…eded, and can be provided as a command-line argument.

…nfigured and work.

This is intended to help with debugging.

sundarshankar89

LGTM, I know why this is required, but we need to simplify on a better lsp_config.yml so this plumbing goes away.

src/databricks/labs/lakebridge/cli.py

sundarshankar89 · 2025-10-08T06:31:08Z

LGTM, I know why this is required, but we need to simplify on a better lsp_config.yml so this plumbing goes away.

Bold Thought, what if we have install interface from the server side ?

asnare · 2025-10-08T09:20:23Z

src/databricks/labs/lakebridge/config.py

        if self.method == LSPPromptMethod.CONFIRM:
            return prompts.confirm(self.prompt)
        if self.method == LSPPromptMethod.QUESTION:
-            default = self.default if self.default else "None"


@sundarshankar89: This is the bit where I think it would insert the string "None" as a value into the config.yml instead of null.

m-abulazm

not blocking. some suggestions need alignment first

docs/lakebridge/docs/installation.mdx

docs/lakebridge/docs/transpile/index.mdx

src/databricks/labs/lakebridge/cli.py

m-abulazm · 2025-10-08T10:02:30Z

src/databricks/labs/lakebridge/cli.py

-            return
+        if transpiler_options is None:
+            transpiler_options = {}
+        elif not isinstance(transpiler_options, Mapping):


can we properly type this since we know what it looks like instead of passing around Maps e.g.

class TranspilerOptions target_tech: str | None overrides_file: Path | None

We don't really know what it looks like though: it's filled in during the questionnaire based on a configuration file bundled with each LSP server.

It certainly can be more constrained than RootJsonValue, but we can't just use a static dataclass.

This PR adds specific support for 2 known options, but there might be others passed via this mechanism. (The code is supposed to prompt for missing values, but #2086 describes how this is futile and leads to the CLI simply hanging.)

src/databricks/labs/lakebridge/cli.py

m-abulazm · 2025-10-08T10:20:55Z

src/databricks/labs/lakebridge/cli.py

+        # As a corner case, if there is no entry present it means the user wasn't prompted. Here we have
+        # some complexity. We have two ways of obtaining a value:
+        #  - The user could provide it on the command-line, using --target-technology or --overrides-file.
+        #    Problem: via command-line options there's no way to indicate 'no value'.


why not use_target_technology and use_overrides_file set <none> if user does not provide it

That's not the way the code was authored, and configuration files are already out there.

src/databricks/labs/lakebridge/install.py

docs/lakebridge/docs/transpile/index.mdx

…ather than having the CLI check directly for the sentinel.

This matches the style from #2089.

asnare · 2025-10-08T16:05:47Z

This PR is blocked on #2089, and once that's merged into this PR the documentation changes in here will need to be refreshed.

## Changes This PR updates the `labs.yml` file used by the Databricks CLI to locate and provide `--help` for `databricks labs lakebridge` and subcommands. Changes include: - Fixing argument placeholders in flags; many of these were accidentally filled with misleading text. - Marking all flags as optional: we don't have any mandatory flags yet. - Consistent and concise styling of descriptions. For example, no trailing periods (`.`). - Removing `null` as default values; this is unnecessary. - Removing the `--skip-validation` default: it's incorrect because the default value comes from `config.yml` in the workspace, which in turn normally comes from the value provided during `install-transpile` (if any). ### Linked issues Blocks: #2072 ### Functionality - [x] updated relevant user documentation - [X] modified existing commands: - `databricks labs lakebridge [--help]` - `databricks labs lakebridge aggregates-reconcile --help` - `databricks labs lakebridge analyse --help` - `databricks labs lakebridge configure-database-provider --help` - `databricks labs lakebridge configure-reconcile --help` - `databricks labs lakebridge describe-transpile --help` - `databricks labs lakebridge install-transpile --help` - `databricks labs lakebridge reconcile --help` - `databricks labs lakebridge transpile --help` ### Tests - [X] manually tested

…bs/lakebridge into bladebridge-dialect-options

asnare added 2 commits October 2, 2025 14:56

Fix accidental boolean value where a string is intended.

9024730

Implement support for --overrides-file and --target-technology argume…

7225971

…nts during `transpile`.

asnare self-assigned this Oct 2, 2025

asnare added enhancement New feature or request feat/cli actions that are visible to the user bb converter Issues related to BB converter labels Oct 2, 2025

asnare temporarily deployed to tool October 2, 2025 13:03 — with GitHub Actions Inactive

asnare added 2 commits October 2, 2025 19:50

Mark static fixtures as session-scope.

aeb6c3f

asnare temporarily deployed to tool October 2, 2025 17:54 — with GitHub Actions Inactive

Merge branch 'main' into bladebridge-dialect-options

f1fea8c

asnare temporarily deployed to tool October 3, 2025 10:14 — with GitHub Actions Inactive

asnare added 2 commits October 3, 2025 14:27

Formatting.

aacadf2

Merge branch 'main' into bladebridge-dialect-options

da249c5

asnare temporarily deployed to tool October 6, 2025 09:05 — with GitHub Actions Inactive

asnare added 3 commits October 6, 2025 18:03

Missed a stray encoding.

af75dd0

Ensure existing unit tests properly set and verify the transpiler_opt…

3dbab6f

…ions attribute of the configuration.

Fix some old invalid fixtures for dialect flags that include a leadin…

5a1eb0e

…g dash ('-').

asnare temporarily deployed to tool October 6, 2025 16:10 — with GitHub Actions Inactive

asnare added 3 commits October 6, 2025 18:10

Internal documentation, to help the next person who looks at this.

734d469

Split some TODO markers into multiple actions.

77e733f

Don't bail out if transpiler_options is not a mapping: log a warning …

070dd38

…and try to continue. Bailing out will lead to a crash.

asnare temporarily deployed to tool October 6, 2025 16:14 — with GitHub Actions Inactive

asnare added 4 commits October 6, 2025 18:47

Properly detect no answer from the user.

13cd9ed

Internal documentation on why prompts.question() is being used this…

ec0f36b

… way.

Don't prompt for optional transpiler-specific options: they're not ne…

ae3077c

…eded, and can be provided as a command-line argument.

Internal documentation for how the transpiler-specific options are co…

79e9d40

…nfigured and work.

asnare had a problem deploying to tool October 6, 2025 16:52 — with GitHub Actions Error

Merge branch 'main' into bladebridge-dialect-options

0c711c3

asnare requested a review from m-abulazm October 6, 2025 17:20

Merge branch 'main' into bladebridge-dialect-options

f280424

asnare temporarily deployed to tool October 7, 2025 07:52 — with GitHub Actions Inactive

asnare added 2 commits October 7, 2025 19:39

Capture and log the content of errors.log during transpiler tests.

5d50103

Capture the logs from the Bladebridge LSP server during transpile tests.

a600885

This is intended to help with debugging.

asnare temporarily deployed to tool October 7, 2025 17:40 — with GitHub Actions Inactive

Formatting.

b79a94a

asnare temporarily deployed to tool October 7, 2025 18:04 — with GitHub Actions Inactive

sundarshankar89 approved these changes Oct 8, 2025

View reviewed changes

src/databricks/labs/lakebridge/cli.py Show resolved Hide resolved

src/databricks/labs/lakebridge/cli.py Show resolved Hide resolved

Merge branch 'main' into bladebridge-dialect-options

52b9fd0

asnare temporarily deployed to tool October 8, 2025 08:26 — with GitHub Actions Inactive

asnare commented Oct 8, 2025

View reviewed changes

m-abulazm reviewed Oct 8, 2025

View reviewed changes

Encapsulate the idea of an option being optional within the option, r…

a65907f

…ather than having the CLI check directly for the sentinel.

asnare temporarily deployed to tool October 8, 2025 13:06 — with GitHub Actions Inactive

Revert change in PR in this PR; another will fix this anyway.

ff56327

asnare mentioned this pull request Oct 8, 2025

Fix (and improve) the CLI documentation in labs.yml #2089

Merged

3 tasks

Update the labs.yml description of the new arguments.

b6e7793

This matches the style from #2089.

asnare temporarily deployed to tool October 8, 2025 16:02 — with GitHub Actions Inactive

asnare added the do-not-merge label Oct 8, 2025

Merge branch 'main' into bladebridge-dialect-options

39a35b3

asnare had a problem deploying to tool October 9, 2025 12:01 — with GitHub Actions Error

asnare added 2 commits October 9, 2025 14:01

Merge branch 'main' into bladebridge-dialect-options

a1f6db4

Merge branch 'bladebridge-dialect-options' of github.com:databricksla…

268b31e

…bs/lakebridge into bladebridge-dialect-options

asnare temporarily deployed to tool October 9, 2025 12:01 — with GitHub Actions Inactive

asnare requested a review from m-abulazm October 9, 2025 12:05

asnare removed the do-not-merge label Oct 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update `transpile` to support `--overrides-path` and `--target-technology` arguments #2072

Update `transpile` to support `--overrides-path` and `--target-technology` arguments #2072

asnare commented Oct 2, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 2, 2025 •

edited

Loading

Uh oh!

sundarshankar89 left a comment

Uh oh!

Uh oh!

Uh oh!

sundarshankar89 commented Oct 8, 2025

Uh oh!

asnare Oct 8, 2025

Uh oh!

m-abulazm left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

m-abulazm Oct 8, 2025

Uh oh!

asnare Oct 8, 2025

Uh oh!

Uh oh!

m-abulazm Oct 8, 2025

Uh oh!

asnare Oct 8, 2025

Uh oh!

Uh oh!

Uh oh!

asnare commented Oct 8, 2025

Uh oh!

Uh oh!

Update transpile to support --overrides-path and --target-technology arguments #2072

Are you sure you want to change the base?

Update transpile to support --overrides-path and --target-technology arguments #2072

Conversation

asnare commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

Functionality

Tests

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sundarshankar89 commented Oct 8, 2025

Uh oh!

asnare Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

m-abulazm Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

asnare Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

m-abulazm Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

asnare Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

asnare commented Oct 8, 2025

Uh oh!

Uh oh!

Update `transpile` to support `--overrides-path` and `--target-technology` arguments #2072

Update `transpile` to support `--overrides-path` and `--target-technology` arguments #2072

asnare commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading