Implement a new `describe-transpile` CLI subcommand that describes the installed transpilers #2051

asnare · 2025-09-24T12:36:52Z

Changes

What does this PR do?

This PR implements a describe-transpile subcommand that describes the currently installed transpilers and associated dialects and configuration. This is intended for diagnostics and use by the UI. When run normally, it provides output like this:

% databricks labs lakebridge describe-transpile
Transpiler   Installed Version  Plugin Configuration
==========   =================  ====================
Morpheus     0.6.6              /Users/me/.databricks/labs/remorph-transpilers/databricks-morph-plugin/lib/config.yml
Bladebridge  0.1.15             /Users/me/.databricks/labs/remorph-transpilers/bladebridge/lib/config.yml

Supported Source Dialects
=========================
 - datastage
 - informatica (desktop edition)
 - informatica cloud
 - mssql
 - netezza
 - oracle
 - snowflake
 - synapse
 - teradata
 - tsql

When the --output=json option is provided to the Databricks CLI, more information is available:

% databricks labs lakebridge describe-transpile --output=json
{
  "available-dialects": [
    "datastage",
    "informatica (desktop edition)",
    "informatica cloud",
    "mssql",
    "netezza",
    "oracle",
    "snowflake",
    "synapse",
    "teradata",
    "tsql"
  ],
  "installed-transpilers": [
    {
      "config-path":"/Users/andrew.snare/.databricks/labs/remorph-transpilers/databricks-morph-plugin/lib/config.yml",
      "name":"Morpheus",
      "supported-dialects": {
        "snowflake": {
          "options": []
        },
        "tsql": {
          "options": []
        }
      },
      "versions": {
        "installed":"0.6.6",
        "latest":null
      }
    },
    {
      "config-path":"/Users/andrew.snare/.databricks/labs/remorph-transpilers/bladebridge/lib/config.yml",
      "name":"Bladebridge",
      "supported-dialects": {
        "datastage": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            },
            {
              "choices": [
                "SPARKSQL",
                "PYSPARK"
              ],
              "flag":"target-tech",
              "method":"CHOICE",
              "prompt":"Specify which technology should be generated"
            }
          ]
        },
        "informatica (desktop edition)": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            },
            {
              "choices": [
                "SPARKSQL",
                "PYSPARK"
              ],
              "flag":"target-tech",
              "method":"CHOICE",
              "prompt":"Specify which technology should be generated"
            }
          ]
        },
        "informatica cloud": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        },
        "mssql": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        },
        "netezza": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        },
        "oracle": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        },
        "synapse": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        },
        "teradata": {
          "options": [
            {
              "default":"\u003cnone\u003e",
              "flag":"overrides-file",
              "method":"QUESTION",
              "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none"
            }
          ]
        }
      },
      "versions": {
        "installed":"0.1.15",
        "latest":null
      }
    }
  ]
}

Relevant implementation details

The formatting of the details is handled by a new TranspilerDescription class, so that the output format can be properly controlled. (For compatibility this will need to be controlled tightly, even if the internal details change.)

Currently there is no lookup to figure out the latest version of an installed transpiler, that's out of scope for this PR although it has a position in the JSON output.

Linked issues

Additional tests in the areas modified by this code are implemented in:

Functionality

added new CLI command: databricks labs lakebridge describe-transpile

Tests

manually tested
added unit tests
added integration tests

…talled transpilers and their version.

In addition, it now sets up the repository to include version information for the transpilers.

…he installed transpilers.

github-actions · 2025-09-24T12:40:58Z

✅ 29/29 passed, 2 flaky, 1m17s total

Flaky tests:

🤪 test_transpiles_informatica_with_sparksql (9.669s)
🤪 test_transpile_sql_file (8.479s)

_{Running from acceptance #2358}

sundarshankar89

LGTM
nit: I would like to see which source is supported by which transpiler
and Docs update about new comment.

asnare · 2025-09-25T07:55:14Z

LGTM nit: I would like to see which source is supported by which transpiler and Docs update about new comment.

The dialects for each transpiler are listed in the JSON output. A choice I made here was that the normal non-JSON output is a high-level summary, and the JSON view provides the gory details.

Agreed on documentation, that's a big gap and something I need to rectify.

m-abulazm

LGTM. I just have one comment/question

m-abulazm · 2025-09-25T10:59:40Z

src/databricks/labs/lakebridge/transpiler/repository.py

        all_configs = self.all_transpiler_configs()
        return frozenset(all_configs.keys())

+    def installed_transpilers(self) -> Mapping[str, TranspilerInfo]:


could we use this for adding the transpiler version in the telemetry? and it might make sense to cache this @cached_property

We could use this for obtaining the transpiler version, but it's overkill: for telemetry you only need the version that's in use, not the whole set or other meta-data.

Regarding @cached_property I considered this but decided not to:

It's reasonably expensive to calculate, which makes it not very property-like.

For all our use-cases it's only called once. (This isn't a property that various components access on a repository instance that is passed around.)

I usually consider caching if something is expensive to calculate otherwise I dont see the benefit.

I get that it is used once but since it is public, it can happen but highly unlikely

m-abulazm

approved. also responded to the previous review

m-abulazm · 2025-09-25T11:54:41Z

src/databricks/labs/lakebridge/transpiler/repository.py

        all_configs = self.all_transpiler_configs()
        return frozenset(all_configs.keys())

+    def installed_transpilers(self) -> Mapping[str, TranspilerInfo]:


I usually consider caching if something is expensive to calculate otherwise I dont see the benefit.

I get that it is used once but since it is public, it can happen but highly unlikely

asnare added 6 commits September 23, 2025 18:47

Extend TranspilerInfo so that it can provide a description of all ins…

c16bf8e

…talled transpilers and their version.

Merge branch 'main' into transpiler-info

7e0d3a3

Support for describing the installed transpilers and their options.

50bae4c

Factor out fixture used in integration tests.

0e4f036

In addition, it now sets up the repository to include version information for the transpilers.

Integration tests for describing the installed transpilers.

bda1912

Update CLI to support a describe-transpile command that describes t…

c66afb6

…he installed transpilers.

asnare self-assigned this Sep 24, 2025

asnare requested a review from a team as a code owner September 24, 2025 12:36

asnare added the enhancement New feature or request label Sep 24, 2025

asnare temporarily deployed to tool September 24, 2025 12:36 — with GitHub Actions Inactive

asnare added the feat/cli actions that are visible to the user label Sep 24, 2025

asnare requested review from gueniai, sundarshankar89 and m-abulazm September 24, 2025 13:26

sundarshankar89 approved these changes Sep 25, 2025

View reviewed changes

Merge branch 'main' into transpiler-info

7b894b9

asnare temporarily deployed to tool September 25, 2025 07:59 — with GitHub Actions Inactive

m-abulazm reviewed Sep 25, 2025

View reviewed changes

m-abulazm approved these changes Sep 25, 2025

View reviewed changes

asnare added this pull request to the merge queue Sep 25, 2025

Merged via the queue into main with commit 8ae887a Sep 25, 2025
9 checks passed

asnare deleted the transpiler-info branch September 25, 2025 12:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement a new `describe-transpile` CLI subcommand that describes the installed transpilers #2051

Implement a new `describe-transpile` CLI subcommand that describes the installed transpilers #2051

Uh oh!

asnare commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025 •

edited

Loading

Uh oh!

sundarshankar89 left a comment

Uh oh!

asnare commented Sep 25, 2025

Uh oh!

m-abulazm left a comment

Uh oh!

m-abulazm Sep 25, 2025

Uh oh!

asnare Sep 25, 2025

Uh oh!

m-abulazm Sep 25, 2025

Uh oh!

m-abulazm left a comment

Uh oh!

m-abulazm Sep 25, 2025

Uh oh!

Uh oh!

Uh oh!

Implement a new describe-transpile CLI subcommand that describes the installed transpilers #2051

Implement a new describe-transpile CLI subcommand that describes the installed transpilers #2051

Uh oh!

Conversation

asnare commented Sep 24, 2025

Changes

What does this PR do?

Relevant implementation details

Linked issues

Functionality

Tests

Uh oh!

github-actions bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

asnare commented Sep 25, 2025

Uh oh!

m-abulazm left a comment

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

asnare Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm left a comment

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Implement a new `describe-transpile` CLI subcommand that describes the installed transpilers #2051

Implement a new `describe-transpile` CLI subcommand that describes the installed transpilers #2051

github-actions bot commented Sep 24, 2025 •

edited

Loading