Unit tests for `TranspilerRepository`, and additional integration tests #2042

asnare · 2025-09-23T16:01:07Z

Changes

This PR implements unit tests for the TranspilerRepository class, as well as extending the coverage of the integration tests. This is aimed mainly at capturing existing behaviour and ensuring bugs are not introduced.

Tests

added unit tests
added integration tests

In addition, we only load configurations in a lazy manner now.

github-actions · 2025-09-23T16:06:24Z

✅ 36/36 passed, 2 flaky, 1m16s total

Flaky tests:

🤪 test_transpiles_informatica_with_sparksql (9.687s)
🤪 test_transpile_sql_file (8.453s)

_{Running from acceptance #2395}

…e installed transpilers (#2051) ## Changes ### What does this PR do? This PR implements a `describe-transpile` subcommand that describes the currently installed transpilers and associated dialects and configuration. This is intended for diagnostics and use by the UI. When run normally, it provides output like this: ``` % databricks labs lakebridge describe-transpile Transpiler Installed Version Plugin Configuration ========== ================= ==================== Morpheus 0.6.6 /Users/me/.databricks/labs/remorph-transpilers/databricks-morph-plugin/lib/config.yml Bladebridge 0.1.15 /Users/me/.databricks/labs/remorph-transpilers/bladebridge/lib/config.yml Supported Source Dialects ========================= - datastage - informatica (desktop edition) - informatica cloud - mssql - netezza - oracle - snowflake - synapse - teradata - tsql ``` When the `--output=json` option is provided to the Databricks CLI, more information is available: ```json % databricks labs lakebridge describe-transpile --output=json { "available-dialects": [ "datastage", "informatica (desktop edition)", "informatica cloud", "mssql", "netezza", "oracle", "snowflake", "synapse", "teradata", "tsql" ], "installed-transpilers": [ { "config-path":"/Users/andrew.snare/.databricks/labs/remorph-transpilers/databricks-morph-plugin/lib/config.yml", "name":"Morpheus", "supported-dialects": { "snowflake": { "options": [] }, "tsql": { "options": [] } }, "versions": { "installed":"0.6.6", "latest":null } }, { "config-path":"/Users/andrew.snare/.databricks/labs/remorph-transpilers/bladebridge/lib/config.yml", "name":"Bladebridge", "supported-dialects": { "datastage": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" }, { "choices": [ "SPARKSQL", "PYSPARK" ], "flag":"target-tech", "method":"CHOICE", "prompt":"Specify which technology should be generated" } ] }, "informatica (desktop edition)": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" }, { "choices": [ "SPARKSQL", "PYSPARK" ], "flag":"target-tech", "method":"CHOICE", "prompt":"Specify which technology should be generated" } ] }, "informatica cloud": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] }, "mssql": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] }, "netezza": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] }, "oracle": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] }, "synapse": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] }, "teradata": { "options": [ { "default":"\u003cnone\u003e", "flag":"overrides-file", "method":"QUESTION", "prompt":"Specify the config file to override the default[Bladebridge] config - press \u003center\u003e for none" } ] } }, "versions": { "installed":"0.1.15", "latest":null } } ] } ``` ### Relevant implementation details The formatting of the details is handled by a new `TranspilerDescription` class, so that the output format can be properly controlled. (For compatibility this will need to be controlled tightly, even if the internal details change.) Currently there is no lookup to figure out the latest version of an installed transpiler, that's out of scope for this PR although it has a position in the JSON output. ### Linked issues Additional tests in the areas modified by this code are implemented in: - #2041 - #2042 ### Functionality - added new CLI command: `databricks labs lakebridge describe-transpile` ### Tests - manually tested - added unit tests - added integration tests

sundarshankar89

LGTM

asnare · 2025-09-26T13:57:10Z

❌ test_describe_installed_transpilers: AssertionError: […]

This is a flaky test on main, addressed in #2060.

The choices field for dialect-specific options now defaults to None instead of an empty list.

m-abulazm

Thank you for the thorough testing. a good example PR on how to write tests

I added a few questions inline that will guide the review forward. Regarding my remark about the interface, I dont expect to solve this in this PR but we should create a follow up to address this. because this point blocked my PR about the transpiler version telemetry so would be helpful to have a plan forward

m-abulazm · 2025-09-26T19:04:54Z

src/databricks/labs/lakebridge/transpiler/repository.py

        """
-        all_configs = self._all_transpiler_configs()
-        return {config.name: config for _, config in all_configs}
+        return {config.name: config for _, config in self._all_transpiler_configs()}


I feel we really need top standardize this. as discussed before, transpiler path is the ultimate identifier of the transpiler but all the methods that deal with transpiler pathes are private.
a few methods return transpiler names instead of path then the public interface is mostly using transpiler names not pathes.
in some places it is called product name and in other name or transpiler name. we should keep one going forward. please check my other comment

m-abulazm · 2025-09-26T19:11:44Z

tests/unit/transpiler/test_repository.py

+
+    transpiler_names = transpiler_repository.all_transpiler_names()
+
+    assert transpiler_names == {"A Transpiler", "B Transpiler"}


should we add a test for all_transpiler_configs() for this scenario

m-abulazm · 2025-09-26T19:17:14Z

tests/unit/transpiler/test_repository.py

+
+
+def test_get_installed_transpiler_version() -> None:
+    """Verify that the installed version of a transpiler can be queried."""


in some places we treat product names as transpiler names and in other places not

m-abulazm · 2025-09-26T19:22:24Z

tests/integration/transpile/test_repository.py

    assert transpilers == {'Bladebridge'}
+
+
+@pytest.mark.parametrize(("product_name", "version"), (("morpheus", "0.4.0"), ("bladebridge", "0.1.9")))


is this a moving target?

m-abulazm · 2025-09-26T19:23:18Z

tests/integration/transpile/test_repository.py

+    assert config_path.is_file()
+
+
+@pytest.mark.parametrize(


asnare added 5 commits September 23, 2025 17:37

Unit tests for TranspilerRepository.

9391b8f

In addition, we only load configurations in a lazy manner now.

Update TranspilerRepository integration tests to cover options.

96662e4

Integration tests for TranspilerRepository covering version information.

c35f041

Test fixture doesn't need to be a generator.

32b5843

Basic integration test for the default transpiler repository.

fde321f

asnare requested review from gueniai, sundarshankar89 and m-abulazm September 23, 2025 16:01

asnare self-assigned this Sep 23, 2025

asnare requested a review from a team as a code owner September 23, 2025 16:01

asnare added tech debt design flaws and other cascading effects internal technical pr's not end user facing labels Sep 23, 2025

asnare temporarily deployed to tool September 23, 2025 16:01 — with GitHub Actions Inactive

Fix formatting.

580f519

asnare temporarily deployed to tool September 23, 2025 16:45 — with GitHub Actions Inactive

Merge branch 'main' into test-transpiler-repository

a6a9401

asnare temporarily deployed to tool September 23, 2025 16:48 — with GitHub Actions Inactive

asnare mentioned this pull request Sep 24, 2025

Implement a new describe-transpile CLI subcommand that describes the installed transpilers #2051

Merged

Merge branch 'main' into test-transpiler-repository

dca730a

asnare temporarily deployed to tool September 25, 2025 07:59 — with GitHub Actions Inactive

Merge branch 'main' into test-transpiler-repository

0585485

asnare temporarily deployed to tool September 25, 2025 12:44 — with GitHub Actions Inactive

Merge branch 'main' into test-transpiler-repository

6d42ba8

asnare temporarily deployed to tool September 25, 2025 15:31 — with GitHub Actions Inactive

Merge branch 'main' into test-transpiler-repository

157c809

asnare temporarily deployed to tool September 26, 2025 09:33 — with GitHub Actions Inactive

sundarshankar89 approved these changes Sep 26, 2025

View reviewed changes

Merge branch 'main' into test-transpiler-repository

3aee3cb

asnare had a problem deploying to tool September 26, 2025 13:48 — with GitHub Actions Failure

Merge branch 'main' into test-transpiler-repository

0879697

asnare had a problem deploying to tool September 26, 2025 15:13 — with GitHub Actions Failure

Fix integration test.

18c6598

The choices field for dialect-specific options now defaults to None instead of an empty list.

asnare deployed to tool September 26, 2025 15:23 — with GitHub Actions Active

m-abulazm reviewed Sep 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unit tests for `TranspilerRepository`, and additional integration tests #2042

Unit tests for `TranspilerRepository`, and additional integration tests #2042

asnare commented Sep 23, 2025

Uh oh!

github-actions bot commented Sep 23, 2025 •

edited

Loading

Uh oh!

sundarshankar89 left a comment

Uh oh!

asnare commented Sep 26, 2025

Uh oh!

m-abulazm left a comment

Uh oh!

m-abulazm Sep 26, 2025

Uh oh!

m-abulazm Sep 26, 2025

Uh oh!

m-abulazm Sep 26, 2025

Uh oh!

m-abulazm Sep 26, 2025

Uh oh!

m-abulazm Sep 26, 2025

Uh oh!

Uh oh!


		transpiler_names = transpiler_repository.all_transpiler_names()

		assert transpiler_names == {"A Transpiler", "B Transpiler"}



		def test_get_installed_transpiler_version() -> None:
		"""Verify that the installed version of a transpiler can be queried."""

		assert transpilers == {'Bladebridge'}


		@pytest.mark.parametrize(("product_name", "version"), (("morpheus", "0.4.0"), ("bladebridge", "0.1.9")))

Unit tests for TranspilerRepository, and additional integration tests #2042

Are you sure you want to change the base?

Unit tests for TranspilerRepository, and additional integration tests #2042

Conversation

asnare commented Sep 23, 2025

Changes

Tests

Uh oh!

github-actions bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

asnare commented Sep 26, 2025

Uh oh!

m-abulazm left a comment

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

m-abulazm Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Unit tests for `TranspilerRepository`, and additional integration tests #2042

Unit tests for `TranspilerRepository`, and additional integration tests #2042

github-actions bot commented Sep 23, 2025 •

edited

Loading