Add support to convert ETL workloads to SDP in switch

[yyoli-db]

Changes

Add output_sdp boolean flag to lakebridge to support convert to Spark Declarative Pipeline in switch. After the change being released, we can run command like this:

databricks labs lakebridge llm-transpile \
  --input-source <local path> \
  --output-ws-folder <workspace folder> \
  --source-dialect <dialect> \
  --output_sdp true \
  --accept-terms true \
  --catalog-name <catalog> \
  --schema-name <schema> \
  --volume <volume> \

Relevant implementation details

Allowlist the boolean flag and passed it to options to switch.

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #..

Functionality

• added relevant user documentation
• added new CLI command
• modified existing command: databricks labs lakebridge ...
• ... +add your own

Tests

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

❌ Patch coverage is 55.55556% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.24%. Comparing base (eebf284) to head (12af1cc).

Files with missing lines	Patch %	Lines
...bricks/labs/lakebridge/transpiler/switch_runner.py	55.55%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2173   +/-   ##
=======================================
  Coverage   65.24%   65.24%           
=======================================
  Files         100      100           
  Lines        8506     8509    +3     
  Branches      876      877    +1     
=======================================
+ Hits         5550     5552    +2     
- Misses       2769     2770    +1     
  Partials      187      187           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 51/51 passed, 10 flaky, 3m56s total

Flaky tests:

• 🤪 test_validate_invalid_source_tech (197ms)
• 🤪 test_validate_mixed_checks (196ms)
• 🤪 test_validate_table_not_found (1ms)
• 🤪 test_validate_invalid_schema_path (1ms)
• 🤪 test_validate_non_empty_tables (7ms)
• 🤪 test_transpiles_informatica_to_sparksql (24.876s)
• 🤪 test_transpile_teradata_sql (26.56s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.024s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (5.783s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (8.436s)

_{Running from acceptance #3076}

[sundarshankar89]

Same changes need to go into SwitchDeployment _generate_switch_job_parameters method as well, which deploys the switch job during
databricks labs lakebridge install-transpile --include-llm-transpile true which deploys artifacts for switch to execute.