Skip to content

feat: preserve typed Python data source failures - #1

Merged
Robin-Everaars merged 2 commits into
integration/dp-enginefrom
fix/typed-python-datasource-failures
Jul 29, 2026
Merged

feat: preserve typed Python data source failures#1
Robin-Everaars merged 2 commits into
integration/dp-enginefrom
fix/typed-python-datasource-failures

Conversation

@Robin-Everaars

@Robin-Everaars Robin-Everaars commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add an opt-in Python DataSource failure marker with exact terminal and transient values.
  • Preserve the finite category through DataFusion and remote task transport while discarding classified Python messages and tracebacks.
  • Map terminal failures to AnalysisException and transient failures to SparkRuntimeException.
  • Keep unknown or absent marker values on the existing Python exception path.
  • Document the contract and cover direct readers, generator readers and generic Python UDF isolation.

Upstream context

  • lakehq/sail#2242 tracks the larger need for structured internal errors translated to Spark conditions. This pull request is intentionally narrower: application opt-in at the Python DataSource boundary rather than a general DataFusion or Arrow error taxonomy.
  • lakehq/sail#2267 is developing a first-party Python Kafka source. This protocol remains connector-agnostic and does not define Kafka retry policy.

Verification

  • cargo test
  • cargo clippy -p sail-common-datafusion -p sail-python-udf -p sail-data-source -p sail-spark-connect --all-targets --all-features -- -D warnings
  • cargo fmt --check
  • Wheel build with maturin
  • Focused installed-wheel Spark Connect tests: 2 passed
  • Existing unclassified DataSource exception check
  • Exact-head GitHub Actions run 30447886867: Rust build/tests, Python tests for four supported Spark versions and docs passed
  • Exact-head Rust and Python lint runs passed

Live Spark, Ibis and catalog integration jobs were not selected by the exact-head CI run.

Signed-off-by: Robin Everaars <robin.everaars@hoist-it.nl>
Signed-off-by: Robin Everaars <robin.everaars@hoist-it.nl>
@Robin-Everaars
Robin-Everaars merged commit 4c981cf into integration/dp-engine Jul 29, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant