[SkipRecovery] Restore randomized regex and decimal window coverage [reduced-it] [databricks]

[wjxiz1992]

JaCoCo production line coverage: +0 lines (measured: delta-lake, iceberg, shuffle-plugin, sql-plugin, udf-compiler; Scala 2.12, shim 350, anchor 2d438c020496, vs compatible nightly b34)

Fixes #9731. Refs #7429 (decimal-sorting test guard only).

Description

This test-only PR removes two stale restrictions after their underlying fixes landed: it restores randomized regex-replacement inputs and one nullable-decimal ROWS-window case on fixed Apache Spark 3.3 releases. Production behavior, SQL queries, and parity assertions are unchanged. The documented decimal RANGE limitation and existing Databricks/vendor guards remain in place.

• Keep the historical Unicode next-line (NEL) inputs from [BUG] Integration test test_re_replace_all fails with a corner case #9731 in the existing replacement test's generator, along with other line-terminator cases, while removing its fixed-seed override. The original failing seed and exact inputs now match CPU/GPU after Simplify Transpilation of $ with Extended Line Separator Support in cuDF Regex #11663 and Drop the regex line-anchor CRLF workaround now that cuDF #22763 landed #15023.
• Narrow the decimal-sorting guard for plain Apache Spark 3.3.1+ releases, following the SPARK-40089 backport. Spark 3.3.0 stays guarded. On Spark 3.3.4, this restores the existing nullable Decimal(23,10) negative-ROWS parameter; it does not remove the two Databricks decimal RANGE xfails associated with Window aggregations for decimal ranges: different output from CPU and GPU #7429. The helper's ranking-test caller already requires Spark 3.5+, so it gains no additional cases.

No new test function or parameter dimension is added. No CPU/GPU divergence was observed in the recovered cases.

AI assistance: The change and PR description were prepared with Codex assistance.

Validation

• Fresh Maven packaging for shims 350 and 334: BUILD SUCCESS for both, using isolated Maven caches.
• Complete regexp_test.py: 95 passed, 1 existing skip on each of Apache Spark 3.5.0 and 3.3.4.
• Complete window_function_test.py on Spark 3.3.4: 914 passed, 136 existing skips, 4 existing xfails; the recovered decimal case passed.
• Regex seeds 1700077791, 0, 42, and 20260911: all passed. Exact historical inputs returned the same PRODPROD<NEL>PROD on CPU/GPU, where <NEL> denotes U+0085, with GpuRegExpReplace captured. The unmasked decimal case compared all 2,048 rows with GPU window execution confirmed.
• Forced-OOM checks: 78 passed on Spark 3.5.0 across the two affected functions and the guard's ranking-test caller; 8 passed on Spark 3.3.4 with TEST_TYPE=pre-commit and REDUCED_IT=true. All eight recovered-function cases were retained. The regex test has no parametrization, and the window test's matrix is seven data generators × one batch size. Thus [reduced-it] retains every recovered combination.
• Version-guard boundary checks: 24 passed, including multi-digit patch versions and unchanged older, Databricks, and vendor-version behavior.

Environment and coverage details

Apache Spark 3.3.4 / 3.5.0, Python 3.10.18, Scala 2.12, JDK 17, UTC, RTX 5880 Ada. Databricks was not run locally; its existing guards are preserved and the title requests Databricks CI.

Both affected functions were replayed with the exact compatible nightly anchor JAR: 8 passed. All five regenerated module baseline counters matched the published report, with no class-ID mismatches and zero marginal production lines. This measures shim 350 against compatible b34, not the separately refreshed latest b21 report or a Spark 3.3 coverage total.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no blocking issues were identified.

Summary

• Retains historical NEL inputs and adds other line-terminator special cases while preserving reproducibility through the shared test seed.
• Uses the Apache runtime predicate to preserve existing Databricks and vendor guards.
• Leaves SQL queries, parity assertions, and decimal RANGE restrictions unchanged.
• No actionable new defects or repository-rule violations were identified. Integration tests were not rerun during this review.

_{Reviews (2) · Last reviewed commit: "Restrict decimal sorting guard recovery ..."}

[Copilot]

🟡 Changes recommended

Restrict the decimal guard to Apache runtimes so vendor Spark environments remain guarded.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This test-only PR restores randomized regex coverage and re-enables nullable decimal window coverage for supported Apache Spark versions.

Changes:

• Restores historical Unicode line-terminator inputs and removes the fixed regex seed.
• Updates decimal version guards while retaining existing vendor and Databricks guards.

File summaries

File	Summary
integration_tests/src/main/python/window_function_test.py	Updates decimal version guarding; the condition should be limited to plain Apache runtimes.
integration_tests/src/main/python/regexp_test.py	Restores randomized regex inputs and regression coverage.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wjxiz1992]

build