Refactor regex quantifiers into base and mode [reduced-it] [databricks] by wjxiz1992 · Pull Request #15899 · NVIDIA/cudf-spark

wjxiz1992 · 2026-09-04T05:26:39Z

JaCoCo production line coverage: +64 lines (sql-plugin +64; shim 330, fix-line measurement against base e313d91fd)

Closes #15831
Contributes to #14733
Follow-up to #15478 and the review suggestion.

Description

This refactor gives regex quantifiers one representation for repetition bounds and matching mode. Parser, transpiler, rewrite, serializer, and fuzz-generator code now use that shared representation, reducing duplicated handling while preserving supported public regex behavior, GPU fallback boundaries, and the 999 repetition limit.

The package-private unchecked parser also matches Java's error descriptions and positions for malformed counted closures and descending repetition ranges. A broader syntax-error classification sweep remains follow-up work; public parsing continues to validate through Pattern.compile first.

Keep greedy, reluctant, and possessive modes independent of repetition bounds.
Keep source positions outside case-class equality, preserving diagnostic locations.
Use qualified nested types in the auxiliary constructor so it does not require a file-wide Base/Mode import.
Exercise parsing for every base/mode combination and generate valid bounded quantifiers in the fuzzer.

AI assistance: AI tools assisted with the code and PR description.

Validation

Validated the source committed in e76806599 on Spark 3.3.0, Scala 2.12, Python 3.10.18, and an RTX 5880 Ada:

RegularExpressionParserSuite, RegularExpressionTranspilerSuite, RegularExpressionRewriteSuite, and RegExpUtilsSuite: 184 succeeded, 0 failed, 6 canceled, 0 ignored, 0 pending; Maven BUILD SUCCESS in 5:53. The six cancellations retain the existing Unicode line-separator exclusions tracked by [BUG] [Regexp] Line anchor '$' incorrect matching of unicode line terminators #7585. These suites include CPU/GPU comparisons and fuzz coverage.
Rebuilt dist,integration_tests: Maven BUILD SUCCESS in 1:24. Verified 21 parser, quantifier, transpiler, and rewrite class files match the distribution JAR.
Focused regexp_test.py selection: 7 passed, 32692 deselected, 14 warnings in 18.29 seconds. It covers negative-limit split, unsupported split fallback, repetition replacement, possessive fallback, extract/extract-all fallback, and lazy quantifiers with CPU/GPU result comparisons.
Repository-wide Scalastyle: 1850 files, 0 errors, 0 warnings, BUILD SUCCESS. Resource-nesting lint and its 20 unit tests also passed.
JaCoCo: 64 of 83 added JVM production lines covered, with 4073 sql-plugin classes analyzed and no class-ID mismatch. All added JVM production source is in sql-plugin; the diff adds no production lines in the API, Delta Lake, Iceberg, shuffle, or UDF modules. This is a measurement of the complete PR diff.

Maven commands used package, the worktree-local Maven repository, and GPU allocation fractions 0.3/0.3/0. The Python run used the rebuilt distribution and explicitly selected Python 3.10 for driver and workers. Local runtime validation covers shim 330; the remaining runtime matrix is delegated to CI.

Performance impact

Regex parsing remains O(pattern length) and runs during expression translation rather than per input row. This representation change adds no JNI calls or GPU work. The constructor qualification follow-up changes only name resolution: its rebuilt JVM instructions and method descriptors match the pre-edit artifact. No runtime benchmark was run for this cold-path refactor.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Closes NVIDIA#15831 Contributes to NVIDIA#14733 Validation: - Tests: succeeded 183, failed 0, canceled 6, ignored 0, pending 0 - 7 passed, 89 deselected, 2 warnings in 13.61s Signed-off-by: Allen Xu <allxu@nvidia.com>

greptile-apps · 2026-09-04T05:35:23Z

Confidence Score: 5/5

The PR appears safe to merge.

Summary

Introduces independent quantifier base and matching-mode dimensions.
Migrates parser and transpiler pattern matching to the unified representation.
Expands unit coverage for base/mode combinations, diagnostic positions, malformed counted closures, and fuzz generation.

Diagram

%%{init: {'theme': 'neutral'}}%%
classDiagram
  class RegexQuantifier {
    +Base base
    +Mode mode
    +Option~Int~ position
    +minLength() Int
    +toRegexString() String
  }
  class Base
  class Mode
  Base <|-- ZeroOrOne
  Base <|-- ZeroOrMore
  Base <|-- OneOrMore
  Base <|-- Fixed
  Base <|-- Variable
  Mode <|-- Greedy
  Mode <|-- Reluctant
  Mode <|-- Possessive
  RegexQuantifier --> Base
  RegexQuantifier --> Mode

_{Reviews (5) · Last reviewed commit: "Restore qualified RegexQuantifier constr..."}

Copilot

🔵 Needs a closer look

It is a broad refactor in a core parsing/transpilation path where subtle semantic or diagnostic-position regressions are possible despite unit coverage.

Pull request overview

Refactors the regex quantifier AST model to a single RegexQuantifier(base, mode) representation, separating repetition shape (base) from matching behavior (mode), while keeping diagnostics positions and the cuDF 999-count boundary behavior intact.

Changes:

Replaces the former quantifier subtype hierarchy with RegexQuantifier.Base + RegexQuantifier.Mode, constructed directly by the parser.
Updates cuDF transpiler quantifier pattern matches to use the new base/mode decomposition.
Updates Scala test suites and fuzz generation to build quantifiers via the new model and adds an independence/equality regression test.

File summaries

File	Description
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionTranspilerSuite.scala	Updates fuzz regex generation to construct `RegexQuantifier(base, mode)` rather than subtype + mode-copy helpers.
tests/src/test/scala/com/nvidia/spark/rapids/RegularExpressionParserSuite.scala	Updates expected ASTs to the new quantifier model and adds a test asserting base/mode independence plus equality ignoring diagnostic position.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala	Refactors parsing and transpilation logic to use `RegexQuantifier(base, mode)`; introduces `Base` ADT and ports quantifier-related matching/serialization accordingly.

Review details

Files reviewed: 3/3 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

wjxiz1992 · 2026-09-05T00:34:49Z

build

Apply the requested parser, AST, and test cleanups while preserving the production parse validation boundary. Performance: these changes run only while parsing and transpiling regex plans; they add no per-row or GPU-kernel work, and the consolidated matches reduce dispatch branches. Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-05T12:25:23Z

build

wjxiz1992 · 2026-09-05T13:41:25Z

build

wjxiz1992 · 2026-09-05T15:07:57Z

build

igorpeshansky

Would like to resolve the "closer to Java semantics" discussion eventually, but it doesn't have to be addressed in this PR. The rest are minor cleanups.

igorpeshansky · 2026-09-08T19:01:04Z

-            Some(QuantifierVariableLength(minLength, maxLength))
+            Some(Variable(minLength, maxLength))
          } else {
            None


FWIW, I think sticking closer to Java semantics helps maintain cleaner code and avoids questions about consistency… Since these edge cases are only reachable from tests, it would be easiest to match Java to decide on the expected behavior.

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-09T07:22:21Z

build

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-09T14:23:51Z

build

igorpeshansky

One bad suggestion to undo (#15899 (comment)), otherwise LGTM . Approving to unblock.

igorpeshansky · 2026-09-10T06:51:31Z


 import com.nvidia.spark.rapids.GpuOverrides.regexMetaChars
 import com.nvidia.spark.rapids.RegexParser.toReadableString
+import com.nvidia.spark.rapids.RegexQuantifier.{Base, Mode}


Apologies for the churn, see #15899 (comment).

Updated. (actually, it's not a bad suggestion at all, by this chance I updated my local review skill like "always check unnecessary call chain if caller is already/can be imported" and "try best to reuse existing objests/functions". It's what review rounds for :)

igorpeshansky · 2026-09-10T06:52:56Z

-          } else {
-            None
+          if (!peek().contains('}')) {
+            throw new PatternSyntaxException("Unclosed counted closure", pattern, pos)


Hmm, the parser didn't use to throw PatternSyntaxExceptions — only RegexUnsupportedExceptions (the former were thrown by Java's Pattern.compile). Many pattern errors (as opposed to unsupported patterns) also currently throw the latter. I like matching the Java ones, but it might make sense to do a sweep and convert the existing pattern errors into those as well… Let's open an issue to track?

Thanks, filed #15963 to track the syntax-error sweep.

igorpeshansky · 2026-09-10T19:51:46Z


    // The caller restores its position when this is a literal brace rather than a quantifier.
    consumeExpected('{')
    consumeInt.flatMap { minLength =>


[Really optional] Almost certainly a follow-up change, but just noting here that we still try to parse the quantifier and fall back on treating invalid strings as literals. At some point we might want to match Java's behavior, which commits to the quantifier given the { prefix and errors out for invalid ones (like a{, a{}, or a{x}).

Filed #15962 to track this follow-up.

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-10T23:42:42Z

build

wjxiz1992 · 2026-09-11T00:54:48Z

build

igorpeshansky

LGTM

fix(regex): unify quantifier base and mode (NVIDIA#15831)

0b26734

Closes NVIDIA#15831 Contributes to NVIDIA#14733 Validation: - Tests: succeeded 183, failed 0, canceled 6, ignored 0, pending 0 - 7 passed, 89 deselected, 2 warnings in 13.61s Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 added the task Work required that improves the product but is not user facing label Sep 4, 2026

wjxiz1992 changed the title ~~[WIP] Refactor regex quantifiers into base and mode [reduced-it] [databricks] [fast-ut]~~ Refactor regex quantifiers into base and mode [reduced-it] [databricks] [fast-ut] Sep 4, 2026

wjxiz1992 marked this pull request as ready for review September 4, 2026 05:33

wjxiz1992 requested review from igorpeshansky and a lite review from Copilot September 4, 2026 05:33

wjxiz1992 self-assigned this Sep 4, 2026

Copilot started reviewing on behalf of wjxiz1992 September 4, 2026 05:34 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

wjxiz1992 changed the title ~~Refactor regex quantifiers into base and mode [reduced-it] [databricks] [fast-ut]~~ Refactor regex quantifiers into base and mode [reduced-it] [fast-ut] [databricks] Sep 4, 2026

igorpeshansky reviewed Sep 5, 2026

View reviewed changes

wjxiz1992 changed the title ~~Refactor regex quantifiers into base and mode [reduced-it] [fast-ut] [databricks]~~ Refactor regex quantifiers into base and mode [reduced-it] [databricks] Sep 7, 2026

igorpeshansky reviewed Sep 9, 2026

View reviewed changes

Address regex parser review follow-ups

cac97d2

Signed-off-by: Allen Xu <allxu@nvidia.com>

Fix RegexQuantifier import ordering

dbff01e

Signed-off-by: Allen Xu <allxu@nvidia.com>

igorpeshansky previously approved these changes Sep 10, 2026

View reviewed changes

Restore qualified RegexQuantifier constructor types

e768065

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 dismissed igorpeshansky’s stale review via e768065 September 10, 2026 23:39

This was referenced Sep 11, 2026

Align RegexParser.parseUnchecked with Java for malformed brace quantifiers #15962

Open

Use PatternSyntaxException for syntax errors in RegexParser #15963

Open

igorpeshansky approved these changes Sep 11, 2026

View reviewed changes

wjxiz1992 merged commit 12641dc into NVIDIA:main Sep 14, 2026
59 checks passed

Uh oh!

Conversation

wjxiz1992 commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Performance impact

Checklists

Uh oh!

greptile-apps Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

wjxiz1992 commented Sep 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjxiz1992 commented Sep 5, 2026

Uh oh!

wjxiz1992 commented Sep 5, 2026

Uh oh!

wjxiz1992 commented Sep 5, 2026

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igorpeshansky Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wjxiz1992 commented Sep 9, 2026

Uh oh!

wjxiz1992 commented Sep 9, 2026

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

igorpeshansky Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

igorpeshansky Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

igorpeshansky Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Sep 10, 2026

Uh oh!

wjxiz1992 commented Sep 11, 2026

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Sep 4, 2026 •

edited

Loading

greptile-apps Bot commented Sep 4, 2026 •

edited

Loading