Align RegexParser.parseUnchecked with Java for malformed brace quantifiers

[wjxiz1992]

Describe the issue

RegexParser.parseUnchecked() accepts malformed counted-quantifier prefixes such as a{, a{}, and a{x} as literal-like AST content, while Java's Pattern.compile rejects all three. After consuming {, tryParseBraceQuantifier() uses consumeInt.flatMap; when no integer follows, it returns None, allowing the caller to restore its position and continue literal parsing.

Igor identified this as an optional follow-up in the PR review.

This is an internal parser-consistency follow-up to #15899 and contributes to #14733. The production RegexParser.parse() entry point already invokes Pattern.compile before parseUnchecked(). The reproduction confirmed that public parsing rejects the seven invalid inputs with the same Java exception description and index. This issue does not report a newly observed production CPU/GPU result mismatch.

Review snapshot: e76806599f06b95bb697ec54342c3245630e593c. The local dist JAR used for the reproduction records revision dbff01effe5439901f2ce7bf1b3ba206761e7e79; the source diff to the review snapshot only removes the top-level Base/Mode import and restores qualified constructor parameter types, leaving the parser behavior in this issue unchanged.

Steps/Code to reproduce

Run the following block in spark-shell with the PR's Spark 3.3 dist JAR loaded and the RAPIDS plugin enabled. Reflection only exposes the package-private unchecked entry point; it does not modify parser logic.

{
import com.nvidia.spark.rapids.ShimLoader
import java.lang.reflect.InvocationTargetException
import java.util.regex.{Pattern, PatternSyntaxException}

val parserClass = ShimLoader.getShimClassLoader()
  .loadClass("com.nvidia.spark.rapids.RegexParser")
val unchecked = parserClass.getDeclaredMethod("parseUnchecked")
unchecked.setAccessible(true)

def inspect(pattern: String, useUnchecked: Boolean): String = {
  def describe(e: Throwable): String = e match {
    case p: PatternSyntaxException =>
      s"PatternSyntaxException: ${p.getDescription}; index=${p.getIndex}"
    case other => s"${other.getClass.getSimpleName}: ${other.getMessage}"
  }
  try {
    if (useUnchecked) {
      val parser = parserClass.getConstructor(classOf[String]).newInstance(pattern)
      val ast = unchecked.invoke(parser)
      "ACCEPT " + ast.getClass.getMethod("toRegexString").invoke(ast)
    } else {
      Pattern.compile(pattern)
      "ACCEPT"
    }
  } catch {
    case e: InvocationTargetException => describe(e.getCause)
    case e: PatternSyntaxException => describe(e)
  }
}

Seq("a{", "a{}", "a{x}").foreach { p =>
  println(s"PATTERN=[$p] JAVA=[${inspect(p, false)}] UNCHECKED=[${inspect(p, true)}]")
}
}

Actual output from the executed reproduction:

RAPIDS Enabled = YES
Plugins Loaded = YES
SPARK_VERSION=3.3.0
JAVA_VERSION=11.0.31
PATTERN=[a{] JAVA=[PatternSyntaxException: Illegal repetition; index=0] UNCHECKED=[ACCEPT a{]
PATTERN=[a{}] JAVA=[PatternSyntaxException: Illegal repetition; index=0] UNCHECKED=[ACCEPT a{\}]
PATTERN=[a{x}] JAVA=[PatternSyntaxException: Illegal repetition; index=0] UNCHECKED=[ACCEPT a{x\}]
PUBLIC_PARSE_MATCHES_JAVA=7/7

The separate public-entry control compared Pattern.compile with RegexParser.parse for a{, a{}, a{x}, (, [a, a trailing backslash, and \x1: all seven matched in exception description and index.

Expected behavior and acceptance criteria

• In quantifier position, commit to parsing a counted quantifier when an unescaped { is encountered; reject a malformed prefix instead of returning to literal parsing.
• Match the running JDK's Pattern.compile exception type, description, and index for a{, a{}, and a{x}. Avoid hard-coding diagnostics from one JDK as universal expectations.
• Extend existing parser tests with these unchecked-parser cases, plus valid counted quantifiers and literal/escaped brace controls, including character classes and quoted text.
• Preserve public Java validation, valid AST serialization, quantifier modes, diagnostic positions for valid input, and GPU unsupported-pattern fallback.
• Keep the broader conversion of already-detected syntax errors to Java-compatible exceptions in the separate exception-classification follow-up.

Relevant source: brace quantifier parsing.

Environment

• Standalone Spark 3.3.0 / Scala 2.12, OpenJDK 11.0.31, Linux, NVIDIA RTX 5880 Ada.
• RAPIDS 26.10.0-SNAPSHOT, Spark 330 dist JAR; provenance above.
• The reproduction used the project's full Spark-shell configuration template with spark.plugins=com.nvidia.spark.SQLPlugin, spark.rapids.sql.enabled=true, spark.rapids.sql.regexp.enabled=true, GPU allocation fractions 0.3/0.3/0, and local[2].
• The configuration-extraction script was executed first, wrapped in a Scala block for REPL loading. Both enabled/plugin checks succeeded. The reproduction completed with shell exit code 0.

Additional context

• [BUG] Parse regular expressions using JDK to make error behavior more consistent between CPU and GPU #11651 addressed the production Java-validation boundary; this follow-up concerns the unchecked parser behind that boundary.
• AI assistance: the reproduction and issue text were prepared with Codex assistance.

Related parser exception-classification follow-up: #15963