Lateral eval expressions supported after Spark upgrading (#544)

Signed-off-by: Lantao Jin <[email protected]> (cherry picked from commit c733d3c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Oct 2, 2024 · f75f048 · f75f048
1 parent daa33e5
commit f75f048
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 7 deletions.
diff --git a/...-test/src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLEvalITSuite.scala b/...-test/src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLEvalITSuite.scala
@@ -480,12 +480,7 @@ class FlintSparkPPLEvalITSuite
     comparePlans(logicalPlan, expectedPlan, checkAnalysis = false)
   }
 
-  // +--------------------------------+
-  // | Below tests are not supported  |
-  // +--------------------------------+
-  // Todo: Upgrading spark version to 3.4.0 and above could fix this test.
-  // https://issues.apache.org/jira/browse/SPARK-27561
-  ignore("test lateral eval expressions references - SPARK-27561 required") {
+  test("test lateral eval expressions references") {
     val frame = sql(s"""
          | source = $testTable | eval col1 = 1, col2 = col1 | fields name, age, col2
          | """.stripMargin)

diff --git a/ppl-spark-integration/README.md b/ppl-spark-integration/README.md
@@ -258,7 +258,7 @@ Assumptions: `a`, `b`, `c` are existing fields in `table`
  - `source = table | eval n = now() | eval t = unix_timestamp(a) | fields n,t`
  - `source = table | eval f = a | where f > 1 | sort f | fields a,b,c | head 5`
  - `source = table | eval f = a * 2 | eval h = f * 2 | fields a,f,h`
- - `source = table | eval f = a * 2, h = f * 2 | fields a,f,h` (Spark 3.4.0+ required)
+ - `source = table | eval f = a * 2, h = f * 2 | fields a,f,h`
  - `source = table | eval f = a * 2, h = b | stats avg(f) by h`
  - `source = table | eval f = ispresent(a)`