Lateral eval expressions supported after Spark upgrading (opensearch-…

…project#544)

Signed-off-by: Lantao Jin <[email protected]>

integ-test/src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLEvalITSuite.scala

@@ -480,12 +480,7 @@ class FlintSparkPPLEvalITSuite
     comparePlans(logicalPlan, expectedPlan, checkAnalysis = false)
   }
 
-  // +--------------------------------+
-  // | Below tests are not supported  |
-  // +--------------------------------+
-  // Todo: Upgrading spark version to 3.4.0 and above could fix this test.
-  // https://issues.apache.org/jira/browse/SPARK-27561
-  ignore("test lateral eval expressions references - SPARK-27561 required") {
+  test("test lateral eval expressions references") {
     val frame = sql(s"""
          | source = $testTable | eval col1 = 1, col2 = col1 | fields name, age, col2
          | """.stripMargin)

ppl-spark-integration/README.md

@@ -249,7 +249,7 @@ Assumptions: `a`, `b`, `c` are existing fields in `table`
  - `source = table | eval n = now() | eval t = unix_timestamp(a) | fields n,t`
  - `source = table | eval f = a | where f > 1 | sort f | fields a,b,c | head 5`
  - `source = table | eval f = a * 2 | eval h = f * 2 | fields a,f,h`
- - `source = table | eval f = a * 2, h = f * 2 | fields a,f,h` (Spark 3.4.0+ required)
+ - `source = table | eval f = a * 2, h = f * 2 | fields a,f,h`
  - `source = table | eval f = a * 2, h = b | stats avg(f) by h`
 
 Limitation: Overriding existing field is unsupported, following queries throw exceptions with "Reference 'a' is ambiguous"