Skip to content

Run and report withResource nesting lint with Jython [reduced-it] #15879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wjxiz1992 merged 12 commits into from
Sep 15, 2026
Merged

Run and report withResource nesting lint with Jython [reduced-it] #15879

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
cad0db9
Add advisory NVIDIA deprecation audit
gerashegalov Sep 1, 2026
4c28634
Fail optional audit when deprecations are found
gerashegalov Sep 1, 2026
910bd0d
Run deprecation checks with Jython
gerashegalov Sep 1, 2026
46edaea
Launch Jython from the plugin classpath
gerashegalov Sep 1, 2026
7781efe
Keep deprecation audit on root POM
gerashegalov Sep 1, 2026
e3b5787
Run withResource nesting lint with Jython
gerashegalov Sep 2, 2026
ffb3abc
Merge remote-tracking branch 'origin/main' into codex/with-resource-j…
gerashegalov Sep 2, 2026
f298d29
Publish withResource nesting audit report
gerashegalov Sep 2, 2026
761ca16
Address withResource audit review feedback
gerashegalov Sep 9, 2026
b1735b6
Fix withResource audit reporting
gerashegalov Sep 11, 2026
1498a8b
Merge origin/main into codex/with-resource-jython
gerashegalov Sep 14, 2026
27d02ec
Merge branch 'main' into codex/with-resource-jython
gerashegalov Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/mvn-verify-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,3 +430,23 @@ jobs:
fi
}
done

with-resource-nesting-audit:
name: withResource nesting audit
runs-on: ubuntu-latest
steps:
- uses: NVIDIA/spark-rapids-common/checkout@main

- name: Report withResource nesting violations
run: |
mvn --batch-mode -N antrun:run@with-resource-nesting-audit \
Comment thread
wjxiz1992 marked this conversation as resolved.
-DwithResource.audit.rawReport="$RUNNER_TEMP/with-resource-nesting-audit.json" \
-DwithResource.audit.summary="$GITHUB_STEP_SUMMARY"

- name: Upload withResource nesting report
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: with-resource-nesting-audit
path: ${{ runner.temp }}/with-resource-nesting-audit.json
if-no-files-found: ignore
51 changes: 33 additions & 18 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,8 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<withResource.audit.rawReport>${project.build.directory}/with-resource-nesting-audit.json</withResource.audit.rawReport>
<withResource.audit.summary>${project.build.directory}/with-resource-nesting-summary.md</withResource.audit.summary>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1993,6 +1995,37 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Run this repository-wide audit only from the canonical root reactor.
Encode option hyphens so the generated Scala 2.13 comment stays valid. -->
<!-- #if scala-2.12 -->
<execution>
<id>with-resource-nesting-audit</id>
<phase>verify</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_check_with_resource_nesting.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="&#45;&#45;root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${withResource.audit.rawReport}"/>
<arg value="&#45;&#45;summary"/>
<arg value="${withResource.audit.summary}"/>
</java>
</target>
</configuration>
</execution>
<!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2030,24 +2063,6 @@ This will force full Scala code rebuild in downstream modules.
<arg line="--xmlEncoding ${project.reporting.outputEncoding}"/>
<arg line="${scalastyle.dirs}"/>
</java>
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-m"/>
<arg value="unittest"/>
<arg value="discover"/>
<arg value="-s"/>
<arg value="scripts/tests"/>
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="--root"/>
<arg value="${spark.rapids.source.basedir}"/>
</exec>
</target>
</configuration>
</execution>
Expand Down
51 changes: 33 additions & 18 deletions scala2.13/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,8 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<withResource.audit.rawReport>${project.build.directory}/with-resource-nesting-audit.json</withResource.audit.rawReport>
<withResource.audit.summary>${project.build.directory}/with-resource-nesting-summary.md</withResource.audit.summary>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1993,6 +1995,37 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Run this repository-wide audit only from the canonical root reactor.
Encode option hyphens so the generated Scala 2.13 comment stays valid. -->
<!-- #if scala-2.12 --><!--
<execution>
<id>with-resource-nesting-audit</id>
<phase>verify</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_check_with_resource_nesting.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="&#45;&#45;root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${withResource.audit.rawReport}"/>
<arg value="&#45;&#45;summary"/>
<arg value="${withResource.audit.summary}"/>
</java>
</target>
</configuration>
</execution>
--><!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2030,24 +2063,6 @@ This will force full Scala code rebuild in downstream modules.
<arg line="--xmlEncoding ${project.reporting.outputEncoding}"/>
<arg line="${scalastyle.dirs}"/>
</java>
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-m"/>
<arg value="unittest"/>
<arg value="discover"/>
<arg value="-s"/>
<arg value="scripts/tests"/>
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="scripts/check_with_resource_nesting.py"/>
<arg value="--root"/>
<arg value="${spark.rapids.source.basedir}"/>
</exec>
</target>
</configuration>
</execution>
Expand Down
9 changes: 8 additions & 1 deletion scripts/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,11 @@ Only hoist values that own their data. Views returned by methods such as `bitCas
`getChildColumnView`, `replaceListChild`, and `splitAsViews` must not outlive their owning parent.
Copy or convert a view to an owning resource before closing its parent.

The check and its unit tests run with the all-modules Scalastyle execution during `mvn verify`.
The script remains directly runnable with Python 3, but is also compatible with Jython 2.7. Maven
uses its managed Jython dependency to run the check and unit tests during root `mvn verify`; the
generated Scala 2.13 reactor does not repeat this repository-wide check.

The pull-request audit publishes every deep scope in the job summary and a JSON artifact. Existing
baseline entries are reported as debt but do not fail the check. New violations, stale baseline
entries, invalid exemptions, or report-generation errors fail it. GitHub source annotations show
the first 50 entries, with the complete set retained in the summary and artifact.
Loading
Loading