Access Javac API from a standalone javac jar instead of the default JDK runtime #924

testforstephen · 2024-11-05T08:57:06Z

This PR tries to introduce a standalone Javac jar from the repository https://github.com/JaroslavTulach/nb-javac for Javac API access within our project. The benefit is that it allows the language server to support latest language features without requiring user to run the language server with the latest JDK.

This PR is an experiment PR, not full ready yet. It faces some issues with new approach, and I open the PR first for suggestions and feedback.

The project can compile fine with the embedded nb-javac, but get the module conflict issue during runtime. It reports module conflicts between jdk.compiler module and unnamed module from the embedded nb-javac jar. When running it in Eclipse, it throws exception below.

java.lang.ClassCastException: class com.sun.tools.javac.api.JavacTool cannot be cast to class com.sun.tools.javac.api.JavacTool (com.sun.tools.javac.api.JavacTool is in module jdk.compiler of loader 'app'; com.sun.tools.javac.api.JavacTool is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @d014059)
   at org.eclipse.jdt.core.dom.JavacCompilationUnitResolver.parse(JavacCompilationUnitResolver.java:690)
   at org.eclipse.jdt.core.dom.JavacCompilationUnitResolver.toCompilationUnit(JavacCompilationUnitResolver.java:541)
   at org.eclipse.jdt.core.dom.ASTParser.internalCreateASTCached(ASTParser.java:1299)
   at org.eclipse.jdt.core.dom.ASTParser.lambda$1(ASTParser.java:1178)
   at org.eclipse.jdt.internal.core.JavaModelManager.cacheZipFiles(JavaModelManager.java:5692)
   at org.eclipse.jdt.core.dom.ASTParser.internalCreateAST(ASTParser.java:1178)
   at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:918)
   at org.eclipse.jdt.internal.core.CompilationUnit.buildStructure(CompilationUnit.java:184)
   at org.eclipse.jdt.internal.core.Openable.generateInfos(Openable.java:246)
   at org.eclipse.jdt.internal.core.JavaElement.openWhenClosed(JavaElement.java:569)
   at org.eclipse.jdt.internal.core.JavaElement.getElementInfo(JavaElement.java:292)
   at org.eclipse.jdt.internal.core.JavaElement.getElementInfo(JavaElement.java:278)
   at org.eclipse.jdt.internal.core.CompilationUnit.getSourceRange(CompilationUnit.java:1087)

The packaged nb-javac only includes the classes of jdk.compiler, no jdk.javadoc classes. To bypass the compilation error, I temporarily comment out the jdk.javadoc classes reference from JavadocConverter.java.
License review. nb-javac is under GPL-2.0 license, require legal review.

mickaelistria · 2024-11-05T09:02:20Z

Does the launch configuration or the java command still set the --add-exports ... flag? If so, please make sure they're removed from most places. The META-INF/p2.inf file could probably be removed too.

testforstephen · 2024-11-05T13:01:21Z

I removed the --add-exports ... flags from the launch configuration. However, Eclipse automatically adds --add-modules=ALL_SYSTEM to the launch command, which includes the java.compiler module in the runtime by default. This causes the call to ToolProvider.getSystemJavaCompiler() to use the default java.compiler api and return the default JDK compiler instance and its associated jdk.compiler module. Since nb-javac already contains all the classes from both java.compiler and jdk.compiler, we need to find a way to ensure that the nb-javac JAR is prioritized over the JDK module.

eclipse-jdt-core-incubator/org.eclipse.jdt.core.javac/src/org/eclipse/jdt/core/dom/JavacCompilationUnitResolver.java

Line 565 in d8b4f12

var compiler = ToolProvider.getSystemJavaCompiler();

rgrunber · 2024-11-07T16:01:57Z

Found this on ClearlyDefined with a score that makes it usable, ~~https://clearlydefined.io/definitions/sourcearchive/mavencentral/com.dukescript.nbjavac/nb-javac/jdk-19+33~~ (not useable, License score is what counts, and it's 25, must be at least 60) . I have filed https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/17140 for the particular version here.

Update: It is Approved ✔️

testforstephen · 2024-11-08T10:34:04Z

In recent commits, I addressed two issues:

The first issue is java.lang.ClassCastException: class com.sun.tools.javac.api.JavacTool cannot be cast to class com.sun.tools.javac.api.JavacTool (com.sun.tools.javac.api.JavacTool is in module jdk.compiler of loader 'app'; com.sun.tools.javac.api.JavacTool is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @d014059). When launching Eclipse plugins with JDK 9+, most java.* modules (including java.compiler) are loaded as root modules by default, which is not configurable. To workaround the module conflicts, I use com.sun.tools.javac.api.JavacTool.create() to obtain a compiler instance, rather than javax.tools.ToolProvider.getSystemJavaCompiler().
The second issue is that nbjavac.VMWrapper fails to resolve ct.sym from the OSGi classloader. The nb-javac library includes wrappers to load JDK resources, relying on the classloader mechanism for this purpose. However, its implementation assumes that NetBeans loads the nbjavac JARs as standard jar dependencies on the classpath. In the Eclipse environment, where the javac JAR is embedded within a bundle, the classloader behaves differently, causing the nbjavac implementation to fail in an OSGi bundle context. To address this, I implemented a workaround to help VMWrapper load ct.sym from the OSGi bundle resources.

In the latest commit, the DOM resolver already works with the embedded Javac. But for the build support, it still has some problems because the apt plugin is not loaded well. Here are some potential issues to look further.

The one unresolved issue is that org.eclipse.jdt.apt.pluggable.core plugin cannot be compiled well because the module namespace conflicts between the java.compiler and unnamed module for the embedded Javac jar. This occurs because the javac plugin is implemented as a fragment host of the org.eclipse.jdt.core plugin, causing any plugin that depends on jdt.core to automatically include the javac plugin in its dependencies. As a result, the transitive unnamed nbjavac module in Javac plugin conflicts with the java.compiler dependency of the apt plugin. To resolve this, we may need to refactor the alternative compiler API support in AbstractImageBuilder into an extension point. This can make the javac plugin into a standalone plugin, and isolate it from conflicts with other plugins.

Another potential issue relates to Lombok compatibility. Similar to ECJ, Lombok support can encounter compatibility issues when used with Javac. Just as a new ECJ version might disrupt Lombok functionality, a new Javac release can also break compatibility with Lombok. In jdt.ls, we typically embed the latest lombok.jar and utilize it alongside ECJ. However, with Javac, it defaults to using a project's own Lombok dependency, loading it as a standard annotation processor in Javac. I recently observed that lombok 1.18.34 is not compatible with javac 23 because of new Javadoc syntax. As a result, applications still using Lombok 1.18.34 will encounter resolution errors with Javac 23. In future, we might consider using the embedded Lombok version in jdt.ls for the Javac parsing as well.

testforstephen · 2024-11-12T07:14:43Z

The one unresolved issue is that org.eclipse.jdt.apt.pluggable.core plugin cannot be compiled well because the module namespace conflicts between the java.compiler and unnamed module for the embedded Javac jar. This occurs because the javac plugin is implemented as a fragment host of the org.eclipse.jdt.core plugin, causing any plugin that depends on jdt.core to automatically include the javac plugin in its dependencies. As a result, the transitive unnamed nbjavac module in Javac plugin conflicts with the java.compiler dependency of the apt plugin. To resolve this, we may need to refactor the alternative compiler API support in AbstractImageBuilder into an extension point. This can make the javac plugin into a standalone plugin, and isolate it from conflicts with other plugins.

I attempted to convert the org.eclipse.jdt.core.javac bundle from a fragment to a plugin and used an extension point to register an alternative Javac compiler with AbstractImageBuilder. The image builder successfully locates and utilizes JavacCompiler. However, JavacCompilationUnitResolver fails because it depends on several package-private classes (e.g., org.eclipse.jdt.core.dom.BindingResolver) from the org.eclipse.jdt.core plugin.

java.lang.IllegalAccessError: class org.eclipse.jdt.core.dom.JavacBindingResolver cannot access its superclass 
org.eclipse.jdt.core.dom.BindingResolver (org.eclipse.jdt.core.dom.JavacBindingResolver is in unnamed module of loader 
org.eclipse.osgi.internal.loader.EquinoxClassLoader @5f618123; org.eclipse.jdt.core.dom.BindingResolver is in unnamed module 
of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @90ba3b1)

I guess this dependency issue is why the javac plugin is set up as a fragment of jdt.core, allowing it to share the same classloader.

mickaelistria · 2024-11-12T07:43:27Z

org.eclipse.jdt.core.javac/META-INF/MANIFEST.MF

@@ -5,7 +5,10 @@ Bundle-SymbolicName: org.eclipse.jdt.core.javac;singleton:=true
 Bundle-Version: 1.0.0.qualifier
 Fragment-Host: org.eclipse.jdt.core
 Automatic-Module-Name: org.eclipse.jdt.core.javac
-Require-Capability: osgi.ee; filter:="(&(osgi.ee=JavaSE)(version=23))"
+Require-Capability: osgi.ee; filter:="(&(osgi.ee=JavaSE)(version=17))"


See https://ci.eclipse.org/ls/job/jdt-core-incubator/job/PR-924/3/console , this breaks current compilation since some Java 21 features are already adopted in the code. Would 21 be fine?

yes, it breaks Java 21 usage (e.g. some utility of String and pattern matching). We need to rewrite them with Java 17 compatible way. The purpose here is to make the minimum JDK requirement same as other JDT plugins.

IMO, at this stage, we should minimize the changes in the code and stick to Java 21 to facilitate progress. When we've adopted nb-javac, then we'll be more easily able to move code back to Java 17. Moreover, m2e already requires Java 21, so JDT-LS already requires Java 21, so trying to go below that won't produce any benefit for JDT-LS nor vscode-java.

I see redhat.java extension still requires a minimum Java 17 because it stays at [email protected]. Good to know we're going to upgrade to Java 21.

- This also fixes the quickfix as a result eg. ```java Function<Integer, Integer> func = x -> { System.out.println(x); }; ``` becomes ```java Function<Integer, Integer> func = x -> { System.out.println(x); return x; }; ``` Signed-off-by: David Thompson <[email protected]>

Different versions of Java allow different modifiers on interface methods, so ECJ generates slightly different problem ids for each of these cases. Use the compiler settings to determine which to use. This affects the logic of the quick fixes, so it should fix some jdt-ls test cases. Signed-off-by: David Thompson <[email protected]>

Signed-off-by: Rob Stryker <[email protected]>

hashcode/equals

Signed-off-by: Rob Stryker <[email protected]>

Signed-off-by: Rob Stryker <[email protected]> Fix regression Signed-off-by: Rob Stryker <[email protected]> Fix regressions Signed-off-by: Rob Stryker <[email protected]>

Signed-off-by: Rob Stryker <[email protected]>

…dt#705)

…compilation unit Signed-off-by: Rob Stryker <[email protected]>

Signed-off-by: Rob Stryker <[email protected]>

testforstephen · 2024-11-27T07:13:34Z

However, one remaining issue with the embedded javac is that it does not include the jdk.javadoc.* packages. To render snippets for Java 23 Markdown Javadoc, we rely on classes from these packages, such as jdk.javadoc.internal.doclets.formats.html.taglets.snippet. Since these classes were introduced in JDK 23 and are missing in the embedded javac JAR, I have commented out this functionality in the PR to ensure the embedded javac passes compilation.

For snippet javadoc issue in the PR #812, this is just a formatting thing. I'm not too worried about its support when we switch to nb-javac. We can implement a simple algorithm to handle it ourselves or reuse the solution of ECJ such as eclipse-jdt#60.

mickaelistria · 2024-12-11T13:44:21Z

For the license, here is the hint from EDP handbook: https://www.eclipse.org/projects/handbook/#ip-sbom . We should probably add an about.html mentioning the 3rd party GPLv2+ClassPathExtension license too.

rgrunber · 2024-12-11T14:10:31Z

See https://github.com/eclipse-orbit/orbit/blob/main/glassfish/hk2/org.glassfish.hk2.api_2.6.1/src/main/resources/about.html , https://github.com/eclipse-orbit/orbit/blob/main/javax/javax.ws.rs_2.1.6/src/main/resources/about.html for some examples of this.

testforstephen · 2025-01-05T11:09:56Z

Split javac jar reference into a separate PR #1082, and add about.html for license explanation.

github-actions bot assigned testforstephen Nov 5, 2024

testforstephen requested review from mickaelistria, fbricon and rgrunber November 5, 2024 08:57

mickaelistria reviewed Nov 12, 2024

View reviewed changes

datho7561 and others added 20 commits November 12, 2024 11:19

Improve JavacBindingResolver.resolveConstantExpression

edddb8b

Fix test0037 - getKey on JavacTypeVariableBinding

079b0c4

Signed-off-by: Rob Stryker <[email protected]>

Avoid requesting getKey early: store bindings identity with

cf96a0e

hashcode/equals

Fix test0150 - malformed package declaration

6235fd8

Signed-off-by: Rob Stryker <[email protected]>

Fix testPatternInstanceOfExpression003

d299a6a

Signed-off-by: Rob Stryker <[email protected]>

Fix issue with isGeneric according to spec

e436143

Signed-off-by: Rob Stryker <[email protected]>

Fix ASTConverter_RecordPattern_Test.testCaseDefaultExpressionPattern

0d0c411

Fix some generic/parameterized in JavacTypeBinding

ec83fe0

Attempt to fix package bindings

843b8c8

Signed-off-by: Rob Stryker <[email protected]> Fix regression Signed-off-by: Rob Stryker <[email protected]> Fix regressions Signed-off-by: Rob Stryker <[email protected]>

Fixes testBug497719_0001 - try with resources dom error

1468cd8

Signed-off-by: Rob Stryker <[email protected]>

Fix findDeclaringNode with binding key

f5846b8

[Javadoc] error handling

30a351c

Signed-off-by: Rob Stryker <[email protected]>

More problems to ignore

59a1ba2

Signed-off-by: Rob Stryker <[email protected]>

Fix equality for generic bindings always hosting generic type

0f0602b

Handle the nested javadoc syntax '@see {@link ...}' better (eclipse-j…

cd81c64

…dt#705)

Partial fix to testBug432175 - keys for types not under main type of …

6006c0a

…compilation unit Signed-off-by: Rob Stryker <[email protected]>

Fix testRecord012 - synthetic flag

c5890e0

Signed-off-by: Rob Stryker <[email protected]>

Fix testRecordPattern003

a8d34c9

Signed-off-by: Rob Stryker <[email protected]>

Decouple javac libraries into a separate bundle for better isolation

128eee9

mickaelistria force-pushed the dom-with-javac branch 2 times, most recently from 7f5c20a to 87832d6 Compare December 6, 2024 13:52

datho7561 force-pushed the dom-with-javac branch from aa3b08f to cc0dccd Compare December 13, 2024 18:57

mickaelistria force-pushed the dom-with-javac branch from 7e220b3 to 525cd23 Compare December 15, 2024 09:45

testforstephen mentioned this pull request Jan 5, 2025

Integrate javac library with a standalone bundle #1082

Merged

mickaelistria force-pushed the dom-with-javac branch 3 times, most recently from 630ab89 to 141942c Compare January 10, 2025 09:11

robstryker force-pushed the dom-with-javac branch 3 times, most recently from 9c17bb6 to 4516c31 Compare January 22, 2025 21:22

datho7561 force-pushed the dom-with-javac branch 2 times, most recently from a1bf22f to 0b6fd1c Compare January 27, 2025 15:37

mickaelistria force-pushed the dom-with-javac branch from 7ad387c to c52dfb2 Compare January 29, 2025 13:39

datho7561 force-pushed the dom-with-javac branch from c52dfb2 to 76e9c9c Compare January 29, 2025 16:38

robstryker force-pushed the dom-with-javac branch 2 times, most recently from 838c386 to 40594bf Compare January 31, 2025 18:15

datho7561 force-pushed the dom-with-javac branch from 15d0bf3 to 811bae7 Compare February 3, 2025 16:06

mickaelistria force-pushed the dom-with-javac branch 2 times, most recently from c906abc to b53a413 Compare February 7, 2025 10:49

robstryker force-pushed the dom-with-javac branch 2 times, most recently from 08ec5cf to 10fe71a Compare February 7, 2025 20:43

mickaelistria force-pushed the dom-with-javac branch 2 times, most recently from c2b9da2 to c640bdc Compare February 10, 2025 10:41

robstryker force-pushed the dom-with-javac branch from 1bb87f3 to 197281f Compare February 13, 2025 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Access Javac API from a standalone javac jar instead of the default JDK runtime #924

Access Javac API from a standalone javac jar instead of the default JDK runtime #924

testforstephen commented Nov 5, 2024 •

edited

Loading

mickaelistria commented Nov 5, 2024

testforstephen commented Nov 5, 2024

rgrunber commented Nov 7, 2024 •

edited

Loading

testforstephen commented Nov 8, 2024

testforstephen commented Nov 12, 2024 •

edited

Loading

mickaelistria Nov 12, 2024

testforstephen Nov 12, 2024

mickaelistria Nov 12, 2024

testforstephen Nov 12, 2024

testforstephen commented Nov 27, 2024

mickaelistria commented Dec 11, 2024

rgrunber commented Dec 11, 2024

testforstephen commented Jan 5, 2025

Access Javac API from a standalone javac jar instead of the default JDK runtime #924

Are you sure you want to change the base?

Access Javac API from a standalone javac jar instead of the default JDK runtime #924

Conversation

testforstephen commented Nov 5, 2024 • edited Loading

mickaelistria commented Nov 5, 2024

testforstephen commented Nov 5, 2024

rgrunber commented Nov 7, 2024 • edited Loading

testforstephen commented Nov 8, 2024

testforstephen commented Nov 12, 2024 • edited Loading

mickaelistria Nov 12, 2024

Choose a reason for hiding this comment

testforstephen Nov 12, 2024

Choose a reason for hiding this comment

mickaelistria Nov 12, 2024

Choose a reason for hiding this comment

testforstephen Nov 12, 2024

Choose a reason for hiding this comment

testforstephen commented Nov 27, 2024

mickaelistria commented Dec 11, 2024

rgrunber commented Dec 11, 2024

testforstephen commented Jan 5, 2025

testforstephen commented Nov 5, 2024 •

edited

Loading

rgrunber commented Nov 7, 2024 •

edited

Loading

testforstephen commented Nov 12, 2024 •

edited

Loading