"implementation" dependencies end up in pom as scope=runtime

[niloc132]

Presently the build file specifies

implementation("com.google.elemental2:elemental2-dom:1.0.0-RC1")
implementation("com.google.elemental2:elemental2-core:1.0.0-RC1")
implementation("com.google.jsinterop:base:1.0.0-RC1")

which results in a pom file where each of these three are set to have scope=runtime. Normally this isn't an issue when you assume that all attached artifacts are themselves already compiled, but in order to build each dependency recursively in j2cl with its own compile-time classpath, this results in j2cl running on the sources of this project and no compile time dependencies at all. Since the sources files of this project do reference various elemental2.dom.* and jsinterop.base.* classes, it ends up failing.

I think that these dependencies should be updated to use compile() instead, though my gradle knowledge is weak enough that I'm not clear on how building gwt-http is successful to begin with without those items on the compile-time classpath. Assuming then that implementation() is correct, is it possible to convince gradle to emit a more accurate pom.xml that indicates that these three jars are in fact compile time dependencies, and not just provided at runtime?

[tbroyer]

I must say I'm actually torn on this.

When using J2Cl, those dependencies are only required for the final Closure-compiler step actually (and of course, J2ClTranslat⋅ing the dependencies themselves), but they are of no use for JavaC or J2ClTranslate itself (J2ClTranslat⋅ing your code that uses gwt-http only needs the gwt-http JAR, as J2ClTranslate, like JavaC, only needs the APIs of your dependencies). If, say, you somehow change elemental2-core to a different version, given that your code only uses gwt-http APIs, and only gwt-http code calls the elemental2-core APIs, you shouldn't have to rerun JavaC or J2ClTranslate on your code: the gwt-http hasn't change, your code hasn't changed, so the output will be the same. But of course, for the "whole world" Closure Compiler step, you'll need those dependencies.

When using GWT2, those dependencies are only required at gwt:compile step; maybe my gwt-maven-plugin should change its defaults of only using compile dependencies for that by default, and use runtime ones instead.

Let's compare that to Proguard for example: if you build an uber-JAR ("whole world") and you want to use Proguard to prune all unused code (I'm not talking about obfuscation here, only dead-code pruning), you'll want to run Proguard on the set of runtime dependencies, not compile-time ones.

So while I understand that this "breaks" with gwt-maven-plugin by default, as it only uses compile-time dependencies, I think it might actually be the right choice going forward. And if it breaks your current J2Cl tooling, then it's still time to change its behavior.

WDYT?

[niloc132]

If, say, you somehow change elemental2-core to a different version, given that your code only uses gwt-http APIs, and only gwt-http code calls the elemental2-core APIs, you shouldn't have to rerun JavaC or J2ClTranslate on your code: the gwt-http hasn't change, your code hasn't changed, so the output will be the same. But of course, for the "whole world" Closure Compiler step, you'll need those dependencies.

Picking nits here, but this isn't correct if gwt-http happened to use a jsinterop method which changed in some way in that new version - adding or removing @JsOverlay for example would cause the generated JS to have to call a different method. But this would fail anyway, in the same way that managing a dependency to incompatible bytecode would fail.

If the j2cl plugin ecosystem assumed that it must compile every dependency from scratch for its own use, then I'm in agreement, it only would care about runtime deps, and could get away with j2cl-ing every dependency on the same giant classpath. Incrementally building across modules though this breaks down - feeding the same classpath to every javac invocation for example means that annotation processors behave somewhat like GWT's whole-world generators, which is likely to be a surprise if you deliberately modularized your project. Alternatively: if we could assume that j2cl output was future-compatible, we could distribute bytecode, stripped-bytecode, and jszip in maven central, but Google has explicitly told us not to do this.

Instead, the current iteration of j2cl-maven-plugin behaves like this:

1. If not yet generated (i.e. for reactor projects), run javac once to get generated sources
2. strip out gwt-incompatible members of the current project's sources (including generated)
3. javac those resulting stripped sources, against that project's classpath of stripped elements (this is where gwt-http fails)
   4)j2cl those stripped sources, against that project's classpath of stripped elements (also fails here, since technically this can run in parallel with 3)

These steps work regardless of the project you are running them on (plus or minus j2cl version, some compiler args), meaning no surprises about a project behaving one way in one project and another way in another project. Likewise, this output can be cached (for now just locally, we can play with sharing those pre-compiled outputs later) for faster subsequent runs.

All of those (unless you are building tests, but doesn't apply right now) run with scope=compile. From the "root" project you are building though, we're only interested in scope=runtime dependencies - starting from that list we run the above steps to produce the required directories/zips of JS files. Then on that scope=runtime "js classpath", we run closure. This is the "proguard" equivalent of your response I think - linking output before distribution.

If the "compile time" work ran using runtime dependencies, then we would miss out on explicitly scope=provided dependencies, such as things we compile against in JRE emulation and we will have specific scope=runtime JS provided for, or annotation processors which will never show up at runtime (this is true either in gradle with real scopes, or maven's "limited" set of scopes).

I could maybe see that it is expected instead to run javac/j2cl with scope=compile+runtime, though I'm having a hard time convincing myself of that, given that maven doesn't do that (and I had assumed until now that gradle didn't either)? But isn't the dependency from gwt-http => elemental2-dom both a compile-time and run-time dependency? If a "runtime" dependencies is needed at compile-time, what does "runtime" even mean?

Maven's scope=compile implies the behavior I would expect you're seeking here, but it appears that gradle's implementation does not (looking at https://stackoverflow.com/questions/44493378/whats-the-difference-between-implementation-and-compile-in-gradle#44493379 I want to suggest that api is the answer here).

[niloc132]

Okay, one coffee and shower later, I get it - gradle is trying to help by splitting "compile" into "do I need this to compile my sources" and "do you need this to compile with me on your classpath". Unfortunately, I'm not seeing where the former is described in either the pom or any other metadata in the publishToMaven* tasks, so it seems gradle is making the assumption "I have done all compilation that could ever be required for this release on my own sources.

This assumption makes sense for bytecode and probably even for other cases like android etc, but gets trickier with the J2CL world. We've been told that we can rely on the JavaPreprocessor not changing what it does, but also that J2CL itself may change, so we shouldn't distribute jszips (at least not without a mechanism to have the author re-distribute each time a new release is made...).

It could make sense for gradle to ship preprocessed bytecode automatically, but I'm not sure it will make sense to ship transpiled JS. Without those outputs already made based on the "do I need this to compile my sources" classpath, a downstream project needs to be able to determine that classpath in some way.

[tbroyer]

Here are the constraints:

• to JavaC one source file, you need its API dependencies; e.g. if your source file makes use of org.gwtproject.http.client.URL.decode(), you only need gwt-http, not jsinterop-base or elemental2. If you want to "strip"/recompile, you only need to do that on gwt-http, as doing it on jsinterop-base and elemental2 won't change anything
• to J2Cl that same source file, you have the exact same needs as JavaC (and changing the version of elemental2 won't change the generated JS for your source file)
• to Closure Compile your app, you need all its API and implementation dependencies; so now you need to "strip"/J2Cl gwt-http, jsinterop-base, and elemental2, and then give everything to the Closure Compiler. This is actually where you need to "strip"/recompile jsinterop-base and elemental2, such that you can J2Cl gwt-http against the correct API.
• note that the "strip"/recompile could actually be done at the bytecode level rather than source-level (you'd still have to "strip" at the source level later so you can J2Cl the stripped sources for the Closure Compiler step; so it's either 2 ways of stripping out gwt-incompatible code, or one way of doing it but having to JavaC the stripped sources)

Fwiw, your understanding and usage of the Maven scopes is influenced by the fact that Maven uses the POM both as the definition for how to build the project and how to use it: https://blog.ltgt.net/maven-is-broken-by-design/#poms-have-two-uses. What I need to compile gwt-http is not the same you need to compile against gwt-http: nothing in jsinterop-base or elemental2 is exposed in any API of gwt-http, they are purely "implementation details", so if they're not in your classpath when you JavaC your source files that use gwt-http, JavaC won't complain. Hence the api and implementation configurations, and the way to model that in a POM, used for consumption, is to put the implementation dependencies in the runtime scope.

Fwiw, your note about annotation processors and the provided scope is also influenced by Maven: https://blog.ltgt.net/maven-is-broken-by-design-2/. Annotation processors should go in JavaC's processor path, you shouldn't rely on the fallback to the compile classpath (unless you really mean it). What this means is that provided (and compileOnly in Gradle) should almost never be used: only compelling use cases are a) for annotations with @Retention(SOURCE), only used, and usable, by annotation processors when building that project, and b) in projects that create "bundles" that won't ever be depended on (such as WARs), to easily exclude dependencies that will be… provided (sic!) by the environment at runtime (e.g. the servlets API) but are needed at compile-time. You'll note that Gradle has a specialized provided for that, because you'll probably need those "compile only" dependencies when compiling and running your tests. Every other usecase should be dealt with downstream if needed (you don't want to include the JsInterop annotations in your WAR or Android app? exclude them)

Back to our constraints above: you'll want to use the compileClasspathElements (after stripping out gwt-incompatible APIs) when running JavaC or J2Cl on the project sources, and the runtimeClasspathElements (after stripping and J2CL-ing) when running the Closure Compiler. Apply that rule recursively up the chain when J2CL-ing the project's dependencies.

[niloc132]

Agreed that maven is flawed, but in the case of "we need to support the use case of a project actually compiling its dependencies", it actually is somewhat closer to accurate for our needs. If you ignore the lack of test-provided. And test-runtime. And anything to distinguish annotation processors at all. And...

Okay, either Gradle-based projects that are j2cl-compatible do not provide the "how do you compile me as if I was on your machine" details and so must ship already-j2cl'd-output, or those details must be provided. I'm not sure I'm following with compileClasspathElements (edit - perhaps this is MavenProject.getCompileClasspathElements()? that is only scope=compile, provided, and system, so the scope=runtime that gwt-http has on elemental2-dom won't be available) - is that metadata that is being deployed (in a way that not-gradle can read it) somehow along side jars, sourcejars etc? If the former, then gradle projects need to embed the j2cl version (and flags passed) so that later j2cl updates can be published too, if the latter then I am not understanding you.

to J2Cl that same source file, you have the exact same needs as JavaC (and changing the version of elemental2 won't change the generated JS for your source file)

You actually must compile against the bytecode-from-stripped-sources, it isn't enough to share the same bytecode as plain javac. I think you clarified that later, but it is an important distinction. I am not 100% sure that you also must javac against stripped bytecode, but I think it does make the world a saner place in at least one instance.

[niloc132]

Follow up to edit: MavenProject.getCompileClasspathElements could potentially work, in as much as we could turn those strings back into Files, assume they are all required without other examination, unzip each, try to figure out its MavenProject, and recurse until you have all the sources, strip, javac, j2cl each. Instead, I'm iterating over the MavenProject.getArtifacts() list, applying a ScopeArtifactFilter to likewise restrict to compile/provided (but not system, since system may vary), and visiting those artifacts. I think this is the same effect as you are describing, but still doesn't resolve this issue.

[tbroyer]

Sorry for the late reply, I only just now found out your rewrite of the j2cl-maven-plugin, to really understand why you were talking about.

Agreed that maven is flawed, but in the case of "we need to support the use case of a project actually compiling its dependencies", it actually is somewhat closer to accurate for our needs. If you ignore the lack of test-provided. And test-runtime. And anything to distinguish annotation processors at all. And...

Nobody needs (should need) test dependencies, they are "implementation details" of how the project was tested (at build time).
You are wrong though that Maven has nothing do “distinguish annotation processors”: https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#annotationProcessorPaths (see also https://blog.ltgt.net/maven-is-broken-by-design-2/ that I already posted above about that topic); and the same is true about them than about test dependencies: they are "build-time details" that users don't need to know about.
Your new j2cl-maven-plugin is flawed; it shouldn't expect to be somehow running annotation processors, or it should directly invoke Maven. It could however (should?) rely on the maven-compiler-plugin (which would be invoked by Maven somehow before invoking the j2cl-maven-plug goals) adding the generated sources as a compile source root; this is what the gwt-maven-plugin does.

Okay, either Gradle-based projects that are j2cl-compatible do not provide the "how do you compile me as if I was on your machine" details and so must ship already-j2cl'd-output, or those details must be provided.

I disagree; and this is equally true for Maven-built projects.

The root cause of all this discussion: your assumptions are flawed.

The inputs to JavaC and J2ClTranspiler are the same: the source files, the J2Cl-specific bootstrap classpath, and the bytecode for classes used in the source files (and all classes referenced from their API). In a J2Cl context, that classpath must not contain anything that is not J2Cl-compatible, i.e. it should not contain anything that is annotated with @GwtIncompatible.
Technically, you're not required to JavaC your application sources, but it provides better error-reporting that J2ClTranspiler (AFAIK), and it will run annotation processors if you have ones (IIUC, that may possibly change as J2Cl is being rebuilt on top of JavaC rather than ECJ). Because annotation processors are likely to use APIs that are not J2Cl-compatible, they really should be in the annotation processor path, separate from the classpath.
The inputs to the Closure Compiler are the outputs of J2ClTranspiler for all dependencies, transitively; this is a whole world step; conceptually similar to, e.g., building an uber-jar and running Proguard on it to prune unused code and obfuscate code: the output is a standalone, dependency-free artifact whose API consists of only the things that have explicitly been exposed (in the case of Closure, it could also contain code that "auto executes", but I think you can grasp the similarities between Closure Compiler to build an optimized JS, and Proguard to build an optimized uber-jar).

Let's create a small project with a single class:

import org.gwtproject.http.client.*;

public class Main {
  public static void main(String[] args) throws RequestException {
    var rb = new RequestBuilder(RequestBuilder.GET, "http://gwtproject.org");
    rb.setCallback(new RequestCallback() {
      @Override public void onResponseReceived(Request request, Response response) {}
      @Override public void onError(Request request, Throwable exception) {}
    });
    rb.send();
  }
}

To JavaC that class, you only need the gwt-http JAR in the classpath, you don't need elemental2 dependencies (really, try it). Same is true for J2ClTranspile.
In the case of gwt-http, there's no @GwtIncompatible. If there were, you would want to JavaC against stripped bytecode to make sure you won't call those @GwtIncompatible APIs (this could also be checked using a JavaC plugin, such as an ErrorProne check, but it would have to take into account overloads: one overload of a method could be @GwtIncompatible while another would not; in short, it's probably easier to use a stripped bytecode).

Then, to Closure Compile the “application”, you need the J2ClTranspile output for all of gwt-http, elemental2-dom, elemental2-core, and jsinterop-base (and possibly jsinterop-annotations).
To J2ClTranspile those, you need their source code (same as GWT), and the bytecode for their API dependencies. To J2ClTranspile gwt-http, you need the bytecode for elemental2-core, elemental2-dom, and jsinterop-base (the versions stripped from their @GwtIncompatible APIs, if any); in this case, those dependencies don't have runtime dependencies, but if they had, you wouldn't need them here (you would however need their J2Cl-Transpile output for the final Closure Compiler step).

Want a slightly more complex example? Make that simple class a library:

package lib;
import org.gwtproject.http.client.*;

public class Ping {
  public static void ping(String url) throws Exception {
    var rb = new RequestBuilder(RequestBuilder.GET, url);
    rb.setCallback(new RequestCallback() {
      @Override public void onResponseReceived(Request request, Response response) {}
      @Override public void onError(Request request, Throwable exception) {}
    });
    rb.send();
  }
}

and make another project that uses that library:

import lib.Ping;

public class Main {
  public static void main(String[] args) throws Exception {
    Ping.ping("http://gwtproject.org");
  }
}

Now, lib has a runtime dependency on gwt-http (you couldn't express that with a Maven build, but really, gwt-http is needed to compile the Ping class, but you don't need it to compile the Main class, only to run it; try it)

So, to JavaC/J2ClTranspile app, you only need lib's bytecode. But to Closure Compile the app, you need to J2ClTranspile lib. To J2ClTranspile lib, you need gwt-http (from the runtime dependencies of lib), but still not elemental2 or jsinterop-base (they're runtime dependencies of gwt-http).

In your new j2cl-maven-plugin, you create a Maven project for each dependency; when processing the lib dependency, you only need the compile (bytecode) dependencies to J2ClTranspile its sources; but you'll need to J2ClTranspile its runtime dependencies (to later give them to the Closure Compiler).
This means that you don't need to strip @GwtIncompatible APIs out of gwt-http when you JavaC or J2ClTranspile app, but you will have to to J2ClTranspile lib.

Now, anywhere above where I said strip @GwtIncompatible APIs, that only applied to bytecode, so it could technically be done using ASM (or you could choose to transform the sources and JavaC them, in which case you need the project's compile dependencies); and anywhere above where I said J2ClTranspile, this is actually strip @GwtIncompatible APIs from the sources and J2ClTranspile the transformed sources.

[niloc132]

To J2ClTranspile gwt-http, you need the bytecode for elemental2-core, elemental2-dom, and jsinterop-base (the versions stripped from their @GwtIncompatible APIs, if any); in this case, those dependencies don't have runtime dependencies, but if they had, you wouldn't need them here (you would however need their J2Cl-Transpile output for the final Closure Compiler step).

or you could choose to transform the sources and JavaC them, in which case you need the project's compile dependencies

This is really the only point I'm after - given a toolchain which considers that option to be the best way forward, and given that you accept this to be a valid option, how does any tooling ask gwt-http what the project's compile-time dependencies are? The published pom doesn't contain anything except scope=runtime, which is certainly accurate for using the jar in a jvm, but insufficient for either j2cl or transforming+javac'ing the sources. How would I read that pom or any other metadata that comes out of a publishToMaven* goal, and discover what classpath is needed to invoke either J2clTranspiler, or Javac?

--

I'm happy to discuss the poor maven conventions rampant across the ecosystem separately, but to achieve these specific steps without building separate required-for-j2cl-but-not-part-of-j2cl tooling, this appears to be the right step forward, and is the process that I had understood bazel to be using. Aside from a few minor quibbles around how maven-compiler-plugin and the greater maven ecosystem describe that they should be used, I believe I agree with everything you said - but the actual "how does the plugin get gwt-http's own compiletime classpath" so that j2cl and javac can be invoked remains.

If you poke around in the current HEAD of the plugin, you'll see that each upstream project gets only its own compile-time classpath passed to j2cl (rather than the whole world) as you suggest (plus or minus broken jsinterop-base jars from goog...). Similarly, stripped bytecode is not generated unless there is actually a downstream project which requires it, though my oversimplification in the earlier post might have made this unclear, in my effort to not make the post too long.

[tbroyer]

To J2ClTranspile lib, you need gwt-http (from the runtime dependencies of lib), but still not elemental2 or jsinterop-base (they're runtime dependencies of gwt-http).

Use the same rule to JavaC the stripped sources of lib; whether it comes from the same "reactor" or is an external module doesn't really matter.

As a simplification, you can use the whole runtime classpath instead (instead of the compile classpath of the direct runtime dependencies). And of you want to be "extra sure", use provided+runtime.