PBJ Compiler fails with FileNotFoundException

[jsync-swirlds]

in Common.java line 634, PBJ compiler attempts to read a Java file for a message during PBJ compilation. This fails in some cases because the message reference is not yet, itself, compiled by PBJ, or because the file is not in the expected location when running in the Github CI environments. There will need to be either a check of the message definition, or a mechanism to ensure the referenced message is compiled before the current message and is read from the correct location.

Note: A partial workaround exists (ensure messages are declared before they are used in the same file), but this still fails in the CI environment.

Sample error message during compilation:

Exception while processing file: /Users/user/Projects/Hashgraph/hedera-services/hedera-node/hapi/hedera-protobufs/services/basic_types.proto
java.lang.RuntimeException: java.io.FileNotFoundException: /Users/user/Projects/Hashgraph/hedera-services/hedera-node/hapi/build/generated/source/pbj-proto/main/java/com/hedera/hapi/node/base/TokenID.java (No such file or directory)
    com.hedera.pbj.compiler.impl.Common.verifyComparable(Common.java:645)
    com.hedera.pbj.compiler.impl.Common.getFieldsCompareToStatements(Common.java:593)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generateCompareTo(ModelGenerator.java:287)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generate(ModelGenerator.java:151)
    com.hedera.pbj.compiler.PbjCompilerTask.compileFilesIn(PbjCompilerTask.java:107)
    com.hedera.pbj.compiler.PbjCompilerTask.perform(PbjCompilerTask.java:72)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)

Another example, this time from a separate file that imports basic_types failing in the CI environment:

Exception while processing file: /runner/_work/hedera-services/hedera-services/hedera-node/hapi/hedera-protobufs/services/state/token/account.proto
java.lang.RuntimeException: java.io.FileNotFoundException: /runner/_work/hedera-services/hedera-services/hedera-node/hapi/build/generated/source/pbj-proto/main/java/com/hedera/hapi/node/base/TokenID.java (No such file or directory)
    com.hedera.pbj.compiler.impl.Common.verifyComparable(Common.java:645)
    com.hedera.pbj.compiler.impl.Common.getFieldsCompareToStatements(Common.java:593)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generateCompareTo(ModelGenerator.java:287)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generate(ModelGenerator.java:151)
    com.hedera.pbj.compiler.PbjCompilerTask.compileFilesIn(PbjCompilerTask.java:107)
    com.hedera.pbj.compiler.PbjCompilerTask.perform(PbjCompilerTask.java:72)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)

Relevant code segment:

	private static void verifyComparable(final Field field, File destinationSrcDir) {
		if (field instanceof final SingleField singleField) {
			if (singleField.type() != Field.FieldType.MESSAGE) {
				// everything else except message and bytes is comparable for sure
				return;
			}
			// let's check if the message implements Comparable
			final String className = singleField.javaFieldType();
			final File javaFile = getJavaFile(destinationSrcDir, singleField.messageTypeModelPackage(), className);
			try (BufferedReader reader = new BufferedReader(new FileReader(javaFile))) {
				String line;
				while ((line = reader.readLine()) != null) {
					if (COMPARABLE_PATTERN.matcher(line).matches()) {
						return;
					}
				}
				throw new IllegalArgumentException(("Field %s.%s specified in `pbj.comparable` option must implement " +
						"`Comparable` interface but it doesn't.").formatted(className, field.nameCamelFirstLower()));
			} catch (IOException e) {
				throw new RuntimeException(e);
			}
        } if (field instanceof final OneOfField oneOfField) {
			oneOfField.fields().forEach(v -> verifyComparable(v, destinationSrcDir));
		} else {
			throw new UnsupportedOperationException("Unexpected field type - " + field.getClass());
		}
	}

Additional failure output. This shows that the issue is file processing order. In order to resolve this the parsing and compilation likely must improve (significantly) to track comparable without relying on reading generated files.

Exception while processing file: /runner/_work/pbj/pbj/pbj-integration-tests/src/main/proto/everything.proto
java.lang.RuntimeException: java.io.IOException: Unable to read Java file /runner/_work/pbj/pbj/pbj-integration-tests/build/generated/source/pbj-proto/main/java/com/hedera/pbj/test/proto/pbj/TimestampTest.java for class TimestampTest in package com.hedera.pbj.test.proto.pbj under source root /runner/_work/pbj/pbj/pbj-integration-tests/build/generated/source/pbj-proto/main/java.
    com.hedera.pbj.compiler.impl.Common.verifyComparable(Common.java:658)
    com.hedera.pbj.compiler.impl.Common.getFieldsCompareToStatements(Common.java:603)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generateCompareTo(ModelGenerator.java:287)
    com.hedera.pbj.compiler.impl.generators.ModelGenerator.generate(ModelGenerator.java:151)
    com.hedera.pbj.compiler.PbjCompilerTask.compileFilesIn(PbjCompilerTask.java:107)
    com.hedera.pbj.compiler.PbjCompilerTask.perform(PbjCompilerTask.java:72)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)

Files in Target Folder:
    pbj/
        MessageWithString.java
        InnerEverything.java
        codec/
            InnerEverythingProtoCodec.java
            InnerEverythingJsonCodec.java
            EverythingProtoCodec.java
            MessageWithStringJsonCodec.java
            EverythingJsonCodec.java
            MessageWithStringProtoCodec.java
       schema/
           EverythingSchema.java
           MessageWithStringSchema.java
           InnerEverythingSchema.java
       Everything.java
       Suit.java

There is a draft PR which exercises this issue #189