[Bug] ProtobufNativeSchemaTest.testSchema order-independent #24804
Description
Search before reporting
- I searched in the issues and found nothing similar.
Read release policy
- I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
User environment
Broker version: 4.2.0-SNAPSHOT
Broker Operating system and hardware type: Linux x86_64 GNU/Linux
Broker Java version: 17.0.16
Issue Description
In ProtobufNativeSchemaTest.testSchema, the json's contents and the FileDescriptorSet field's contents do not have a deterministic order but the hardcoded string assertion assumes a deterministic order. The json serializer did not guarantee attribute order and inside FileDescriptorSet the contents can also be in different orders due to different generation paths or environments producing the contents in different orders despite the logical content being the same. Since the test compares the raw strings/trees "as-is", harmless re-ordering could flip the test from pass to fail without any real schema change.
I discovered the problem with the NonDex tool. NonDex systematically detects incorrect tests that rely on non-deterministic behaviors in Java APIs—like assuming order of name/value pairs in json files — by exploring all specification-allowed outcomes. It does this by instrumenting undetermined APIs and randomizing the returned order/behavior within what the spec allows. Failures it exposes reliably indicate flawed assumptions in order that were never promised and should be fixed.
Error messages
[ERROR] Failures:
[ERROR] org.apache.pulsar.client.impl.schema.ProtobufNativeSchemaTest.testSchema
[INFO] Run 1: PASS
[ERROR] Run 2: ProtobufNativeSchemaTest.testSchema:71 expected [{"fileDescriptorSet":"CtMDCgpUZXN0LnByb3RvEgVwcm90bxoSRXh0ZXJuYWxUZXN0LnByb3RvImUKClN1Yk1lc3NhZ2USCwoDZm9vGAEgASgJEgsKA2JhchgCIAEoARo9Cg1OZXN0ZWRNZXNzYWdlEgsKA3VybBgBIAEoCRINCgV0aXRsZRgCIAEoCRIQCghzbmlwcGV0cxgDIAMoCSLlAQoLVGVzdE1lc3NhZ2USEwoLc3RyaW5nRmllbGQYASABKAkSEwoLZG91YmxlRmllbGQYAiABKAESEAoIaW50RmllbGQYBiABKAUSIQoIdGVzdEVudW0YBCABKA4yDy5wcm90by5UZXN0RW51bRImCgtuZXN0ZWRGaWVsZBgFIAEoCzIRLnByb3RvLlN1Yk1lc3NhZ2USFQoNcmVwZWF0ZWRGaWVsZBgKIAMoCRI4Cg9leHRlcm5hbE1lc3NhZ2UYCyABKAsyHy5wcm90by5leHRlcm5hbC5FeHRlcm5hbE1lc3NhZ2UqJAoIVGVzdEVudW0SCgoGU0hBUkVEEAASDAoIRkFJTE9WRVIQAUItCiVvcmcuYXBhY2hlLnB1bHNhci5jbGllbnQuc2NoZW1hLnByb3RvQgRUZXN0YgZwcm90bzMKoAEKEkV4dGVybmFsVGVzdC5wcm90bxIOcHJvdG8uZXh0ZXJuYWwiOwoPRXh0ZXJuYWxNZXNzYWdlEhMKC3N0cmluZ0ZpZWxkGAEgASgJEhMKC2RvdWJsZUZpZWxkGAIgASgBQjUKJW9yZy5hcGFjaGUucHVsc2FyLmNsaWVudC5zY2hlbWEucHJvdG9CDEV4dGVybmFsVGVzdGIGcHJvdG8z","rootMessageTypeName":"proto.TestMessage","rootFileDescriptorName":"Test.proto"}] but found [{"rootFileDescriptorName":"Test.proto","rootMessageTypeName":"proto.TestMessage","fileDescriptorSet":"CtMDCgpUZXN0LnByb3RvEgVwcm90bxoSRXh0ZXJuYWxUZXN0LnByb3RvImUKClN1Yk1lc3NhZ2USCwoDZm9vGAEgASgJEgsKA2JhchgCIAEoARo9Cg1OZXN0ZWRNZXNzYWdlEgsKA3VybBgBIAEoCRINCgV0aXRsZRgCIAEoCRIQCghzbmlwcGV0cxgDIAMoCSLlAQoLVGVzdE1lc3NhZ2USEwoLc3RyaW5nRmllbGQYASABKAkSEwoLZG91YmxlRmllbGQYAiABKAESEAoIaW50RmllbGQYBiABKAUSIQoIdGVzdEVudW0YBCABKA4yDy5wcm90by5UZXN0RW51bRImCgtuZXN0ZWRGaWVsZBgFIAEoCzIRLnByb3RvLlN1Yk1lc3NhZ2USFQoNcmVwZWF0ZWRGaWVsZBgKIAMoCRI4Cg9leHRlcm5hbE1lc3NhZ2UYCyABKAsyHy5wcm90by5leHRlcm5hbC5FeHRlcm5hbE1lc3NhZ2UqJAoIVGVzdEVudW0SCgoGU0hBUkVEEAASDAoIRkFJTE9WRVIQAUItCiVvcmcuYXBhY2hlLnB1bHNhci5jbGllbnQuc2NoZW1hLnByb3RvQgRUZXN0YgZwcm90bzMKoAEKEkV4dGVybmFsVGVzdC5wcm90bxIOcHJvdG8uZXh0ZXJuYWwiOwoPRXh0ZXJuYWxNZXNzYWdlEhMKC3N0cmluZ0ZpZWxkGAEgASgJEhMKC2RvdWJsZUZpZWxkGAIgASgBQjUKJW9yZy5hcGFjaGUucHVsc2FyLmNsaWVudC5zY2hlbWEucHJvdG9CDEV4dGVybmFsVGVzdGIGcHJvdG8z"}]
[INFO]
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
INFO: Surefire failed when running tests for c8A8RmGSuO8XYxjfW87EPw3D18O8xdB4vMgX7ULah0=
[INFO] NonDex SUMMARY:
[INFO] *********
[INFO] mvn nondex:nondex -DnondexFilter='.*' -DnondexMode=FULL -DnondexSeed=974622 -DnondexStart=0 -DnondexEnd=9223372036854775807 -DnondexPrintstack=false -DnondexDir="/home/leby2/pulsar/pulsar-client/.nondex" -DnondexJarDir="/home/leby2/pulsar/pulsar-client/.nondex" -DnondexExecid=c8A8RmGSuO8XYxjfW87EPw3D18O8xdB4vMgX7ULah0= -DnondexLogging=CONFIG
[WARNING] org.apache.pulsar.client.impl.schema.ProtobufNativeSchemaTest#testSchema
[INFO] *********
[INFO] All tests pass without NonDex shuffling
[INFO] ####################
[INFO] Across all seeds:
[INFO] org.apache.pulsar.client.impl.schema.ProtobufNativeSchemaTest#testSchema
[INFO] Test results can be found at:
[INFO] file:///home/leby2/pulsar/pulsar-client/.nondex/c8A8RmGSuO8XYxjfW87EPw3D18O8xdB4vMgX7ULah0=/test_results.html
[INFO] [NonDex] The id of this run is: c8A8RmGSuO8XYxjfW87EPw3D18O8xdB4vMgX7ULah0=
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26.577 s
[INFO] Finished at: 2025-10-01T10:30:04-05:00
[INFO] ------------------------------------------------------------------------
[INFO] 0 goals, 0 executed
[ERROR] Failed to execute goal edu.illinois:nondex-maven-plugin:2.2.1:nondex (default-cli) on project pulsar-client-original: Unable to execute mojo: There are test failures.
Reproducing the issue
You can replicate the problem with the NonDex tool command:
mvn -pl pulsar-client -Dtest=org.apache.pulsar.client.impl.schema.ProtobufNativeSchemaTest#testSchema -DforkCount=1 -DreuseForks=false -DnondexSeed=974622 -DnondexRuns=1 edu.illinois:nondex-maven-plugin:2.2.1:nondex
Additional information
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!