[Bug] readOffload NoClassDefFoundError

[KannarFr]

Search before asking

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

4.0.1

Minimal reproduce step

It can't reproduce on demand, but when a broker has the issue, it always has it.

What did you expect to see?

Nothing

What did you see instead?

Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,810+0000 [offloader-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,811+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0] Error opening ledger for reading at position 162128641:0 - org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] WARN  org.apache.bookkeeper.mledger.impl.OpReadEntry - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0][reader-9bd1511927] read failed from ledger at position:162128641:0
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Anything else?

Offloaders nar is present. When restarting the broker, the topic is assigned to another broker and it doesn't have the issue now. Also, this error is mixed in many exceptions related to #23735.

Are you willing to submit a PR?

• I'm willing to submit a PR!