Does current implementation of package manager support Oxia?

[anosulchik]

Hi, I'm trying to enable pulsar packages management for functions (also for sinks and sources) in the cluster that uses Oxia as a metadata service. Brokers fails to initialize package management throwing an error java.net.URI.getAuthority() for oxia:// URLs. Does package management supports Oxia at this stage or ZK is required?

Brokers config excerpt:

      # ...
      bookkeeperMetadataServiceUri: metadata-store:oxia://mi-apache-pulsar-staging-oxia-svc:6648/bookkeeper
      configurationMetadataStoreUrl: oxia://mi-apache-pulsar-staging-oxia-svc:6648/broker
      metadataStoreUrl: oxia://mi-apache-pulsar-staging-oxia-svc:6648/broker
      # ...
      enablePackagesManagement: "true"
      packagesManagementStorageProvider: "org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider"
      packagesReplicas: "1"
      packagesManagementLedgerRootPath: "/ledgers"
      functionsWorkerEnablePackageManagement: "true"

Logs can be found below, please see org.apache.pulsar.broker.PulsarService.startPackagesManagementService in a stacktrace:

2025-02-25T18:51:48,511+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-metadata-tailer-reader-c276757b0f] Subscribed to topic on mi-apache-pulsar-staging-broker-1.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local/<ipadress>:6650 -- consumer: 4
2025-02-25T18:51:48,515+0000 [main] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataManager - MetaData Manager Tailer started
2025-02-25T18:51:48,518+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - /** Starting Cluster Service Coordinator **/
2025-02-25T18:51:48,518+0000 [main] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - /** Starting cluster service coordinator **/
2025-02-25T18:51:48,537+0000 [main] INFO  org.apache.pulsar.functions.worker.PulsarWorkerService - /** Started worker id=c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080 **/
2025-02-25T18:51:48,537+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Function worker service started
2025-02-25T18:51:48,596+0000 [main] INFO  org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage - Applying DLog BookKeeper client configuration setting bkc.tlsHostnameVerificationEnabled=false
2025-02-25T18:51:48,602+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initializeDlogNamespace(BookKeeperPackagesStorage.java:100) ~[org.apache.pulsar-pulsar-package-bookkeeper-storage-4.0.2.jar:4.0.2]
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initialize(BookKeeperPackagesStorage.java:87) ~[org.apache.pulsar-pulsar-package-bookkeeper-storage-4.0.2.jar:4.0.2]
	at org.apache.pulsar.broker.PulsarService.startPackagesManagementService(PulsarService.java:1965) ~[org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1055) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:275) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:130) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine.access$1500(CommandLine.java:148) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at picocli.CommandLine.execute(CommandLine.java:2170) [info.picocli-picocli-4.7.5.jar:4.7.5]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:151) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
org.apache.pulsar.broker.PulsarServerException: java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.broker.PulsarServerException.from(PulsarServerException.java:56)
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1088)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:275)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.call(PulsarBrokerStarter.java:130)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:151)
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:366)
Caused by: java.lang.NullPointerException: Cannot invoke "String.replace(java.lang.CharSequence, java.lang.CharSequence)" because the return value of "java.net.URI.getAuthority()" is null
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initializeDlogNamespace(BookKeeperPackagesStorage.java:100)
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:366) [org.apache.pulsar-pulsar-broker-4.0.2.jar:4.0.2]
	at org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorage.initialize(BookKeeperPackagesStorage.java:87)
	at org.apache.pulsar.broker.PulsarService.startPackagesManagementService(PulsarService.java:1965)
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:1055)
	... 12 more
2025-02-25T18:51:48,655+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - Stopping function metadata tailer
2025-02-25T18:51:48,660+0000 [function-metadata-tailer-thread] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - metadata tailer thread exiting
2025-02-25T18:51:48,666+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata] [c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-metadata-tailer-reader-c276757b0f] Closed consumer
2025-02-25T18:51:48,668+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionMetaDataTopicTailer - Stopped function metadata tailer
2025-02-25T18:51:48,669+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.FunctionAssignmentTailer - Closing function assignment tailer
2025-02-25T18:51:48,672+0000 [assignment-tailer-thread] INFO  org.apache.pulsar.functions.worker.FunctionAssignmentTailer - assignment tailer thread exiting
2025-02-25T18:51:48,677+0000 [pulsar-client-io-48-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/assignments] [c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080-function-assignment-tailer-reader-f137cfeee3] Closed consumer
2025-02-25T18:51:48,678+0000 [pulsar-service-shutdown] WARN  org.apache.pulsar.functions.worker.FunctionRuntimeManager - Will not stop any functions since they are externally managed
2025-02-25T18:51:48,679+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - Stopping Cluster Service Coordinator for worker c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080
2025-02-25T18:51:48,681+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.ClusterServiceCoordinator - Stopped Cluster Service Coordinator for worker c-mi-apache-pulsar-staging-fw-mi-apache-pulsar-staging-broker-2.mi-apache-pulsar-staging-broker.mi-apache-pulsar.svc.cluster.local-8080
2025-02-25T18:51:48,682+0000 [pulsar-service-shutdown] INFO  org.apache.pulsar.functions.worker.SchedulerManager - Closing scheduler manager

[lhotari]

No, Oxia isn't supported. The supported option currently is file storage, but that's not available in the Apache Pulsar Helm chart configs. There would need to be NFS or some other shared storage configured in the cluster. Cloud provider k8s support this, but adding the support to the Helm chart would be needed.
An alternative would be to have a package service that uses cloud storage (S3) directly. That's missing from Apache Pulsar.

[beelis]

Hi @lhotari Are there any plans to support oxia soon? I didn't find a ticket, but I would also be interested in switching to Oxia, which is currently not possible since functions are used.

[lhotari]

@beelis I don't think that anyone is currently working on it. It would require some volunteers to contribute such a feature.
A package service directly using cloud storage (S3) would be generic for the purpose.
You could start a discussion on the dev list to get a broader audience.

[lhotari]

@beelis The existing org.apache.pulsar.packages.management.storage.filesystem.FileSystemPackagesStorage would be usable if there's NFS or some other shared storage configured in the cluster. Perhaps adding support for that would be the first way to get forward. Are you using Apache Pulsar Helm chart for deployments?
Cloud provider shared filesystem options for k8s clusters:

• GCP Filestore, integrates with GKE
• AWS Elastic File System (EFS), integrates with AKS
• Azure Files, integrates with AKS

It shouldn't be a huge effort to make Apache Pulsar Helm chart changes to support those options. We just need someone to do it.

[lhotari]

@anosulchik @beelis One possible OSS solution would be to add support to Apache Pulsar Helm chart for configuring JuiceFS to be used as a shared file system together with org.apache.pulsar.packages.management.storage.filesystem.FileSystemPackagesStorage. That would be a low effort solution to have the packages stored in cloud storage (S3) in a portable and cloud agnostic way.
It would be useful to have the Apache Pulsar Helm chart have separate Function Worker pods instead of the current setup where all broker nodes are also Function Workers.

Would you be interested in contributing this type of improvement to Apache Pulsar Helm chart?