Why is deadLetterPolicy attribute in the ConsumerConfigurationData class marked as transient?

[AnuragReddy2000]

As a user of Apache Pulsar, I have been utilising PulsarSpout exposed by the pulsar-storm adaptor within an Apache Storm topology for message consumption from Pulsar topics. Recently, we encountered an issue where the deadLetterPolicy that we set on the PulsarSpout seems to be lost when the topology is started. Upon investigation, we found that this is due to the deadLetterPolicy attribute in ConsumerConfigurationData being marked as transient.

Due to Storm topologies being serialised and transferred from the nimbus to workers for execution, the transient nature of this attribute results in the loss of our specified configuration. I was wondering if anyone knows why this attribute needs to be transient?

Ideally implementing the java.io.Serializable interface in the DeadLetterPolicy class should eliminate the need for the transient keyword. If this is accurate and there is no other reason for it, then I am prepared to submit a pull request for the same.

[AnuragReddy2000]

I see that this change was introduced in this PR which aimed to integrate SpotBugs in the pulsar-client module. Considering that the ConsumerConfigurationData class extends the java.io.Serializable interface , SpotBugs would check if all the attributes of that class are serializable or not. The attribute deadLetterPolicy whose type DeadLetterPolicy did not extend the aforementioned interface, would surely be flagged by SpotBugs and I believe this led to the author of that PR to mark this attribute as transient to get around it.

If some one can confirm this, I can submit a PR with the changes to make the DeadLetterPolicy class serializable & remove the transient keyword before the deadLetterPolicy attribute.

[lhotari]

@AnuragReddy2000 Thanks for investigating this issue. It would be useful to first create an issue in https://github.com/apache/pulsar/issues about the problem that you are facing with the pulsar-storm adaptor when the deadLetterPolicy gets lost.

I don't see a reason why field deadLetterPolicy should be transient. In the referenced PR #9630, the correct resolution would have been to make DeadLetterPolicy class implement java.io.Serializable and add private static final long serialVersionUID = 1L; field instead of making it a transient field. It looks like there are multiple other problems in the PR and deadLetterPolicy isn't the only issue. The same issue applies to keySharedPolicy and messageCrypto fields.

Some of the classes and fields in the object tree of ConsumerConfigurationData would require custom serialization to properly resolve the issue and possibly some other solutions. For example MessageCrypto implementations are such. It might not be trivial to resolve it.

deadLetterPolicy and keySharedPolicy seem straightforward to resolve. Making the implementation classes implement java.io.Serializable and adding private static final long serialVersionUID = 1L; field.

Please go ahead, create the issue and the PR to address this.

[AnuragReddy2000]

Thank you for the response! I shall do as instructed.

[AnuragReddy2000]

@codelipenghui Can you comment on this, as the aforementioned PR was raised by you?

[AnuragReddy2000]

Created an issue for the same: #23704.