Bound nesting depth of deserialized payloads

[claude-code-staging]

Motivation

Serializers for wrapper messages deserialize the wrapped payload by re-entering the serialization infrastructure, and the nesting depth of such payloads is decided by the sender of the message. Nothing tracked the cumulative depth, so a single bounded message that nests payloads deep enough made deserialization recurse until the thread stack was exhausted. The resulting StackOverflowError is a VirtualMachineError, so it is not caught by the NonFatal handlers around deserialization and escaped the deserialization stage instead of only failing that message.

QuerySerializer is an example that is reachable from the remoting inbound port. It is registered as serializer id 39 whenever pekko-persistence-query is on the classpath, and the event and metadata of a QueryMessages.EventEnvelope are opaque ContainerFormats.Payloads, so each one can name serializer id 39 and manifest "a" again:

fromBinary -> WrappedPayloadSupport.deserializePayload -> fromBinary -> ...

Protobuf's own nested-message recursion limit does not bound this, because each level lives in the bytes enclosedMessage field and is only parsed by the next fromBinary call. A level costs only a few tens of bytes, so thousands of levels fit inside the frame limits. The offset route (serialization.deserialize from fromStorageRepresentation) has the same shape, as do the other wrapper serializers (MiscMessageSerializer, StreamRefSerializer, ShardingSerializer, ReliableDeliverySerializer, distributed-data's SerializationSupport, ...).

Modification

Track the nested deserialization depth of the current thread in Serialization and abort with a NotSerializableException when it would exceed pekko.serialization.max-nested-deserialization-depth, which defaults to 100.

The counter is incremented at the points where deserialization can be re-entered, once per nesting level:

• Serialization.deserializeByteArray, which covers deserialize(bytes, serializerId, manifest) and the array fallback of deserializeByteBuffer
• the ByteBufferSerializer branch of Serialization.deserializeByteBuffer
• Serialization.deserialize(bytes, clazz)
• the two branches of WrappedPayloadSupport.deserializePayload that invoke the payload serializer directly, bypassing Serialization; its third branch goes through Serialization.deserialize and is already counted

The counter is a thread local, deliberately shared by all ActorSystems of the JVM, because the resource it protects, the thread stack, is shared too. The guard method is kept small so it stays inlinable on the artery hot path.

Result

Deserializing one message recurses at most max-nested-deserialization-depth levels, no matter how deeply the wire payload nests envelopes or payloads. Over-limit input fails with a NotSerializableException, which is NonFatal and is treated like any other deserialization failure, so artery (Codecs.Deserializer) and classic remoting (Endpoint) log it and drop the message, and the inbound lane stays up.

The default is far above the nesting depth that Pekko itself produces, so legitimate traffic is unaffected.

Tests

• Not run - sbt, scalafmt and a Scala toolchain are not installed in this environment and it has no network access to a Maven repository. sbt "actor-tests / Test / testOnly org.apache.pekko.serialization.NestedDeserializationDepthSpec", sbt "persistence-query / Test / testOnly org.apache.pekko.persistence.query.internal.QuerySerializerSpec", sbt "actor-tests / Test / testOnly org.apache.pekko.config.ConfigSpec", sbt +mimaReportBinaryIssues, sbt scalafmtAll and sbt headerCreateAll were all skipped and need to be run in CI. No new files were added, so no license headers are needed, and the change only adds members, so binary compatibility is preserved.
• Added NestedDeserializationDepthSpec in actor-tests: a serializer that unwraps one nesting level per call, exercised at the limit and one above it, through both deserialize(bytes, id, manifest) and deserializeByteBuffer, plus a case asserting that a rejected message leaves the thread's counter clean for the next one.
• Added two QuerySerializerSpec cases that build a nested QueryMessages.EventEnvelope by hand: one within the limit that must deserialize to the expected nesting, and one nested well beyond it that must fail with NotSerializableException. Before this change the second one overflows the stack or returns a deeply nested envelope.
• Added the default of the new setting to ConfigSpec.

References

None - hardening of the deserialization paths that re-enter on attacker-controlled nested payloads.

---

Generated by Claude Code