NettyNioAsyncHttpClient:NPE in HandlerSubscriber.maybeRequestMore (null subscription) when channelWritabilityChanged fires during the Expect: 100-continue window

[Adiiigo]

Describe the bug

HandlerSubscriber.maybeRequestMore() calls subscription.request(...) with no null check, and channelWritabilityChanged() calls it unconditionally (unlike channelActive(), which is state-gated). subscription is assigned only in onSubscribe() and never reset to null.

For Expect: 100-continue requests, HttpStreamsClientHandler.subscribeSubscriberToStream() deliberately defers the body subscription until the 100 Continue response arrives. The HandlerSubscriber is nonetheless already in the pipeline (added by HttpStreamsHandler.unbufferedWrite()), so it keeps receiving channel events while subscription == null — for a full server round-trip. If channelWritabilityChanged fires in that window, maybeRequestMore() NPEs.

The S3 client sets Expect: 100-continue on every PutObject/UploadPart (StreamingRequestInterceptor), so every async S3 upload takes this path.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The async client should stream the body and complete the upload. During the Expect: 100-continue wait, channelWritabilityChanged must not dereference a not-yet-assigned subscription; it should no-op until onSubscribe runs, then request demand normally.

Current Behavior

Non-retryable failure of the upload:

  Caused by: java.lang.NullPointerException: Cannot invoke "org.reactivestreams.Subscription.request(long)" because "this.subscription" is null
      at ...netty.internal.nrs.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:303)
      at ...netty.internal.nrs.HandlerSubscriber.channelWritabilityChanged(HandlerSubscriber.java:157)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelWritabilityChanged(...)
      ...

Wrapped as SdkClientException: Unable to execute HTTP request: ... "this.subscription" is null and classified non-retryable, so the operation fails.

Reproduction Steps

No standalone repro attached yet. Deterministic conditions: S3AsyncClient PutObject/UploadPart (so Expect: 100-continue is set) against a server that delays the 100 Continue, with a channel writability transition forced during the wait (small write-buffer high-water mark / backpressured body). Happy to contribute a minimal repro.

Possible Solution

Null-guard the deref. (Illustrative — not yet compiled or covered by a test.) When subscription == null the method simply returns; demand is requested when onSubscribe later runs (provideSubscription → maybeStart → maybeRequestMore):

  private void maybeRequestMore() {
      if (subscription != null
              && outstandingDemand <= demandLowWatermark
              && ctx.channel().isWritable()) {
          long toRequest = demandHighWatermark - outstandingDemand;
          outstandingDemand = demandHighWatermark;
          subscription.request(toRequest);
      }
  }
  

Alternatively (or additionally), gate channelWritabilityChanged on state == RUNNING, mirroring channelActive.

Note: S3Configuration.expectContinueEnabled(false) (newer SDK versions) suppresses the header and hides the symptom, but that's a workaround — the unguarded deref in HandlerSubscriber is the actual defect and stays reachable by any path that defers the body subscription past the current event-loop task.

Additional Information/Context

Root cause (line numbers from tag 2.30.31; master has identical channelWritabilityChanged / maybeRequestMore / channelActive bodies)

1. HandlerSubscriber#maybeRequestMore — subscription.request(toRequest) guarded only by isWritable(), no null/state check.
2. HandlerSubscriber#channelWritabilityChanged — calls maybeRequestMore() unguarded (contrast channelActive, which checks state).
3. HttpStreamsHandler#unbufferedWrite — adds the HandlerSubscriber to the pipeline, then calls subscribeSubscriberToStream(...).
4. HttpStreamsClientHandler#subscribeSubscriberToStream — when is100ContinueExpected(msg), stashes the subscriber and does not subscribe; the subscribe (→ onSubscribe, which sets subscription) happens later in channelRead() on the 100 Continue.

Since subscription is set only in onSubscribe and never nulled: subscription == null ⇔ onSubscribe hasn't run ⇔ (for these requests) still awaiting 100 Continue. Any channelWritabilityChanged in that window hits the unguarded deref.

Why this isn't seen more widely

• Only PutObject/UploadPart carry Expect: 100-continue, so only uploads defer the subscription; reads never reach this state. (In our workload, 100% of occurrences are UploadPart, none on reads.)
• It also requires a writability change (channel becoming writable) inside that round-trip window — most plausibly the request-head/TLS write draining back below the low-water mark; we haven't traced the exact per-occurrence trigger. That's rare on an idle connection but frequent under high-concurrency uploads with backpressured sockets, so low-volume callers effectively never hit it.
• The same defect was reported upstream long ago (playframework/netty-reactive-streams#30) and never fixed; the code was vendored here and is still unguarded on master.

AWS Java SDK version used

2.30.31 (also on master); client: netty-nio-client; Netty 4.1.135.Final

JDK version used

21

Operating System and version

Linux

---

Additional context (caller-side use case, domain load, upload code path, observed signature): see comment below (#7271 (comment))

[Adiiigo]

Caller context and observed load

Use case

The affected deployment is AWS OpenSearch Service (managed OpenSearch) with the remote-store feature enabled (S3-backed segment durability). Under sustained high-volume ingest, every data node continuously uploads new segment files to S3 via the async S3 client — the workload that reliably exposes this NPE.

Domain shape and load (incident of manifestation, 24 h window)

Cluster: OpenSearch 3.5, ~177 data nodes (mixed r-family, hot tier only), 3 dedicated cluster-managers, single region. Bundled aws-sdk-java v2.30.31, netty 4.1.135.Final, reactive-streams 1.0.4.

Concurrent write drivers: 4 concurrent OpenSearch Ingestion (OSIS) pipelines feeding the cluster; ~150 primary shards across active indices at peak; indexing rate at incident peak ≈ 4× the 7-day pre-incident baseline.

Indicators: Incident peak that establish the scale of the caller — enough concurrent upload traffic and backpressured sockets to make the writability-flip trigger fire in bulk:

Signal	Peak value	Notes
Data-node CPU	94 – 97 %	Multi-hour saturation.
JVM heap pressure	~88 %	No OOM / no circuit breakers tripped.
IndexingLatency (max)	~70 s	Heavy end-to-end backpressure.
ThreadpoolWriteRejected (cumulative)	~55 M	Write threadpool saturated.
RemoteStorePressureService — bytes_lag vs threshold	~3.5 GB vs 1.12 GB (≈ 3.5×)	Rejecting writes on lag.
RemoteStorePressureService — time_lag vs threshold	~10.8 s vs 7.5 s (≈ 1.4×)	Same.

(These indicators are not the mechanism of the NPE — the NPE is a latent SDK race. They quantify the caller-side load under which the race actually fires.)

Client build

OpenSearch's repository-s3 plugin builds the S3 async client in S3AsyncService.buildAsyncClient(...) using the default NettyNioAsyncHttpClient (the path that carries this defect):

  // OpenSearch: plugins/repository-s3 -> S3AsyncService#buildAsyncClient (simplified)
  NettyNioAsyncHttpClient.Builder http = NettyNioAsyncHttpClient.builder()
      .connectionAcquisitionTimeout(...)
      .maxConcurrency(...)
      .tcpKeepAlive(true)
      .tlsTrustManagersProvider(...);
  
  S3AsyncClient client = S3AsyncClient.builder()
      .httpClient(http.build())
      .region(region)
      .credentialsProvider(...)
      .overrideConfiguration(o -> o.retryPolicy(...).addMetricPublisher(...))
      .build();

An opt-in CRT-client wiring exists (buildAsyncCrtHttpClient) which bypasses this path, but the default (and this deployment) uses netty-nio.

Upload code path (OpenSearch → SDK)

Segment uploads flow through OpenSearch's AsyncTransferManager:

• Single-part: AsyncTransferManager.uploadInOneChunk → s3AsyncClient.putObject(req, body)
• Multipart: AsyncTransferManager.doUploadInParts → AsyncPartsHandler.uploadPart → s3AsyncClient.uploadPart(req, body)

Both build the request body the same way, on a background stream-read executor:

  AsyncRequestBody body = AsyncRequestBody.fromInputStream(
          inputStream,
          inputStreamContainer.getContentLength(),
          streamReadExecutor);
  putObjectRespFuture = s3AsyncClient.putObject(putObjectRequest, body);   // or .uploadPart(...)  

Every one of these calls is a PutObjectRequest / UploadPartRequest, so the S3 client's StreamingRequestInterceptor unconditionally adds Expect: 100-continue on every upload — i.e. every upload takes the deferred-subscription path where the reported NPE lives.

Impact

RetryableStageHelper.retryPolicyDisallowedRetryException classifies the NPE as non-retryable, so each affected PutObject/UploadPart fails on first attempt. OpenSearch surfaces it as a failed segment upload, which feeds directly into the remote-store upload backlog and amplifies the very lag pressure that produced the writability churn — a small feedback loop.

[bhoradc]

Hi @Adiiigo

Thanks for the detailed report. I was able to reproduce the issue at my end. We'll look into this further.

For anyone else hitting this in the meantime, S3Configuration.builder().expectContinueEnabled(false).build() avoids the affected code path (as rightly pointed above).

[gbbafna]

@bhoradc : Is there any downside of using S3Configuration.builder().expectContinueEnabled(false).build() ? Would there be some reason as to why it is not a default .

[bhoradc]

HI @gbbafna - The downside of disabling it is that the client will send the full body immediately without waiting for the server to confirm. If the server ends up rejecting the request, the entire upload has already been streamed for nothing.

And I believe it defaults to enabled as S3 recommends using 100-continue for PUT operations to avoid sending the request body twice on redirects or rejections.

[Adiiigo]

Hi @bhoradc , thanks for explaining the trade-off.

In our case the uploads are large segment files, so waiting for the server's acknowledgement before streaming the body is exactly the behavior we want to keep and disabling expectContinueEnabled wont be a good fit for us.

Is there any other client-side mitigation in the meantime, or is the null/state guard in HandlerSubscriber the only real path forward (in which case we'll wait for the fix)?. Happy to help validate a patch.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[bhoradc]

@Adiiigo - Fix is included in version 2.54.2. Thanks again for the detailed report and analysis.