Skip to content

Regression in 2.52.0: hangs from certain exceptions, rather than errors propagating to callers #7313

Description

@eager-signal

Describe the bug

We have an integration test that shuts down the DynamoDBLocal server and then makes a call that is expected to fail. Starting with 2.52.0, that test times out, rather than getting the expected error. It appears to have been a subtle case not handled by #7197.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

Scenarios like a remote server being unavailable should be visible to callers via standard exception mechanisms.

Current Behavior

Certain errors are not propagated back to callers.

Reproduction Steps

/**
 * Regression test for a hang introduced in 2.52.0 by "Fix adaptive retries implementation" (#7197).
 *
 * <p>{@code AsyncRetryableStage.maybeAttemptExecute} used to run synchronously, so anything it threw was caught by
 * {@code maybeRetryExecute}'s {@code catch (Throwable t) { future.completeExceptionally(t); }}. In 2.52.0 the body moved
 * inside a {@code whenComplete} callback, and {@code whenComplete} does not propagate to its caller -- the throwable is
 * captured into the returned (discarded) stage. That catch block is now unreachable, so the response future is never
 * completed and the caller waits forever.
 *
 * <p>The throw used here is the {@code RejectedExecutionException} from
 * {@code scheduledExecutor.schedule(() -> attemptExecute(future), ...)}, which is what a shut-down SDK scheduled
 * executor produces (e.g. a request racing client {@code close()}, or a caller-supplied
 * {@code ClientOverrideConfiguration#scheduledExecutorService} that rejects).
 *
 * <p>Expected: the future completes exceptionally (passes on 2.51.4).
 * Actual on 2.52.0+: the future never completes, and {@code get} times out.
 */
public class AsyncRetryableStageRejectedScheduleTest {

    @Test
    void execute_retryBackoffScheduleRejected_completesFutureExceptionally() throws Exception {
        // A scheduled executor that rejects everything, as a shut-down one does.
        ScheduledExecutorService rejectingExecutor = Executors.newScheduledThreadPool(1);
        rejectingExecutor.shutdown();

        RetryToken token = mock(RetryToken.class);
        RetryStrategy retryStrategy = mock(RetryStrategy.class);

        // First attempt: allowed immediately, so it runs inline and never touches the scheduler.
        when(retryStrategy.acquireInitialTokenAsync(any()))
            .thenReturn(CompletableFuture.completedFuture(AcquireInitialTokenResponse.create(token, Duration.ZERO)));

        // The attempt failed and the strategy says "retry" -- which makes the stage schedule the next attempt.
        when(retryStrategy.refreshRetryTokenAsync(any()))
            .thenReturn(CompletableFuture.completedFuture(RefreshRetryTokenResponse.create(token, Duration.ZERO)));

        RequestPipeline<SdkHttpFullRequest, CompletableFuture<Response<SdkResponse>>> delegate = mock(RequestPipeline.class);
        when(delegate.execute(any(), any()))
            .thenReturn(CompletableFutureUtils.failedFuture(new IOException("connection reset")));

        AsyncRetryableStage<SdkResponse> stage =
            new AsyncRetryableStage<>(mock(TransformingAsyncResponseHandler.class),
                                      HttpClientDependencies.builder()
                                                            .clientConfiguration(clientConfig(retryStrategy,
                                                                                              rejectingExecutor))
                                                            .build(),
                                      delegate);

        SdkHttpFullRequest request = SdkHttpFullRequest.builder()
                                                      .method(SdkHttpMethod.GET)
                                                      .uri(URI.create("https://my-service.amazonaws.com"))
                                                      .build();

        CompletableFuture<Response<SdkResponse>> executeFuture = stage.execute(request, requestExecutionContext());

        // On 2.52.0+ this reports TimeoutException instead: the future is never completed at all.
        ExecutionException e = assertThrows(ExecutionException.class, () -> executeFuture.get(2, TimeUnit.SECONDS));
        assertInstanceOf(RejectedExecutionException.class, e.getCause());
    }

    private static SdkClientConfiguration clientConfig(RetryStrategy retryStrategy, ScheduledExecutorService scheduler) {
        return SdkClientConfiguration.builder()
                                     .option(SdkClientOption.RETRY_STRATEGY, retryStrategy)
                                     .option(SdkClientOption.SCHEDULED_EXECUTOR_SERVICE, scheduler)
                                     .build();
    }

    private static RequestExecutionContext requestExecutionContext() {
        ExecutionAttributes attrs =
            ExecutionAttributes.builder()
                               .put(SdkInternalExecutionAttribute.NEW_RETRIES_2026_ENABLED, true)
                               .build();

        return RequestExecutionContext.builder()
                                      .originalRequest(mock(SdkRequest.class))
                                      .executionContext(ExecutionContext.builder()
                                                                        .metricCollector(NoOpMetricCollector.create())
                                                                        .executionAttributes(attrs)
                                                                        .build())
                                      .build();
    }
}

Possible Solution

In AsyncRetryableStage#attemptFirstExecute and AsyncRetryableStage#maybeAttemptExecute, add a catch to the logic inside retryableStageHelper.tryRefreshTokenAsync(suggestedDelay()).whenComplete( … ) that calls future.completeExceptionally(t)

Additional Information/Context

No response

AWS Java SDK version used

2.52.0

JDK version used

25.0.3

Operating System and version

Linux

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.p1This is a high priority issuepotential-regressionMarking this issue as a potential regression to be checked by team member

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions