Intermittent ClientExecutionTimeoutException while accessing S3 (get and put)

[aantony23]

We have been encountering ClientExecutionTimeoutException while accessing AWS resources like S3,SQS and SNS. Our application has had no changes regarding connection set up or account related changes. The only changes have been that we moved to JDK 17 in May2024 while being on the AWS SDK version 1.12.720. We have had no changes in the traffic to the application as well.
Below is our current client configuration. The application has been running fine with these configurations for years. We started seeing this issue only recently.

public final static ClientConfiguration s3_read_put_copy = new ClientConfiguration()
.withMaxErrorRetry(3)
.withRetryPolicy(retryPolicy)
.withConnectionTimeout(1000)
.withClientExecutionTimeout(10000)
.withSocketTimeout(5000) //read timeout
.withMaxConnections(50);
public final static ClientConfiguration sns_publish = new ClientConfiguration()
.withMaxErrorRetry(2)
.withRetryPolicy(retryPolicy)
.withConnectionTimeout(1000)
.withClientExecutionTimeout(10000)
.withSocketTimeout(5000) //read timeout
.withMaxConnections(50);
public final static ClientConfiguration sqs_listener = new ClientConfiguration()
.withMaxErrorRetry(3)
.withRetryPolicy(retryPolicy)
.withConnectionTimeout(1000)
.withMaxConnections(50);

Below are the issues we encounter on a weekly basis . Application may or may not recover from the below. A restart however resolves the issue. We have gone through similar issues faced by others as well to reach a resolution :
#1357

Additional info :
1.This exception does not happen all the time. There is always a gap of approximately 7 to 10 days between occurrences.
2.The initial occurrences does not affect the application since the retries are successful. The occurrences then exponentially rise and then the application hangs - high CPU usage, unsuccessful retries.
3.We also note apache ConnectionClosedException during the same time frame as the ClientExecutionTimeoutException - sample below

2025-03-03 02:16:18.117,"org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 10,389; received: 0)"
2025-03-03 02:16:18.117, at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
2025-03-03 02:16:18.117, at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
2025-03-03 02:16:18.117, at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2025-03-03 02:16:18.119,com.amazonaws.http.timers.client.ClientExecutionTimeoutException: Client execution did not complete before the specified timeout configuration.
2025-03-03 02:16:18.119, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:875)
2025-03-03 02:16:18.119, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
2025-03-03 02:16:18.119, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)

SNS PUBLISH :
Unable to execute HTTP request: The target server failed to respond,}, attempt #1"

S3 GET

2025-03-04 12:07:18.194,com.amazonaws.http.timers.client.ClientExecutionTimeoutException: Client execution did not complete before the specified timeout configuration.
2025-03-04 12:07:18.194, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:875)
2025-03-04 12:07:18.194, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
2025-03-04 12:07:18.194, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)

S3 PUT:

2025-03-04 12:07:09.113,com.amazonaws.http.timers.client.ClientExecutionTimeoutException: Client execution did not complete before the specified timeout configuration.
2025-03-04 12:07:09.113, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:875)
2025-03-04 12:07:09.113, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
2025-03-04 12:07:09.113, at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)

Any input would be greatly appreciated!Thanks!

[debora-ito]

Hi @aantony23

for timeout investigations we recommend to generate the client-side metrics, they'll give you more insights of what's going on with the client. Check the Developer Guide for instructions on how to enable (link) and check this blog post that shows how to interpret the metrics (link).

[aantony23]

Hi @debora-ito
Thanks for the info . We have now enabled metrics and are trying to identify any patterns. Also need some clarification of below :

As per documentation we see that the client execution time is "the amount of time (in milliseconds) to allow the client to complete the execution of an API call. This timeout covers the entire client execution except for marshalling. This includes request handler execution, all HTTP request including retries, unmarshalling, etc. "

Could you please confirm that this time does not include the time it takes to acquire an http connection and that the time to acquire a connection is separately controlled by the ConnectionTimeout setting.

Also, for connection pool issues would the sdk throw a "Unable to execute HTTP request: Timeout waiting for connection from pool" exception?