feat: Add presigned URL download support to S3AsyncClient and S3 Transfer Manager (#7115)

* Adding pre-signed URL operation core API and Get Object Request Pojo (#6216)

* Add presigned URL request models and tests

* Pre-signed URL API and Get Object Request definitions

* JavaDoc changes, String to URL change

* Delete services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/presignedurl/DefaultPresignedUrlManager.java

* Delete services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/presignedurl/PresignedUrlGetObjectRequestMarshaller.java

* Delete services/s3/src/test/java/software/amazon/awssdk/services/s3/presignedurl/model/PresignedUrlTest.java

* Format PresignedUrlGetObjectRequestWrapper.java

* Format PresignedUrlManager.java

* Format PresignedUrlGetObjectRequest.java

* Format PresignedUrlGetObjectRequestWrapperTest.java

* Format PresignedUrlGetObjectRequestTest.java

* sonarcube issues fixed

* Delete .idea/codeStyles/Project.xml

* added consumer builder, javadoc for builder, assert range value test

* removing codestle from gitignore

* Add custom marshaller for pre-signed URL GET operations (#6231)

* presigned url get object request custom marshaller

* used protocolFactory and fixed sonar cube issues

* removed unused test method, removed overriden method

* Add DefaultPresignedUrlManager getObject implementation for S3 pre-signed URL (#6241)

* getobject implementation and tests

* corrected the getObject params in test:

* fixed build failure-dependency issue with Retrystrategy in test

* fixed sonarcube issues

* addressed sonarcube issue

* edited javadocs, added invalidurl test

* Add AsyncPresignedUrlManager for S3 GetObject operations with presigned URLs (#6255)

* async presignedurlmanager api and implementation

* Refactor DefaultAsyncPresignedUrlManager: move config to constructor, declare variables inline

* change operation name

* Removed presignedurl api and implementation for sync client (#6270)

* Add PresignedUrlManager support to code generated S3AsyncClient (#6269)

* async s3 client codegen changes

* minimised service-2.json

* Modify Operation Name for Presigned URL, add  Null Check for ResolvedEndpoint, enhance AsyncPresignedUrlManager API (#6275)

* modify operation name, null check for resolvedendpoint, add getObject APIs

* added null check for resolvedendpointHeader, doc updates, test additions

* fixed checkstyle issue

* Rename PresignedUrlManager to PresignedUrlExtension following API review (#6286)

* surface api name changes

* rename files, remove consumer builder from downloadrequest

* some nitpicks

* renamed variables

* Add AsyncPresignedUrlManager integration tests (#6290)

* Amazon DataZone Update: This release adds support for 1) highlighting relevant text in returned results for Search and SearchListings APIs and 2) returning aggregated counts of values for specified attributes for SearchListings API.

* Amazon Omics Update: Add Git integration and README support for HealthOmics workflows

* Updated endpoints.json and partitions.json.

* Release 2.32.8. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.32.9-SNAPSHOT

* Amazon AppIntegrations Service Update: Amazon AppIntegrations introduces new configuration capabilities to enable customers to manage iframe permissions, control application refresh behavior (per contact or per browser/cross-contact), and run background applications (service).

* AWS Key Management Service Update: Doc only update: fixed grammatical errors.

* AWS Config Update: Documentation improvements have been made to the EvaluationModel and DescribeConfigurationRecorders APIs.

* AWS End User Messaging Social Update: This release introduces new WhatsApp template management APIs that enable customers to programmatically create and submit templates for approval, monitor approval status, and manage the complete template lifecycle

* AWS Budgets Update: Adds IPv6 and PrivateLink support for AWS Budgets in IAD.

* Amazon Simple Queue Service Update: Documentation updates for Amazon SQS fair queues feature.

* AWS Elemental MediaPackage v2 Update: This release adds support for specifying a preferred input for channels using CMAF ingest.

* Amazon Elastic Compute Cloud Update: Transit Gateway native integration with AWS Network Firewall. Adding new enum value for the new Transit Gateway Attachment type.

* Release 2.32.9. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.32.10-SNAPSHOT

* AWS IoT SiteWise Update: Add support for native anomaly detection in IoT SiteWise using new Computation Model APIs

* Amazon OpenSearch Ingestion Update: Add Pipeline Role Arn as an optional parameter to the create / update pipeline APIs as an alternative to passing in the pipeline configuration body

* AWS Direct Connect Update: Enable MACSec support and features on Interconnects.

* Updated endpoints.json and partitions.json.

* Release 2.32.10. Updated CHANGELOG.md, README.md and all pom.xml.

* Update to next snapshot version: 2.32.11-SNAPSHOT

* Rebase #6269: Update PresignedUrlManager to PresignedUrlExtension naming

* Integ tests

* Refactor AsyncPresignedUrlManager integration tests to extend S3IntegrationTestBase

* updated test method names to follow testing guidelines

* simplified tests

* added checksum test for largeobjects

* rename in test files, address nit comment, fix rebase issues

---------

Co-authored-by: AWS <>

* Add multipart presigned URL download support to MultipartS3AsyncClient (#6318)

* Add multipart presigned URL download support with range-based routing

* Renamed AsyncPresignedUrlExtension variable

* Commented the failing test with TODO

* Add S3 Transfer Manager presigned URL download request models (#6356)

* Transfer Manager presigned URL download request models

* Fixed checkstyle issues

* Add Range-based Multipart download Subscriber for Pre-signed URLs (#6331)

* Range based multipart download subscriber

* Refactor code and add tests

* Fixed sonarcube issue

* Address PR comments: use Long type, inline variables, allow empty files

* Refactored subscriber and tests

* delete testutil

* Added validation tests

* Fixed failing tests for IfMatch header

* Refactor: split validatePartAndRequestMore into separate methods

* Fixed validation tests

* Remove onComplete after subscription cancel

* Make validatePart boolean function

* Fix multipart presigned URL download: parallel support and error propagation

* Fix CodeBuild failure: update codegen file for EndpointResolverInterceptorSpec null check

* Address PR Comments

* Fix check-then-act race condition in parallel subscriber (#6956)

* Fix check-then-act race condition in parallel subscriber

* address comments

* Add presigned URL download support to S3 Transfer Manager (#6951)

* Add presigned URL download support to S3 Transfer Manager

* Add presigned URL download methods to DelegatingS3TransferManager

* Address PR Comments

* Fix presigned url multipart download for empty objects (#6952)

* Add retry tests and onError cancel fix for presigned URL multipart downloads (#6985)

* Fix presigned URL download progress tracking for range requests and add listener tests (#6977)

* fix progress tracking for ranged requests

* Add WireMock listener tests

* Add PresignedFileDownload return type for presigned URL downloads (#7004)

* Add PresignedFileDownload return type for presigned URL downloads

* Address PR Comments

* Fix first part response validation in parallel presigned URL multipart subscriber (#7013)

* Fix first part validation and refactor

* Use String.format for exceptions

* Fix issues in parts calculation (#7026)

* Enable checksum validation for presigned URL downloads (#7035)

* Enable checksum validation for presigned URL downloads

* Add public method to return all checksum algorithms

* assert checksum validation status

* Deprecate IS_DISCOVERED_ENDPOINT, add SKIP_ENDPOINT_RESOLUTION attribute (#7061)

* Fix multipart download response metadata for presigned URL and normal paths (#7077)

* Add response rewrite infrastructure for multipart download metadata

* Fix presigned URL multipart download response metadata and 416 fallback

* Fix normal multipart download response metadata

* Consolidate normal path integration tests with parameterized tests

* Address review: Builder ctor, split(config,mapper) interface method, parameterized tests

* Fix transfer manager presigned serial wrapper to use correct progress wrapper

* Move responseMapper to SplittingTransformerConfiguration and simplify split wiring

* Revert normal object path changes and add presigned custom transformer metadata test

* Use single splitWithResponseRewrite

* Clarify responseMapper Javadoc on SplittingTransformerConfiguration

* Merge branch 'master' into feature/master/pre-signed-url-getobject

* Add changelog entry for presigned URL download feature

* Remove contributor name

* Merge branch 'master' into feature/master/pre-signed-url-getobject

Author: jencymaryjoseph

.changes/next-release/feature-AmazonS3-a6f238d.json

@@ -0,0 +1,6 @@
+{
+    "type": "feature",
+    "category": "Amazon S3",
+    "description": "Add presigned URL download support to S3AsyncClient and S3 Transfer Manager. Customers can now download S3 objects using pre-signed URLs through the SDK's async client pipeline without needing AWS credentials configured.",
+    "contributor": ""
+}

codegen/src/main/java/software/amazon/awssdk/codegen/AddMetadata.java

@@ -62,6 +62,7 @@ public static Metadata constructMetadata(ServiceModel serviceModel,
                 .withDocumentation(serviceModel.getDocumentation())
                 .withServiceAbbreviation(serviceMetadata.getServiceAbbreviation())
                 .withBatchmanagerPackageName(namingStrategy.getBatchManagerPackageName(serviceName))
+                .withPresignedUrlPackageName(namingStrategy.getPresignedUrlPackageName(serviceName))
                 .withServiceFullName(serviceMetadata.getServiceFullName())
                 .withServiceName(serviceName)
                 .withSyncClient(String.format(Constant.SYNC_CLIENT_CLASS_NAME_PATTERN, serviceName))

codegen/src/main/java/software/amazon/awssdk/codegen/internal/Constant.java

@@ -78,6 +78,8 @@ public final class Constant {
 
     public static final String PACKAGE_NAME_BATCHMANAGER_PATTERN = "%s.batchmanager";
 
+    public static final String PACKAGE_NAME_PRESIGNEDURL_PATTERN = "%s.presignedurl";
+
     public static final String PACKAGE_NAME_CUSTOM_AUTH_PATTERN = "%s.auth";
 
     public static final String AUTH_POLICY_ENUM_CLASS_DIR = "software/amazon/awssdk/auth/policy/actions";

codegen/src/main/java/software/amazon/awssdk/codegen/model/config/customization/CustomizationConfig.java

@@ -351,6 +351,11 @@ public class CustomizationConfig {
      */
     private boolean batchManagerSupported;
 
+    /**
+     * A boolean flag to indicate if Presigned URL Extension is supported.
+     */
+    private boolean presignedUrlExtensionSupported;
+
     /**
      * A boolean flag to indicate if the fast unmarshaller code path is enabled.
      */
@@ -951,6 +956,14 @@ public void setBatchManagerSupported(boolean batchManagerSupported) {
         this.batchManagerSupported = batchManagerSupported;
     }
 
+    public boolean getPresignedUrlExtensionSupported() {
+        return presignedUrlExtensionSupported;
+    }
+
+    public void setPresignedUrlExtensionSupported(boolean presignedUrlExtensionSupported) {
+        this.presignedUrlExtensionSupported = presignedUrlExtensionSupported;
+    }
+
     public boolean getEnableFastUnmarshaller() {
         return enableFastUnmarshaller;
     }

codegen/src/main/java/software/amazon/awssdk/codegen/model/intermediate/Metadata.java

@@ -73,6 +73,8 @@ public class Metadata {
 
     private String batchManagerPackageName;
 
+    private String presignedUrlPackageName;
+
     private String endpointRulesPackageName;
 
     private String authSchemePackageName;
@@ -815,4 +817,20 @@ public String getFullBatchManagerPackageName() {
         return joinPackageNames(rootPackageName, getBatchManagerPackageName());
     }
 
+    public Metadata withPresignedUrlPackageName(String presignedUrlPackageName) {
+        setPresignedUrlPackageName(presignedUrlPackageName);
+        return this;
+    }
+
+    public String getPresignedUrlPackageName() {
+        return presignedUrlPackageName;
+    }
+
+    public void setPresignedUrlPackageName(String presignedUrlPackageName) {
+        this.presignedUrlPackageName = presignedUrlPackageName;
+    }
+
+    public String getFullPresignedUrlPackageName() {
+        return joinPackageNames(rootPackageName, getPresignedUrlPackageName());
+    }
 }

codegen/src/main/java/software/amazon/awssdk/codegen/naming/DefaultNamingStrategy.java

@@ -220,6 +220,11 @@ public String getJmesPathPackageName(String serviceName) {
     public String getBatchManagerPackageName(String serviceName) {
         return getCustomizedPackageName(concatServiceNameIfShareModel(serviceName), Constant.PACKAGE_NAME_BATCHMANAGER_PATTERN);
     }
+    
+    @Override
+    public String getPresignedUrlPackageName(String serviceName) {
+        return getCustomizedPackageName(concatServiceNameIfShareModel(serviceName), Constant.PACKAGE_NAME_PRESIGNEDURL_PATTERN);
+    }
 
     @Override
     public String getSmokeTestPackageName(String serviceName) {

codegen/src/main/java/software/amazon/awssdk/codegen/naming/NamingStrategy.java

@@ -93,6 +93,11 @@ public interface NamingStrategy {
      * Retrieve the batchManager package name that should be used based on the service name.
      */
     String getBatchManagerPackageName(String serviceName);
+    
+    /**
+     * Retrieve the presignedUrl package name that should be used based on the service name.
+     */
+    String getPresignedUrlPackageName(String serviceName);
 
     /**
      * Retrieve the smote test package name that should be used based on the service name.

codegen/src/main/java/software/amazon/awssdk/codegen/poet/PoetExtension.java

@@ -209,4 +209,8 @@ public ClassName getBatchManagerAsyncInterface() {
         return ClassName.get(model.getMetadata().getFullBatchManagerPackageName(),
                              model.getMetadata().getServiceName() + "AsyncBatchManager");
     }
+    
+    public ClassName getPresignedUrlExtensionAsyncInterface() {
+        return ClassName.get(model.getMetadata().getFullPresignedUrlPackageName(), "AsyncPresignedUrlExtension");
+    }
 }

codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/AsyncClientClass.java

@@ -548,6 +548,26 @@ protected void addBatchManagerMethod(Builder type) {
 
         type.addMethod(batchManager);
     }
+    
+    @Override
+    protected void addPresignedUrlExtensionMethod(Builder type) {
+        ClassName returnType = poetExtensions.getPresignedUrlExtensionAsyncInterface();
+        String internalPresignedUrlPackage = model.getMetadata().getFullInternalPackageName() + ".presignedurl";
+        ClassName implClass = ClassName.get(internalPresignedUrlPackage, "DefaultAsyncPresignedUrlExtension");
+        
+        MethodSpec presignedUrlExtension = MethodSpec.methodBuilder("presignedUrlExtension")
+                                                  .addModifiers(PUBLIC)
+                                                  .addAnnotation(Override.class)
+                                                  .returns(returnType)
+                                                  .addStatement("return new $T(clientHandler,"
+                                                                + " protocolFactory, "
+                                                                + "clientConfiguration,"
+                                                                + " protocolMetadata)",
+                                                                implClass)
+                                                  .build();
+        
+        type.addMethod(presignedUrlExtension);
+    }
 
     private MethodSpec resolveMetricPublishersMethod() {
         String clientConfigName = "clientConfiguration";

codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/AsyncClientInterface.java

@@ -99,6 +99,9 @@ public TypeSpec poetSpec() {
         if (model.getCustomizationConfig().getBatchManagerSupported()) {
             addBatchManagerMethod(result);
         }
+        if (model.getCustomizationConfig().getPresignedUrlExtensionSupported()) {
+            addPresignedUrlExtensionMethod(result);
+        }
         result.addMethod(serviceClientConfigMethod());
         addAdditionalMethods(result);
         addCloseMethod(result);
@@ -174,6 +177,16 @@ protected void addBatchManagerMethod(TypeSpec.Builder type) {
                                                            + "configuration set on this client.", returnType);
         type.addMethod(batchManagerOperationBody(builder).build());
     }
+    
+    protected void addPresignedUrlExtensionMethod(TypeSpec.Builder type) {
+        ClassName returnType = poetExtensions.getPresignedUrlExtensionAsyncInterface();
+        MethodSpec.Builder builder = MethodSpec.methodBuilder("presignedUrlExtension")
+                                               .addModifiers(PUBLIC)
+                                               .returns(returnType)
+                                               .addJavadoc("Creates an instance of {@link $T} object with the "
+                                                           + "configuration set on this client.", returnType);
+        type.addMethod(presignedUrlExtensionOperationBody(builder).build());
+    }
 
     @Override
     public ClassName className() {
@@ -550,5 +563,10 @@ protected MethodSpec.Builder batchManagerOperationBody(MethodSpec.Builder builde
         return builder.addModifiers(DEFAULT, PUBLIC)
                       .addStatement("throw new $T()", UnsupportedOperationException.class);
     }
+    
+    protected MethodSpec.Builder presignedUrlExtensionOperationBody(MethodSpec.Builder builder) {
+        return builder.addModifiers(DEFAULT, PUBLIC)
+                      .addStatement("throw new $T()", UnsupportedOperationException.class);
+    }
 
 }