V4 Development Tracker

[normj]

V4 Development Tracker

This issue is for tracking the changes being made for V4 of the AWS SDK for .NET. Development of V4 is happening in the v4-development branch in this repository. V4 is an evolutionary major version change with minimal breaking changes so application code using the AWS SDK can upgrade to V4 with low effort.

Update August 15, 2024

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

Breaking changes

The list of breaking changes made in V4.

• .NET Framework 3.5 target has been removed.
• .NET Framework 4.5 target has been updated to 4.7.2
• Properties using value types on classes used for making requests and responses have been changed to use nullable value types. Essentially properties of type bool are changed to bool?.
• Properties using collections on classes used for making requests and responses will now default to null. The V3 behavior of initializing collections can be restored by setting the Amazon.AWSConfigs.InitializeCollections to true. This property also exists in V3 for users that want to try this behavior change before upgrading to V4.
• S3 service clients configured for us-east-1 will no longer be able to access buckets in other regions. Buckets musts be accessed with S3 service clients configured for the region the bucket is in. This change avoids surprises in applications using the S3 service client when under the covers it makes multiple AWS requests to access buckets that are not in service clients configured us-east-1 region.
• The IAM action identifiers have been removed from the Amazon.Auth.AccessControlPolicy.ActionIdentifiers namespace. These identifiers were previously marked as obsolete due to be being out of date with no mechanism to keep them up to date. Code using these ActionIdentifiers should be upgrade to use the string value of the IAM action name.
• The S3 encryption client has been removed from the AWSSDK.S3 package. This client had been marked as obsolete with the encryption client moved to its separate package Amazon.Extensions.S3.Encryption package. The following migration guide to transition to the separate package. https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/s3-encryption-migration.html
• The DetermineServiceURL method on the ClientConfig, the base class of service client configs like AmazonS3Config, was removed. This had been marked obsolete and was tied to an internal endpoint resolution system that was removed in V4. Application code calling this method should switch to using the DetermineServiceOperationEndpoint method on the service client.
• When using credential providers that rely on Amazon Security Token Service (STS) the calls will always use the regional endpoint. V3 by default used us-east-1 regardless of the configured region when running in the public partition. As part of this change the StsRegionalEndpointsValue enum was removed. Any code using that enum should be removed.
• On service client configs the RetryMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• On service client configs the DefaultConfigurationMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• DefaultClientConfig in AWSSDK.Extensions.NETCore.Setup no longer extends from service client config base class ClientConfig. The service client config properties have been replicated on DefaultClientConfig using nullable value types to allow detecting when a value has been set on DefaultClientConfig when copying the values to the service client config being created for the service client.
• CloudFront's CookieSigner and UrlSigner have been moved to a separate extension package called AWSSDK.Extensions.CloudFront.Signers to support SSL3 and take a dependency on BouncyCastle.Cryptography.
• EC2's GetDecryptedPassword has been moved to a separate package called AWSSDK.Extensions.EC2.DecryptPassword to support OpenSSL 3 and take a dependency on BouncyCastle.Cryptography.
• The SDK's source copy of BouncyCastle has been removed.

Other noteworthy changes

DynamoDB Changes

In V4 we're addressing issues around testability, improving configuring the high-level Table and DynamoDBContext clients, and potentially additional fixes that require breaking changes.

• In DynamoDB's high-level programming model, initializing the Table with a mocked IAmazonDynamoDB now returns an InvalidOperationException instead of a NullReferenceException . Async Table methods should now work with a mocked client but you may still see exceptions when calling sync methods from .NET/Core/Standard. See PR #3388, this mostly addresses DynamoDB Table misrepresents dependency on IAmazonDynamoDB #1589.
• In DynamoDB's object persistence programming model, we've separated the shared DynamoDBOperationConfig into new operation-specific objects (SaveConfig, LoadConfig, QueryConfig, etc.). The shared config has grown overtime, and contains properties that don't apply to every operation which can lead to confusion. we've marked the methods that take DynamoDBOperationConfig as obsolete. We won't remove them from V4 yet, though may do so in a future version. See PR #3421
• In DynamoDB's object persistence programming model, we've removed MetadataCachingMode and DisableFetchingTableMetadata from DynamoDBOperationConfig, and did not include these on the new operation-specific configs that were introduced above. These are table-level settings that should be specified on the global AWSConfigsDynamoDB.Context or on DynamoDBContextConfig. See PR #3422
• In DynamoDB's object persistence programming model, DynamoDBOperationConfig no longer inherits from DynamoDBContextConfig. This prevents you from passing a DynamoDBOperationConfig in to the constructor for DynamoDBContext, where some properties on the operation-specific config (such as OverrideTableName) do not apply. See PR #3422

The following changes were made in the SDK. They do not require changes to applications code using the SDK.

• The following dependencies were added to the .NET Framework 4.7.2 and .NET Standard 2.0 targets
  • System.Buffers
  • System.Memory
  • System.Text.Json
• AWSSDK.Extensions.NETCore.Setup has been made Native AOT compatible.
• In .NET Standard 2.0, .NET Core 3.1 and .NET 8 the AssemblyVersion will match the AssemblyFileVersion. In .NET Framework 4.7.2 target AssemblyVersion will continue to use the V3 pattern of locking the version to the first 2 parts of the the AssemblyFileVersion. This is done because .NET Framework treats the AssemblyVersion as part of the identity assembly. This causes assembly collisions and forced recompilation when mixing versions of the SDK.
• The embedded endpoints.json file in AWSSDK.Core was removed. This was a 1 MB json file that was parsed at startup for the SDK to determine the service endpoint for a region. Endpoint resolution has been replaced with a new system that generates rules into each individual service assembly for determining the service endpoints.
• Community PR #3359 Optimize GetExtension execution time on .NET 8. Thanks Steven Weerdenburg
• Community PR #3293 improving AWSSDK.ToHex performance. Thanks Steven Weerdenburg
• Community PR #3292 improving AWSSDK.CopyTo performance. Thanks Daniel Marbach
• Community PR #3324 optimizing the AWS SigV4 signer. Thanks Daniel Marbach
• Community PR #3363 improving AWSSDKUtils.ParameterAsString performance. Thanks Daniel Marbach
• Community PR #3365 improving AWSSDKUtils.DetermineService performance. Thanks Daniel Marbach
• Community PR #3307 Optimizing AWSSDKUtils.UrlEncode performance. Thanks Daniel Marbach
• Community PR #3425 Avoid allocating byte[] when converting MemoryStream to String. Thanks Paulo Morgado
• Community PR #3423 Refactor user agent construction to avoid creating unnecessary string objects. Thanks Paulo Morgado
• Community PR #3439 Refactor JSON request marshaller to avoid creating unnecessary intermediate strings. Thanks Paulo Morgado

[Dreamescaper]

Is there any way to test those packages? Some preview nuget source?

[dscpinheiro]

We'll publish the preview packages to NuGet (e.g. <PackageReference Include="AWSSDK.AccessAnalyzer" Version="4.0.0.0-preview" />), we're just working on some changes to our internal build system at the moment (we'll update this issue as soon as the packages are available).

[normj]

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

[martincostello]

Are you able to provide a preview version of Amazon.AspNetCore.DataProtection.SSM that depends on AWSSDK.Extensions.NETCore.Setup?

Trying to enable native AoT for a project that depends on the SSM package gives a bunch of warnings:

ILC : Trim analysis error IL2026: Microsoft.Extensions.DependencyInjection.ExtensionMethods.PersistKeysToAWSSystemsManager(IDataProtectionBuilder,String,Action`1<PersistOptions>): Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.TryAddAWSService<IAmazonSimpleSystemsManagement>(IServiceCollection,ServiceLifetime)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. 
C:\codebuild\tmp\output\src1982900738\src\aws-sdk-net\sdk\src\Core\Amazon.Runtime\ConstantClass.cs(84): Trim analysis error IL2072: Amazon.Runtime.ConstantClass.Intern(): 'type' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields' in call to 'Amazon.Runtime.ConstantClass.LoadFields(Type)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
  ILC: Method '[AWSSDK.Extensions.NETCore.Setup]Amazon.Extensions.NETCore.Setup.DefaultClientConfig..ctor()' will always throw because: Missing method 'Void Amazon.Runtime.ClientConfig..ctor()'
  ILC: Method '[Amazon.AspNetCore.DataProtection.SSM]Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository+<GetAllElementsAsync>d__9.MoveNext()' will always throw because: Missing method 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'

It's possible some of the above are also coming from internal libraries for transient dependencies that haven't been compiled against v4 yet either.

The app is still blocked from 0 AoT warnings due to a dependency on another third-party library that hasn't published some fixes I made yet, but otherwise this SSM package is the final hurdle.

[normj]

@martincostello I don't have a timeline yet but we would like push out preview versions of all of our high level packages like Amazon.AspNetCore.DataProtection.SSM but we have some internal work first we need to do setting up a dual preview pipeline across our repositories.

[martincostello]

FYI this is the error I get from an .NET 8 app running as a Lambda with the latest version of Amazon.AspNetCore.DataProtection.SSM. It builds fine, but then fails at runtime:

System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information.
 ---> System.MissingMethodException: Method not found: 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'.
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElements()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Serialize(AntiforgeryToken token)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.Serialize(IAntiforgeryFeature antiforgeryFeature)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetAndStoreTokens(HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.AntiforgeryExtensions.GetHtml(IAntiforgery antiforgery, HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.TagHelpers.FormTagHelper.Process(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.TagHelpers.TagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.RunAsync(TagHelperExecutionContext executionContext)
   at AspNetCoreGeneratedDocument.Pages_Shared_SignIn.ExecuteAsync() in /_/src/Codeflowbot/Pages/Shared/SignIn.cshtml:line 13
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at MyApp.MyLoggerMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/CurrentUserLoggerMiddleware.cs:line 20
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at MyApp.MyMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/LambdaLoggerMiddleware.cs:line 38
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2.ProcessRequest(ILambdaContext lambdaContext, Object context, InvokeFeatures features, Boolean rethrowUnhandledError)