User journey moving to EfficientDynamoDb from AWS SDK

[devtekve]

Sorry for the long wall of text, take a cup of coffee and read away. I took the liberty of using ChatGPT to help me write and phrase some of the text below, but it's not something I simply dropped and left. I am writing some background context so that you can understand what I've tried and how I am trying to use EfficientDynamoDb.

Background and Current Practices

With 2-3 years of experience using the AWS SDK, I've become proficient with its features and limitations. My architectural approach employs a modified repository pattern to effectively manage data operations across different application layers, enhancing data abstraction and encapsulation.

Data Model Architecture

I've described a structured approach to data management using three primary models, each serving distinct purposes:

• UserResponseModel: An internal class used to format data for user-facing outputs, ensuring sensitive information remains concealed.
• UserModel: This model circulates within the service layer for business logic operations, protected from direct user access to maintain data integrity.
• UserEntity: Directly interacts with the database and is tailored for data persistence. This model simplifies database transactions and is isolated from other application layers.
• RoleModel Integration: The RoleModel is embedded within the UserEntity as a collection, rather than as a standalone entity. This setup allows for dynamic user role management within our systems.

C# Implementation Examples

Here’s a combined code block showing how these models could be implemented in C#:

// Models defined for different layers of the application with varying data exposure

// UserResponseModel for API output, no sensitive data exposed
internal class UserResponseModel
{
    public string Username { get; set; }
    public int Age { get; set; }
    public List<RoleModel> Roles { get; set; }  // Include roles to display user permissions
}

// UserModel for internal business logic, contains sensitive data
public class UserModel
{
    public string Username { get; set; }
    public string Email { get; set; }
    public string SSN { get; set; }  // Sensitive data, used internally
    public List<RoleModel> Roles { get; set; }
}

// UserEntity for direct database interactions, includes sensitive data
internal class UserEntity
{
    public string UserId { get; set; }
    public string Email { get; set; }
    public string SSN { get; set; }  // Sensitive data part of the database schema
    public List<RoleModel> Roles { get; set; }
}

// RoleModel used across different models to represent user roles
public class RoleModel
{
    public string RoleName { get; set; }
    public string Description { get; set; }
}

Advantages of Modified Repository Pattern

This structure not only ensures a clean separation of concerns across different layers but also supports complex data relationships seamlessly. By defining specific roles for each model, we maintain data security and enhance the system's ability to manage complex interactions efficiently.

The feedback, structured (mostly).

1. Lack of comparative examples against AWS SDK

As an experienced user of the AWS SDK transitioning to EfficientDynamoDb, I appreciate the streamlined and more efficient query capabilities of your library. However, I've noticed a gap in the documentation when it comes to complex operations. For instance, while the transition to EfficientDynamoDb simplified my code, I initially struggled with how to handle sort keys effectively due to a lack of detailed examples.

It would be immensely helpful if the documentation could include more comprehensive, comparative examples that show how to translate complex AWS SDK operations into EfficientDynamoDb operations, especially those involving sort keys and complex filters. Detailed guidance on query configurations and best practices in EfficientDynamoDb would also be invaluable for developers looking to fully utilize your promising tool without resorting to guesswork.

Ideas

• Maybe we can pick and translate examples from https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/dotnetv3/dynamodb ?

One wonderful example of how EfficientDynamoDb makes you enjoy life more

// Implementation with native AWS SDK.
protected async IAsyncEnumerable<TEntity> FindAsync<TKey>(TKey key, object? rangeKey, List<ScanCondition> scanConditions, string? indexName = null, [EnumeratorCancellation] CancellationToken cancellationToken = default)  
{  
    var dynamoDbOperationConfig = new DynamoDBOperationConfig  
    {  
        QueryFilter = scanConditions,  
        IndexName = indexName  
    };  
    var entitiesAsync = rangeKey != null  
        ? context.QueryAsync<TEntity>(key, QueryOperator.Equal, new[] { rangeKey }, dynamoDbOperationConfig)  
        : context.QueryAsync<TEntity>(key, dynamoDbOperationConfig);  
  
    do  
    {  
        var batch = await entitiesAsync.GetNextSetAsync(cancellationToken);  
        foreach (var item in batch)  
            yield return item;  
    } while (!entitiesAsync.IsDone);  
}

// Implementation with EfficientDynamoDb
public IAsyncEnumerable<TEntity> FindAsync(FilterBase keyExpression, IEnumerable<FilterBase> conditions, string? indexName = null)  
{  
	var query = context.Query<TEntity>().WithKeyExpression(keyExpression);;  
	  
	foreach (var condition in conditions)   
	    query = query.WithFilterExpression(condition);  
	  
	if (!string.IsNullOrEmpty(indexName))  
	    query = query.FromIndex(indexName);  
	  
	return query.ToAsyncEnumerable();
}

2. Lack of a compatibility layer with AWS SDK

I recognize that some of the design decisions in EfficientDynamoDb—such as not providing a direct 'drop-in' replacement for the AWS SDK—were likely made with specific intentions, possibly to encourage best practices or optimize performance. However, I believe that implementing a compatibility layer could significantly streamline the adoption process, particularly for teams that are generally more resistant to change.

Introducing such features would not undermine the original goals of EfficientDynamoDb but rather enhance its accessibility and ease of integration. This approach could serve as a bridge, allowing teams to gradually familiarize themselves with EfficientDynamoDb’s unique capabilities while still maintaining their existing workflow during the transition phase.

By lowering the initial barriers to entry, EfficientDynamoDb could appeal to a broader audience and ease the transition process, ultimately facilitating a smoother shift to more efficient and readable code practices. This strategy aligns with the broader goals of enhancing user satisfaction and expanding the user base, making it easier to 'sell' the adoption of EfficientDynamoDb within organizations that may be hesitant about adopting new technologies.

3. Attribute Annotation Requirements in EfficientDynamoDb

One more aspect that I’ve found challenging with EfficientDynamoDb is its approach to property annotations. The library requires developers to use the [DynamoDbProperty(name)] attribute for each property in a class that interacts with DynamoDB. This requirement has posed two main problems for me:

1. Attribute Flexibility: Currently, the attribute necessitates specifying a property name, which could be more streamlined. An overload that automatically derives the property name from the class property it’s mapped to would greatly simplify the implementation. This would allow developers to maintain clean code while still ensuring that properties are correctly mapped to DynamoDB attributes. (See Adding [CallerMemberName] to 3 new overloads for the DynamoDbPropertyAttribute #240 for a proposal to fix)

2. Model Agnosticism: Some of my models, like RoleModel, are designed to remain database-agnostic, adhering to the principle of separation of concerns. However, EfficientDynamoDb’s current approach requires these models to be explicitly annotated with DynamoDB attributes, which introduces unnecessary coupling between the data models and the database implementation details. Ideally, there should be an option to enable automatic mapping of class properties without explicit annotations, with the ability to use something like [DynamoDbIgnore] for exceptions, similar to the AWS SDK.

• While not ideal, this PR brings interface attributes, maybe models and entities can sometimes inherit the same interface? Use interface attributes #241

Being able to toggle between explicit and implicit mapping would provide the flexibility needed in various development scenarios and align better with the principles of clean architecture by reducing the need for database-specific details in domain models

4. Table Name Flexibility

While migrating to EfficientDynamoDb, I've encountered difficulties in managing table names dynamically, a feature crucial for maintaining different environments (development, staging, production) seamlessly. In the AWS SDK, I could easily configure table names at runtime using environment variables / appsettings, like so:

var awsTypeMappings = AWSConfigsDynamoDB.Context.TypeMappings;  
awsTypeMappings[typeof(UserEntity)] = new TypeMapping(typeof(UserEntity), configuration["AWS:TableName"]);  

However, EfficientDynamoDb seems to require hardcoding the table name as an attribute or passing it with every query (although not every query supports this properly at the moment, read below for details), which is less efficient and introduces redundancy in the code. An alternative or enhancement in EfficientDynamoDb that allows for similar flexibility as AWS SDK would greatly streamline development processes and better support multiple environment configurations.

5. Issues encountered with the WithTableName when using QueryAsync<T>

I have encountered an inconsistency with the WithTableName method when attempting to dynamically set table names for queries. For instance, using the following code results in a type compatibility error:

await context.GetItemAsync<TEntity>(id, version, cancellationToken).WithTableName("");

The error suggests a type mismatch with System.Threading.Tasks.Task<TEntity> and EfficientDynamoDb.Operations.Shared.ITableBuilder<System.Threading.Tasks.Task<TEntity>>. I've made a separate bug report for this #235

Final thoughts

As I reflect on my experience with EfficientDynamoDb, I am truly impressed with what it offers. The simplicity, ease of use, and the notable performance enhancements have significantly exceeded my expectations. These aspects make EfficientDynamoDb a compelling choice for my personal projects and a potential candidate for professional applications.

Although there are a few areas I believe could be refined—such as dynamic table naming and more detailed documentation for complex queries—my feedback is meant to be constructive. I see great potential in this tool, and these suggestions are intended to help reach that potential.

I am eager to hear from current contributors about their views on these suggestions. My goal is to foster a constructive dialogue that could lead to further enhancements, making this excellent tool even better.

Thank you to everyone involved in this project for your hard work and dedication. It's exciting to be part of such an innovative effort, and I look forward to seeing how it evolves.

[firenero]

Hey @devtekve, thanks for such a detailed feedback, I really appreciate it! I'd like to go through your points and comment on them.

1. Lack of comparative examples against AWS SDK

That's a valid point. Docs with comparison to the .NET AWS SDK can be really handy and help a lot in learning how to do stuff with EfficientDynamoDb. The link you provided can be quite useful to compile examples. I'm not a fun of how these examples are structured but there is still some code that can be repackaged for better representation.

It's going to take quite a lot time to create examples for all features so I think it makes sense to select the most painful/complicated area and start with that. In my opinion it can be using queries as there are quite a lot operations and some of them can be confusing and/or complicated. Do you have any thoughts on that topic as you did the migration recently?

2. Lack of a compatibility layer with AWS SDK

That's true that we didn't want to support some features because they encourage inefficient operations against the DB. And you have a valid point that a compatibility layer can be helpful during the adoption of EfficientDynamoDb. That said, we do have some compatibility features that are supposed to help with migration from AWS SDK, such as re-using credentials, SaveAsync(...) and DeleteAsync(...) extension methods and so on. There is a compatibility page in docs that goes over that in details.

Are there any particular features that you missed during the migration? There is a chance that we overlooked something and it can be added to the compatibility layer after all.

3. Attribute Annotation Requirements in EfficientDynamoDb

This one is really tricky. On one hand I see what you mean and want and your suggestion make sense. But on the other hand, there are some reasons why things were done this way.

1. Attribute Flexibility. There was a request for the same behavior a couple years ago and I wrote our reasoning there: Support default name of property #180. Shortly speaking, we want to encourage people to think about attribute names because there negative consequences to automatic naming. And since the workaround using nameof(PropertyName) is easy and type-safe, I didn't want to simplify doing dangerous things in the codebase. Arguably, we are not responsible for the choices made on the consumer side but as a library providers, we want to minimize possibility of mistakes. However, I'm not 100% set on this and there is still a chance that I can be convinced if there are good safeguards in place :)
2. Model Agnosticism. This is a feature I wanted to do for a long time but never got a chance to think it through. Removing [DynamoDBIgnore] attribute and not mapping by default is a deliberate choice. We've seen quite a lot of cases when properties were accidentally mapped with .NET SDK because someone forgot to add an attribute. But back in the days I was thinking about what can be done about decoupling models from the DB and thought that some configuration API like this could do the trick:

config.ForEntity<TEntity>()
    .SetProperty(x => x.PartitionKeyProperty, "pk", DynamoDbAttributeType.PartitionKey)
    .SetProperty(x => x.SortKeyProperty, "sk", DynamoDbAttributeType.SortKey)
    .SetProperty(x => x.FirstName, "first_name")
    .SetProperty(x => x.Age, "age");

So the idea that it looks similar to attributes but doesn't require changing the model classes. It's not a final API at this point and I haven't checked what would be an effort of implementing this but it's something I would like to do at some point.

Any thoughts on going this way?

4. Table Name Flexibility

This one can be solved as well by implementing the external configuration from the previous section. I agree that it can be a pain if you're loading table names at runtime. The only workaround I can suggest for now is using an empty name in [DynamoDbTable("")] attribute and set the DynamoDbContextConfig.TableNamePrefix so the full name of the table in runtime. This way you can create a separate DynamoDbContext for every table with different name to avoid name clashing.

I understand that it's not pretty but unfortunately, this is the only way for now.

5. Issues encountered with the WithTableName when using QueryAsync

I already responded to the bug report you created so not going to repeat everything here again.

TL;DR for someone who stumble on this comment looking for answer. You have to use GetItem<T>() method (without Async) that returns a builder to be able to chain it with WithTableName(...).

[devtekve]

Lack of comparative examples against AWS SDK

Do you have any thoughts on that topic as you did the migration recently?

I can share more of my repository code, it's not the best but shows many example of some CRUD operations that people would do. Specially aimed at versioned entities. Maybe we can start from there and make some examples. How does that sound?

---

Lack of a compatibility layer with AWS SDK

Are there any particular features that you missed during the migration? There is a chance that we overlooked something and it can be added to the compatibility layer after all.

Yeah, the main thing is I wasn't looking for "familiar" methods. Quite frankly, the AWS SDK methods really suck, which is why EfficientDynamoDb is such a step up!. However, migrating to EfficientDynamoDb was a "all or nothing" which I can definitely do for a hobby project, as I seem to love the pain of huge changes 🤣 but in my professional day-to-day work I would not be able to pull this, I'd have a ton of resistance.

What I was hoping to have is a compatible drop-in replacement, leveraging the dependency injection. AKA make an implementation of AWS's IDynamoDBContext that uses EfficientDynamoDb methods under the hood. That way I could make the switch by simply changing

// Full AWS  SDK
services.TryAddSingleton<Amazon.DynamoDBv2.DataModel.IDynamoDBContext, Amazon.DynamoDBv2.DataModel.DynamoDBContext>(); 

// Compatibility swap
services.TryAddSingleton<Amazon.DynamoDBv2.DataModel.IDynamoDBContext, EfficientDynamoDb.DynamoDbContext>(); 

Obviously we'd need to make sure users understand that's a stop-gap solution to get jump-started. We can add warnings to the methods. But that would reduce a lot of the friction.

---

Attribute Annotation Requirements in EfficientDynamoDb

Any thoughts on going this way?

I like that approach, I'd drop a WithTableName to that config for a given entity. Similar to AWS SDK's. Though one downside is that it could grow quickly if your entity has many properties.

I still would like to have the option for a "less managed" implementation, where it can auto discover the properties around. I actually have implemented an auto property attribute with the Ignore capability like I mentioned, but that's on a draft MR on my fork. If you wanna take a look, here it is devtekve#1 , I have examples on how it's useful.

---

Table Name Flexibility

I have 2 issues though:

1. That approach would work for me if I had only 1 table, but I actually have 3 dynamo tables
2. I could not make it work, I was getting errors due to the name being null but frankly it could've been that I was making other mistakes while adapting to EfficientDynamoDb so I might need to revisit it.

---

Issues encountered with the WithTableName when using QueryAsync

#235

[firenero]

Sorry @devtekve, I missed your reply here.

I can share more of my repository code, it's not the best but shows many example of some CRUD operations that people would do. Specially aimed at versioned entities. Maybe we can start from there and make some examples. How does that sound?

Yes, I think it can be useful so that I have some base examples that I can convert to EfficientDynamoDb.

migrating to EfficientDynamoDb was a "all or nothing"

For what it worth, you can mark entities with both official SDK and EfficientDynamoDb attributes and then convert it on per query basis. That's what I did in my previous production project. So basically you make your entities to work with both libraries and then inject both contexts in your services. This way you can convert a single query to EfficientDynamoDb without changing all other queries.

AKA make an implementation of AWS's IDynamoDBContext that uses EfficientDynamoDb methods under the hood.

This is an interesting idea. I think using EfficientDynamoDb attributes will still be required but theoretically all the DB access code can remain the same.

Though one downside is that it could grow quickly if your entity has many properties. I still would like to have the option for a "less managed" implementation, where it can auto discover the properties around.

I think it's possible to have some .AutoDetectProperties() method as well in this external config approach. However, since there is an initiative to add support for native AOT #245, properties discovery might be tricky to implement. But maybe codegen approach to that will be similar to codegen required for current attributes approach.