Scaling modulith without the database becoming a bottleneck

[bones418]

When using the modulith, the @ApplicationModuleListener annotation adds a transaction around every workflow. This holds a DB connection open throughout the entire workflow. Suppose a workflow is as follows...

• Get event with an ID
• Look up the ID and read a large array of hashes into memory
• Loop through array and do lengthy file I/O operations for each element
• After the loop, update DB to say work has been completed
• Send another modulith event

In the above example, the lengthy file I/O operations cause the database, and the maximum number of DB connections configured in your application, to become the bottleneck. Some options to address this problem:

• Increase DB connections allocated to the application. This still succumbs to a noisy neighbor problem if a lot of events with the lengthy file I/O operation needs to be completed
• Use multiple DB connection pools. This could help limit the number of concurrent lengthy file I/O operations and avoids the noisy neighbor problem in the previous approach. However, the database is still a bottleneck that limits the maximum throughput

Feature Request
Can the modulith event_publication table include another column to provide features similar to the SQS visibility timeout concept. The column can be a datetime and the event can only be consumed if the value is null or in the past. For discussion purposes, let's call the proposed column invisible_until. When inserting a new event the invisible_until column would start out as null. When the event gets picked it would update this column to now() + X where X would have to be some number of milliseconds configured by the user for the given event type. If the process finishes it updates the existing completion_date column. If the completion is finished within the allotted X milliseconds then that's the happy path case. If it does not finish within the allotted X milliseconds then a second thread could pick up the event. This runs the risk of multiple threads processing the event at the same time, and applications would need to have resiliency/idempotency built into their workflows, but this should already be the case for any event driven architecture.

Revisiting the workflow from earlier, we can now imagine something like...

• Get event with an ID and update invisibile_until column
• Close the DB transaction
• Look up the ID and read a large array of hashes into memory
• Loop through array and do lengthy file I/O operations for each element
• After the loop, open a DB transaction
• Update DB to say work has been completed
• Send another modulith event

Now we have the best of both worlds. The transactional outbox pattern is preserved while the lengthy file I/O operations are performed without holding onto a DB connection

Thoughts?

[bones418]

@odrotbohm. Do you have any insights here? Is there, or could there be a way for consumers of modulith events to avoid holding a DB transaction open for the entire duration?

[gdupontf]

Maybe you may not want to pursue this further?
I've opened this question because I'm of the opinion that you shouldn't reinvent the wheel.

Using SI you can get ApplicationEvents as messages
https://docs.spring.io/spring-integration/reference/event.html

Which you can store and retrieve using your strategy of choice
https://docs.spring.io/spring-integration/reference/jdbc/message-store.html#jdbc-message-store-channels

I would even argue that a gateway is an even much stronger alternative than events since you can decouple completely from a dependency to the framework.

[odrotbohm]

I think your jumping to specifics of a feature request too quickly. @ApplicationModuleListener is just cosmetics for @Async, @TransactionalEventListener and @Transactional(Propagation.REQUIRES_NEW). The latter because it's higly likely that the listener will have to run in a transaction as well and the usual @Transactional would not work as expected due to the way that @TEL are implemented. In other words, it's a means to avoid non-obvious misapplication of annotations.

That said, if you don't need a transaction in the listener, simply use @Async and @TransactionalEventListener manually and take things from there. This will still kick off the event publication registry but you get full control about the transactional behavior of the listener. Bullets 5 would be covered by the EPR, 6 can be realized by simply returning the event to be published from the listener.

One thing you still seem to have not considered is what happens if one of the latter array elements results in an error and how you'd handle that. That's something crucial to the overall transaction handling for the listener but totally dependent on what you're trying to achieve.

[bones418]

Thank you Oliver. I just noticed your elegant write up here which talked about this exact scenario about a month before I asked this question. https://spring.io/blog/2023/09/22/simplified-event-externalization-with-spring-modulith