Pekko Cluster Sharding - Race condition

[Susmit07]

In a Cluster Sharding what are probabilities of a file being processed once at a given time across all the pods in the cluster?

Each entity in a sharded cluster is uniquely identified by its entityId as far the documentation mentions.

I have 2 doubts:

• For each file within a HDFS directory, if we provide a unique entity ID by hashing the file path, will it ensure at a given time the file is processed by exactly one actor in the cluster, or we need to have a locking mechanism in place / or implement the file-processing logic in such a way that it ensure to be idempotent
• If there are too many files in the source HDFS directory then there will be good number of actors at a given time will be created - will it add to a performance bottleneck

Considering the above requirements is Cluster Sharding an appropiate technique to adopt for distributed file download (Singleton Cluster mode deployment will ensure exactly one time pull for a file to be processed which is good but problem is it won't scale when directories and files increase, and idle pods in the cluster is another drawback, so we thought of exploring sharding cluster)

Hoping for contributors to provide some insight, grateful !

[Susmit07]

@Roiocam @pjfanning @raboof

I know you people are the contributors to this repository, and shouldn't be tagging you folks, anyone can watch and reply, but sadly no-one does

To be honest we are deciding on the factors before finalising with Pekko so need your inputs as you folks are already an expert

[Susmit07]

Opened discussion here: apache/pekko#1508