Clusters

[samyfodil]

To support computing that requires data persistence or consistent orchestration (e.g., running a Postgres container), I propose the concept of Clusters.

Permanent Clusters

These are clusters established and maintained by nodes. They can be dynamically matched based on pre-configured conditions or custom code. The orchestration of persistent resources will be matched and routed to an appropriate permanent cluster.

Transient (Virtual) Clusters

These clusters are created on the fly to accommodate a project resource called a "Cluster." Virtual clusters can potentially be matched with permanent clusters if the conditions align, which helps reduce provisioning time.

Consensus Mechanisms

The possible mechanisms to consider include Raft, PBFT (Practical Byzantine Fault Tolerance), and CRDT (Conflict-free Replicated Data Types).

[samyfodil]

Permanent Clusters should be configured as follow:

• Name
• Type (raft, etc.)
• Nodes (multi-addresses)
• Discover/Advertise (true/false)
• Max number of (voting) Nodes
• Instance name (i.e. region-1)
• Attributes (tags, ...etc) - used to match clusters. for example it could be a region tag
• Cluster shared secret

Full cluster name will be /@. Example: main/compute@dal

A node can be part of many clusters.