VMware Cloud Director Named Independent Disk use in talos

[sinkog]

I have a Talos cluster running under vSphere, where each worker node has a shared disk attached (essentially, all nodes see the same storage, making replication straightforward). I've read a lot and experimented with it, but how would you integrate this into the cluster to ensure Kubernetes handles it properly?

[smira]

I don't have much experience myself with VMWare, so my answer would be generic.

First of all, there's no Talos Linux specific issue as I see it right now. At least the question is not specific to Talos.

If the disk is shared between the nodes, how is it shared? How does it work with many writers?

The answer would depend on the use case, and I think in general shared disks across nodes go against the Kubernetes ideas and design, but ymmv.

[sinkog]

This question might not be entirely specific to Talos, but it’s also not something we can exclude from being relevant. It’s true that sharing the same storage across multiple nodes somewhat contradicts the Kubernetes philosophy. However, why should this necessarily be an issue in this case?

Storage provisioners typically create Persistent Volumes (PVs) through an API, providing the necessary access details to Kubernetes. This ensures a strict 1:1 relationship between the PV and its allocated storage location. This aligns perfectly with Kubernetes logic. However, when the storage class creator (node/pod) has full access to the storage location, it effectively means it can write to it.

If the storage is handled at the block level, the 1:1 relationship exists internally within the storage locations themselves, while the metadata of the storage is managed by two entities (e.g., the provisioner and the node). This is particularly important because during creation and deletion, only the provisioner manages this metadata, while the operating system treats it as an API.

If the system is designed well, the operating system handles these storage units as API-based interfaces. In this case, Talos also becomes relevant, as its minimalist design can effectively support such an architecture.

Additionally, if a file system that supports concurrent access (e.g., GFS2, OCFS2, VMFS) is implemented, shared disks could work under Kubernetes, assuming the infrastructure and the CSI driver are properly configured.

Why is this worth considering? Simply put: curiosity and exploring solutions are fundamental. Such an implementation could also bring practical benefits, like simplifying storage or optimizing performance. While it might not align perfectly with Kubernetes principles, there are scenarios where this approach could be the best or most intriguing option.

[smira]

I'm not sure, there's already https://github.com/kubernetes-sigs/vsphere-csi-driver you can use.

That would be "what everybody does" approach.

[sinkog]

The VMware-CSI is a specific tool that works perfectly (I have currently used it to solve the issue). Its advantage is that it only occupies as much disk space as the PV actually requires. From a DevOps perspective, its configuration is almost entirely automatable. However, its limitation is that in the case of multi-write, the storage server itself does not handle maintaining consistency.

The technology I envision, on the other hand, addresses these issues at the storage level, which theoretically results in lower latency and more stable operation by reducing the number of intermediary services. In other words, it seems to be a more native solution compared to others. This concept aligns with the philosophy of Talos and Kubernetes.

So, if you also don’t have a solution to this issue… If I find one, I’ll get back to you and share it with the community.