S3 Adapter

[nguyenm100]

Hey folks, looking to set this up in HA mode and the suggestion is to use S3 or some S3 API Compatible service. We can't access S3 from work and the docs mentioned Cassandra which is possible for us, but the only S3 Cassandra Adapter I'm seeing in the wild is deprecated.

Anyone running rdf-delta in HA mode in production and, if so, what is your setup? If not S3/Cassandra, any other HA success stories to share?

TIA

[afs]

Not answer.

It would be good to hear of usages with S3 compatible network APIs.

The storage of blobs (the patches) is intentionally simple (it is interface PatchStorage) so implementing for a different object stores should be quite forward.

[afs]

Not specifically. It does depend on what you want from Delta.

If you want "rebuild from the beginning " then deleting anything will break that.

If you want an FT cluster, then a periodic clear-out by getting the system into sync, and using the database at the time as a new starting point for all servers means you can clear zk and S3. There isn't specific support for this but a process like:

• Wait until no updates.
• Poke each Fuseki to get it to sync.
• Backup the database from one of the Fuseki servers.
• Stop the cluster
• Clear out ZK and S3.
• Restart: empty logs, new starting point.

[nguyenm100]

thanks for all your help andy. while I understand the logical simplicity of separating out zk with storage, is there anything inherently wrong with storing the indexes in the central store itself? seems like less moving parts wrt being able to remove zk.

on top of which, I saw that jena, at one time, supported mysql but it was then removed. any history there that you know of? I'm just looking for the simplest path to a HA/Scalable system and while I get the modular approach with rdf-delta, there are a few things to choreograph that adds to the risk/complexity.

[afs]

The simplest arrangement is a single patch server, using file storage and make the file storage robust (e.g. a disk array with redundancy). This single patch server starts up very quickly (a few seconds) so while it is a SPOF it is simple and restartable - it is a tradeoff for simplicity.

Jena over SQL was two systems - the second SDB was for SPARQL .

SPARQL externally over SQL is slow, and loading MySQL does not scale very well. The single triple table usage pattern is not the focus of SQL engines and they do badly on them (optimizer, loading). Also SQL evaluation semantics is not the same as XSD F&O which is what SPARQL uses for FILTER expressions. If you can get inside the SQL engine, then there are possibilities; Oracle (and others) do this. But without getting inside the SQL engine, it's limited.

[sempike]

For my HA solution, I was limited to use Azure Blob Storage to store Patch data. And Azure Blob Storage does not have an S3 compatible API.

I ended up to use MinIO in API Gateway mode that adds Amazon S3 compatibility to the Azure Blob Storage. https://docs.min.io/docs/minio-gateway-for-azure.html

MinIO in Gateway mode acts as a proxy between the S3 API Client and the backend Storage. And that might solves your problem as well because it supports other backend storage types: HDFS, NAS. Both should be fine even behind a corporate firewall.

edit:
I was so focused on my setup that I forgot to mention the obvious. Minio is an S3 compatible object storage out of the box. You don't need to switch it to gateway mode just create a bucket on your local filesystem and and access key to it.

[afs]

Thanks for the update. Good to know that MinIO works and provides a solution that can be used over file storage because not everything is "cloud" (yet). I'd like to make Delta simpler to use and operate in different environments.

The issue with MinIO is the license which is AGPL3 or commercial. Including in the Delta development would require AGPL3 as the license of the Delta "product" [*].

It could be done as a separate project that provides the PatchStorage of the Delta framework because Delta's license (Apache License v2) can be incorporated into an AGPL3 project/product.

[*] "Product" not in the commercial sense - "product" as in the items produced.

[afs]

User report: MinIO in API Gateway mode that adds Amazon S3 compatibility to the Azure Blob Storage.