Question: Different data types

[burdiyan]

Hi, Seph!

I'm curious about the way you plan to implement different data types within diamond (maps, graphs?, etc.), do you already have something specific in mind in this regard? Would you implement an approach similar to the one of Yjs where users can create different data types with different "names" under the main CRDT document, or you have something different in mind?

[josephg]

Hey! Sorry I didn't see this until now - for some reason notifications on issues here have been turned off.

The next step with this is to expand the current list type to support arbitrary objects. It currently only supports lists of characters - ie, plain text. But I want it to be a list of anything.

Then I want to add an implementation of Shelf into diamond-types. Thats a very simple CRDT (simpler than yjs) that should work for tuples, single values, counters and objects (maps). The first version of that will work with dynamic objects, but ideally I'd like to make some derive macros and such as well, for performance and ergonomics in rust.

The trick there is integrating them - I want to be able to have a shelf value, which contains a list, which contains objects inside (powered by shelf). And so on. That'll take some work to figure out the right APIs.

Shelf + the current list CRDT will be enough for most applications, but notably won't be good enough for:

• Rich text annotations. Rich text will probably need to be integrated more deeply with the current list implementation if I want to add full support
• Graphs. I don't think about graphs much. Do you have any opinion about what a good CRDT for graphs would look like?

And out of the box it won't support moving items around inside the structure. Its an open question how that should work.

[nickdbush]

@josephg is this something I can help with?

[josephg]

Sure! Feel like doing a first pass porting Greg Little's Shelf to rust?

https://github.com/dglittle/shelf

[nickdbush]

Making good progress with this, should have a draft soon!

[josephg]

Awesome - looking forward to seeing it!

[nickdbush]

@josephg do you think that the serialized form should be compatible with the shelf JS library? Or something more compatible with the serde library?

And, in terms of logic, the rule that "if JSON(A) > JSON(B), then X is A, else X is B" seems pretty arbitrary to me and could be confusing to debug. But moving away from that would make results different from the shelf library.

[josephg]

I think something more compatible with rust's ecosystem is the right play here. Eventually it would be really cool to have something like serde's macros, where you can define a struct and derive whatever is needed for shelf to do its magic. And for that I think we probably need to separate the data and the versions, rather than marry them together.

And, in terms of logic, the rule that "if JSON(A) > JSON(B), then X is A, else X is B" seems pretty arbitrary to me

It is 100% arbitrary. Shelf just needs an arbitrary ordering such that it has a winner it can pick in the case of conflicts. The rule doesn't apply to objects - just primitives and tuples. But yeah, interleaving the ordering of strings and numbers is gross and I don't like it. I'm happy to change this ordering from how it is in shelf.

[chanced]

Is there anything else that can be done to help object support along? I'd love to use diamond-types with prosemirror.

[josephg]

Object support won't be enough to support prosemirror. We probably want something like peritext - which will be an extention / overlay on top of text documents. If you want to start exploring how to implement that, I'd be happy to talk to you about it. But it won't be my focus for the next few months.

[chanced]

This definitely seems to solve the biggest issue of using Yjs with prosemirror, and that’s history. With yjs+prosemirror, the entire document state is replaced which is problematic for certain use cases.

So far this has been an amazing read. I’m mostly through it but I’ll likely need another pass to comprehend it better.

[chanced]

Wow, that was an incredibly informative rabbit hole. I had never really put much thought into the complexities of async collab.

So it seems that peritext, in its current form, wouldn't be enough either due to blocks. Looking over that thread, it sort of, from my naive perspective, seems like the blocks themselves would need to be responsible for synchronization. Otherwise you risk having a very rigid, top-down implementation which only permits a set of well-known blocks. Yet I have no idea how you would keep the top-level in sync while also translating each nested CRDT into to the editor of choice's transactions.

I've definitely discovered more respect for folks working in this space. This is a tough challenge.