Integration with mapeo-core - decoupling sync and discovery

[tomasciccola]

The proposal of mapeo-cloud is to sync with peers over just websockets.
This means that mapeo-core needs a way of getting a stream to replicate with that can be easily hooked to a websocket.
Currently mapeo-core has a sync method which is a noop.
Maybe, ideally, this sync method should just return a duplex stream so that I can bee hook to discovery over any method (hyperswarm, websocket, etc...).
Or maybe there should be a method in mapeo core like getReplicationStream and sync is a wrapper over that.
Also, the order of replication cores (first authcore so that we know what can be replicated, the data, lastly media), is it going to be handled directly in mapeo-core so mapeo-cloud doesn't need to worry?

@gmaclennan , @sethvincent wdyt?

[sethvincent]

Right now sync happens using the replicate method on CoreManager: https://github.com/digidem/mapeo-core-next/blob/main/lib/core-manager/index.js#L281

Currently it takes a noise stream as input, which, yeah, might not be what we need for this use case. Or it might just work by following this example but using @hyperswarm/secret-stream or perhaps another framed stream.

Also, the order of replication cores (first authcore so that we know what can be replicated, the data, lastly media), is it going to be handled directly in mapeo-core so mapeo-cloud doesn't need to worry?

Yes, this is handled by the replication state machine in mapeo-core-next that Gregor worked on: https://github.com/digidem/mapeo-core-next/blob/main/lib/core-manager/replication-state-machine.js

[gmaclennan]

Yes, we can do:

const wss = createWebSocketStream(new WebSocket(url))
const noiseStream = new SecretStream()
noiseStream.pipe(wss).pipe(noiseStream)
const replicateStream = coreManager.replicate(noiseStream)

The encryption from @hyperswarm/secret-stream is superfluous, because we will run the websocket over TLS, but the handshake that validates the public keys of each peer is useful / necessary — we need to be able to trust the server ID to authenticate it for sync beyond auth cores. I opened an issue for turning off encryption for secret-stream.

[tomasciccola]

for this we need this issue merged right ? (so that coreManager is exposed from mapeo-core)

[sethvincent]

@tomasciccola for now replicating a hypercore would be a good enough test. I'll have that replicate method available to use by the end of my day today.

[EvanHahn]

Closing because we've dramatically revamped our cloud server work.