Share all hypercore keys via extension over project owner hypercore replication stream

[gmaclennan]

Description

To replicate a hypercore you need to know its key. We share these keys via extension messages over the replication stream for the project owner hypercore. Since every project member has the project key, which is the key for this hypercore, they can always send and receive these extension messages.

Currently this sharing is only implemented for hypercores in the "auth" namespace. The initial idea was that other keys would be shared via the coreOwnership records: a client would replicate all the auth cores, then read the core ownership records to get the keys of the other cores to sync. This approach is probably the most efficient, because it minimizes the data that needs to be sent (the coreOwnership records are being sent anyway), but it will add complexity to the code.

Proposal: share all hypercore keys over extension messages, the same as we currently do with auth cores. The overhead in terms of data that needs to be sent/received is minimal: approx. 32-bytes for each hypercore. For a new device joining a project with 50 members, e.g. 250 hypercores, this would require <10kb of data. This seems minimal in the bigger scheme of things and the amount of data that is shared.

Tasks

• Add fields to the ProjectExtension protobuf for config, data, blob and blobIndex keys.
• Update extension handling in CoreManager to send keys from any namespace and add cores from all namespaces.

[gmaclennan]

I was thinking about this more, and it introduces a small security risk: we will add cores from peer with a project key. We won't actually download data to those cores from that peer if they are not part of a project (e.g. that peer does not have a capability record), but there could be a "bad" peer that has the project key, is part of the project, and then starts injecting a lot of other cores into the project and they just get added.

A more robust solution is what we were initially planning to do: only inject auth cores, then check capability records and core ownership records, and only add cores that belong to devices that are actually part of the project. I have opened an issue for this: #268.

I think it's ok to go ahead with this solution temporarily - it allows us to move ahead with sync and the change won't change the API, but it will change the protocol, so ideally we make the change in #268 before we launch the MVP to avoid having to maintain backwards compatibility with the "less secure" key sharing protocol.