APis for interacting with Hyperdrive and Hypercore

[RangerMauve]

What

I think it'd be useful to have a set of APIs for interacting with the p2p primitives provided by the extension. I've been building apps for a while and here's some of the stuff I've found useful (after adding it to the dat-sdk)

• Being able to read / write files from hyperdrives
• Creating hyperdrives within an application
• Able to read / write metadata in hypertrie of a hyperdrive
• Being able to listen on updates
• Being able to listen on / emit extension messages
• Being able to listen on network changes (added / removed peers) - important for making use of extensions
• Having 'named' hyperdrives in an application so you don't need to save the key for later use. E.g hyperdrive('My profile')
• Being able to read / write blocks from hypercores
• read/write ranges or streams from hypercores
• all the same networking options / extensions as hyperdrive

Hyperdrive

I think that since we're targeting the web and register protocol handlers, it'd be cool to use fetch() and http-like interfaces for doing all this stuff.

For example fetch('hyper://domain/example.txt') should already be handled for reading. Maybe we can extend that to support uploads like so:

fetch('hyper://domain/example.txt', {
  method: 'PUT',
  body: 'Hello World!'
})

What's cool here is that we can get modern streaming from all this by providing a ReadableStream for the body, or using the ReadableStream returned in the response.

The application specific archive name thing could be useful for creating archives using the namespace method of Corestore. For example you could have something like this.

const ARCHIVE_NAME = 'Main archive'

const {url, version} = (await fetch(`hyper://${ARCHIVE_NAME}/`, {method: 'HEAD'})

alert(`Your archive is ${url} and is at version ${version}`)

The application knows that it can always use Main archive to generate the same archive under the hood, and can get the public URL for sharing with others from there. The namespace could probably calculated with corestore.namespace(origin + name) to make scoping easy.

Changes / Events

Listening for changes / getting updates / extensions is a bit of a question mark for me, since I'm not sure what support the protocol handler code has, but maybe it could be done by opening Websockets, or Server Sent Events. The types of events to subscribe on could be specified in the query string. You could subscribe to multiple extensions by including them multiple times. For example.

const source = new EventSource('hyper://domain?change&extension=example&extension=example2`)
source.onmessage = ({data}) => {
  const {type} = JSON.parse(data)
  if(type === 'change') console.log('change', data.path, data.version)
  if(type === 'extension') console.log('extension message', data.extension, data.message)
}

fetch('hyper://domain`, {
  method: 'post',
  body: 'Example',
  headers: {
    'x-hypercore-protocol-extension': 'example'
  }
})

Hypercore

Similarly to the hyperdrive use case, we could support hypercores. This has been something that's been talked about a lot with regards to brave support and support for apps like cabal.

For example, we could create a hypercore by adding a parameter the first time we reference it.

const {url} = fetch('hyper://my core/`, {
  method: 'HEAD',
  headers: {
    'x-hypercore-protocol-structure': 'core'
  }
})

// Or when we first write to it

const response = await fetch('hyper://my core/`, {
  method: 'POST',
  body: 'Example',
  headers: {
    'x-hypercore-protocol-structure': 'core'
  }
})

const version = response.headers.get('x-hypercore-protocol-version')

const response = await fetch('hyper://my core/0`)

const data = await response.text()

One thing that would be important is to get ranges of data out of a hypercore. Not sure what would be best here. Maybe a Range header? Maybe server-sent events again? Maybe something in the URL like hyper://core/4..20.

Benefits of approach

One of the advantages of doing this over injecting JS APIs is that it'll be available in all worker contexts. For example, you'd be able to easily use this stuff inside service workers or extensions without having to fuss around with injecting APIs into those contexts. This has been a bit of a pain when making Beaker apps since it was hard to get the DatArchive API within iframes or Worker threads. As far as I know every JS context has access to fetch() so code could be reused anywhere the same way.

These APIs are a bit more low level than what Beaker provides, but I think that's a good thing because it gives applications more flexibility. I could also see somebody building up the Beaker specific APIs on top of this without too much hassle.

I could also see a nice JS wrapper over top of the fetch API being made in something like the dat-sdk or other projects. Or even intercepting window.fetch to provide these abilities with a pure JS implementation for cases where dat-webext or an equivalent isn't installed. It being an async API helps a lot in that regard.

This could also be useful as a foundation for a HTTP proxy for interacting with all this Dat stuff. There's been a lot of talk in the community and we could standardize around it. Instead of using the built in hyper:// support, an application could instead use http://localhost:4200/domain/path with all the other HTTP-isms unchanged.

Having hypercore be a first class citizen of the API will also help progress efforts in getting kappa-core based apps working in the browser with proxy-less p2p support. Though I think they'd be happier with having access to the p2p networking in the short term. :P

Also, archive authors could mess with access control using CORS with the index.json file.

TODOs

• Could this apply to the brave integration folks have been talking about?
• Hypercore stuff should be thought about more with regards to ranges

Roadmap

Here's a rough roadmap I made up on the spot which could yield useful APIs for applications at each step

• Read from archives using GET
• Read metadata from archives using HEAD and custom headers
• Get archive creation using POST with names / write files using PUT
• Get basic Hypercore support get / put by index
• Listen for changes
• Extension messages / Peer events
• Hypercore ranges

[RangerMauve]

cc @okdistribute @substack Regarding the brave API we talked about a while ago.

[calm-rad]

Gateway has a similar custom fetch() approach as its primary method of reading/writing P2P content, so I love this and think it would be very cool if more parties were on board!

I currently only have a GET method defined for hyper:// that reads Hyperdrive content like how your proposal describes. Right now it's very minimal for the browser's basic needs, so I'd be happy to synchronize approaches moving forward @RangerMauve

I also think your idea of using corestore's namespace when possible is brilliant, and I think headers are a good solution for easily specifying version and extension in case there isn't a universally agreed upon URL scheme for those things yet. (Small tangent: Gateway currently detects version with the + in the hostname, like Beaker, with a - to separate version numbers if you want a Hypercore range).

One thing I added to my implementation that may be of interest to others is being able to pass an array of requests as an argument, which returns all of the responses when they're done. It deviates more from expected fetch behavior so I'm not sure how viable it is for others, but I figure it doesn't hurt if it doesn't change how singular requests work.

[sammacbeth]

Thanks for this. I'm just wondering why is this approach (patching existing web APIs), preferable to exposing a subset of hyperdrive/hypercore APIs directly? There may be some benefit of familiarity for web devs, but it other cases the proposal departs from expectations from these APIs, for example with HEAD requests.

[RangerMauve]

Yeah, I agree the HEAD stuff is a little weird. Maybe all non-content related things could be POST with JSON responses?

I think the main thing is that not all JS contexts have an easy way to inject APIs. I think iframes and workers can be a bit more difficult to modify from what I recall. Also, having an HTTP based API could be nice for gateways and stuff.

I think wrapping over the HTTP APIs with cleaner JS APIs would be good too, and we could make it easier for them to auto-detect the fetch support or fallback to proxies / in-browser code.

[pfrazee]

Hey sorry for the delay on responding to this.

@RangerMauve A lot of good thinking here and a solution like this could be great.

I figure it's worth asking, would there be any benefit or blocker to using WebSockets instead, perhaps using something like an extended JSON-RPC over CBOR encoding?

The main reason I ask is, it might be a little easier to develop these APIs that way. With HTTP, you have to do the judo of translating between the API and the wire protocol, whereas with WebSockets it's trivial to do.

Another advantage of WebSockets is that they will behave the same way PeerSockets do, so you'll get reuse across every possible channel. That would make it possible for these APIs to communicate over the hyper:// network for remote access.

(As an aside, the wire-protocol would need to extend JSON-RPC to support streams. I'd also suggest a binary encoding such as CBOR so that binary transfers efficiently.)

[RangerMauve]

Yeah, some sort of RPC over Websockets would be cool. Sadly I don't think there's a way to register custom websocket handlers in any browsers. :(

The closest we have is long-polling HTTP connections in the libdweb API. https://github.com/libdweb/libdweb#protocol-api

[RangerMauve]

Starting on getting this into my dweb-browser project to mess around with the UX.

[RangerMauve]

Got the initial draft into dat-fetch and soon into Agregore.

So far I've just added a PUT and DELETE method in addition to the existing GET methods. And some special treatment for directory rendering. Also extended index.json to have a url and writable field injected into the JSON.

@pfrazee How does that stuff look to you with regards to getting it into Beaker? :o

Also @calm-rad How does that look for Gateway?

I don't have time to get stuff into dat-webext at the moment, but it wouldn't be too hard to get dat-fetch in there eventually.

[calm-rad]

@RangerMauve Just took a look at the repo, looks great!

I haven't implemented PUT or DELETE in Gateway yet, so I'll make it a priority to get those in there and have it behave the same way

As for Gateway's GET, the behavior is identical with the only difference being that I separated out things like special index.json handling and resolving html/md files from folder paths into Gateway's higher-level navigation handling, but moving them to fetch wouldn't be an issue :)