WebTorrent support

[rektide]

Hello. It would be wonderful if aquatic could also serve as a tracker to webtorrent folks.

At the moment there is still no BEP to guide this implementation. In the future webtorrent/webtorrent#168 may link to a specification.

[greatest-ape]

Hello @rektide ,

This looks interesting. With some more clarity on the exact protocol (ideally an accepted bittorrent protocol extension), I would be interested in investigating it further.

At first look, my understanding is that a webtorrent tracker acts as a middleman for sending messages between peers, and then the peers commence some kind of file transfer connection without the involvement of the tracker? Is this correct?

Is there any point to intermingling peers using different protocols, ie udp and ws? Can they communicate with each other? This impacts whether it would make sense to attempt to integrate the various protocols or run the ws tracker essentially as an independent system from the udp tracker, possibly even with its own binary (especially if bringing in the whole tokio stack.)

Are there any deficiencies in the existing (NodeJS) implementation? I assume performance (or possibly correctness) would be the motivation for a Rust implementation?

Some info for future reference: there seems to be multiple websocket implementations in Rust. These look interesting:

• https://crates.io/crates/ws
• https://crates.io/crates/tungstenite / https://crates.io/crates/tokio-tungstenite

Some info on alternatives: https://github.com/flosse/rust-web-framework-comparison#websocket-libraries
WebTorrent FAQ: https://github.com/webtorrent/webtorrent/blob/master/docs/faq.md

[rektide]

Hello! Thanks for the speedy & very well considered reply! What a delight to be so well met with such a broad ask. I'm going to try to provide answers as best I can, but I'd say they really are more like educated guesses based off a somewhat weak understanding.

a webtorrent tracker acts as a middleman for sending messages between peers, and then the peers commence some kind of file transfer connection without the involvement of the tracker?

Yup, right on. Just like regular BitTorrent, where a peer can announce content and scrape to find people offering content, in WebTorrent a peer talks to a tracker to announce and scrape, again registering itself or finding those registered as having a bit of content (identified by info_hash).

We can also see in that websocket example, that the webtorrent announce/scrape commands are JSON encoded messages with a couple fields (action, info_hash, & some message dependent fields).

Is there any point to intermingling peers using different protocols, ie udp and ws?

This is an interesting question. What I'm seeing again from checking out WebTorrent's bittorrent-tracker is that it maintains a global list of peers, but will not intermix between ws and (udp or http).

I misread it at first & was assuming it might, & leaving clients to figure out which peers it can connect to however it might, but if there's only a handful of peers on the client's protocol, it might take quite a while to find another using the same protocol that it can connect to. But it would be neat, in cases where one was using something like webtorrent-hybrid, where the peer has multiple connectivity options built in, and some protocols might have connectivity advantage that would let them make the connection that the other protocol lacked (webrtc having stun/turn for example). I suppose in that case the hybrid peer/client could scrape the tracker via both protocols. :)

Are there any deficiencies in the existing (NodeJS) implementation? I assume performance (or possibly correctness) would be the motivation for a Rust implementation?

Performance, security, & correctness are my main desires, yes. There has been much discussion about moving to a more efficient C++ based websocket implementation, uws. Two interesting enhancements I'd mention are pruning old torrents/swarm and a discussion around enhancing client privacy, but I wouldn't call these deficiencies.

Cheers. Thanks for the great work already.

[greatest-ape]

Thanks @rektide for the info. I've started to look into it and will mention progress in this issue :)

[greatest-ape]

Hey @rektide,

I've been working quite a bit on this and have implemented a working WebTorrent tracker and merged it into master. There are two executables now, aquatic_udp and aquatic_ws, with the ws suffix of the latter indicating the web socket based WebTorrent protocol. I've successfully used it as the tracker for a file transfer, so all essential parts are up and running, including TLS support for the wss protocol!

The new tracker already supports pruning old connections, torrents and peers (webtorrent/bittorrent-tracker#164). It is likely already quite performant, but I haven't done any profiling. Getting a load test like the one I wrote for the udp protocol up and running would be nice and greatly aid both performance and correctness analysis, but I probably won't have time for that in a while.

One remaining issue is that of whether peers announcing over IPv6 should receive IPv4 peers. They probably shouldn't, and in that case, state should be split based on IPv4/IPv6 (which shouldn't be that difficult). Some other (mostly minor) issues are listed in the TODO file.

[breebee]

@greatest-ape so I see you are a super hero. Thank you on behalf of everyone who dont yet know they need webtorrents on rustoids!

[greatest-ape]

WebTorrent support is pretty much up and running now, so I'm closing this issue.