Discussion: composable dialing

[rphmeier]

Dialing hasn't really been sorted out yet. It's easy to do for simple multiaddrs, but for complex ones, particularly those with multiple hops, things get much fuzzier.

Imagine we have registered two forms of transport: TCP and p2p-circuit, which is used to relay connections.

One example which is difficult to make work is something like
ip4/1.2.3.4/tcp/8888/p2p-circuit/p2p/DestPeer

(circuit relay multiaddr format is <relay-peer>/p2p-circuit/<remote-peer>)

This address, when used for dialing, says "Connect to the peer DestPeer on any available address, through a relay node we will connect to via tcp on port 8888 over the ipv4 address 1.2.3.4"

ip4/4.5.6.7/tcp/30042/p2p-circuit/ip4/1.2.3.4/tcp/8888/p2p-circuit/p2p/DestPeer

Here's a double-hop relay:
Connect to the peer DestPeer through the relay on tcp://1.2.3.4:8888, which we will connect to through the relay on tcp://4.5.6.7:30042.

We'll need to require dialers to handle the whole address, and give them a closure or similar required to instantiate connections to different encapsulated multi-addresses.

So the logic for a circuit dial would (I think?) be

1. find the left-most mention of p2p-circuit.
2. split this multiaddr along that mention into the relay and remote addresses.
3. context.dial(relay).and_then(move |conn| RelayProtocol::dial_through(remote, conn))

This reads as: dial to the relay node, and then when we get a connection to it, attempt to dial the rest of the address through it.

One interesting thing to note is that our individual transport Conns to peers can actually be instances of a Socket being multiplexed over a different Conn to the same peer.

Even in the multi-hop case, we only open one logical connection, through TCP. this means that we're limited to basically one layer of virtual calls. I am not sure this holds true in general, but it depends how fine-grained we want to go.

As in, if we have the multi-addr ip4/1.1.1.1/tcp/20202/ws, we can either have the ws transport wrap a dialled tcp connection, or have the ws transport handle the tcp connection internally. The first option is ideal for "composability" because it means that the ws transport can handle any kind of wire beneath it, but then you can get into the situation of multiple levels of boxed trait objects, which are bad for performance in a multitude of ways.

It may not really be avoidable though, as long as we ensure the cost of connecting/communicating with a peer remains roughly proportional to how complicated the address for that peer is: simple peers will have fewer layers of indirection. More complex peer connections pay a cost for their complexity.

[rphmeier]

I thought about this a bit more today, and the conclusion that I'm coming to is that we should set up our trait hierarchy such that a relay protocol can't be implemented as a Transport, but rather only as a Dialer.

Something like this:

/// A transport allows you to initiate connections with no further dependencies.
pub trait Transport {
    /// The connection type this produces.
    type Raw: AsyncRead + AsyncWrite;

    /// The future type this resolves to.
    type Dial: IntoFuture<Item=Self::Raw, Error=DialError>;

    /// Attempt to initiate a connection to an address.
    fn dial(&self, addr: MultiAddr) -> Self::Dial;

    // ...listening methods?
}

/// ...Whereas a Dialer allows you to initiate _composable_ connections
/// through the provision of a "context" to the dial method to allow for sub-dials.
pub trait Dial {
    /// The connection type this produces.
    type Raw: AsyncRead + AsyncWrite;

    /// Dial to the remote address. This can express connections contingent on others.
    /// For example, a relay-style dialer will work contingent on some other connection.
    fn dial(&self, context: &Dial<Raw=Self::Raw>, addr: MultiAddr) 
        -> BoxFuture<Item=Self::Raw, Error=DialError>;
}

/// Blanket implementation which simply ignores the context.
impl<T, R> Dial for T where T: Transport<Raw=R> { ... }

Our Swarm implementation would provide a context that iterates over all registered Dial instances to try and find a match.

[rphmeier]

We can also create specialized Dials for gluing common combinations of protocols together, like WS over TCP. That could probably be done as a generic struct which recognizes certain multiaddr postfixes. Then we would need to implement the swarm in such a way that it prioritizes those attached dialers over the combination of corresponding Transports that can dial the same address with another level of indirection.

[whyrusleeping]

cc @Stebalien who has been doing a lot of thinking on this (and writes rust)

[Stebalien]

We are handling this in go in two ways:

1. We're making all transports "fully featured". That is, transports need to return fully upgraded (multiplexed, secured, etc) connections. This allows us to easily incorporate new transports like QUIC that already provide these features. We'll also provide transports with some helper types/functions for upgrading connections so this logic only gets implemented in one place (basically, we're trying to give the maximum amount of control to transports).
2. More relevant to this issue, we're introducing the concept of "proxy" transports. Proxy transports are special transports that handle the rest of the multiaddr themselves (e.g., in the case of relays, pass it off to another node). IMO, everything after the first instance of /p2p-circuit/ should be an argument to /p2p-circuit/ but it isn't, unfortunately. Then we'd just use the last protocol in the multiaddr.

So, we now pick the appropriate transport by:

1. Searching for the first proxy protocol in the address, if we find one, we use the transport registered to handle that protocol.
2. If we don't find a proxy protocol, we pick the transport registered to handle the last protocol.

Once we've picked the transport, it's up to the transport to deal with the rest of the multiaddr. In the case of the relay transport, we'll have to give the relay transport access to the swarm itself so it can dial the first hop.

I'm also getting rid of "dialers" (at the transport level, at least). I found that the swarm can't really make an educated decision on the correct source address while the transport can so it's better to just leave it up to the transport. Transports can (and probably will) construct dialers internally but, externally, the user will just call my_tpt.dial(remote_address, remote_peer_id).

[Stebalien]

You can find the new interfaces here: https://github.com/libp2p/go-libp2p-transport/blob/feat/refactor/transport.go

We've also been working on an "upgrader" library to allow less featured transports automatically upgrade their connections to fully multiplexed/secure connections: https://github.com/libp2p/go-libp2p-transport-upgrader

[tomaka]

Note that this issue was opened before any code was in the repository. The design of transports has been pretty much figured out now, although we're still doing some modifications from time to time.

We're making all transports "fully featured". That is, transports need to return fully upgraded (multiplexed, secured, etc) connections. This allows us to easily incorporate new transports like QUIC that already provide these features. We'll also provide transports with some helper types/functions for upgrading connections so this logic only gets implemented in one place (basically, we're trying to give the maximum amount of control to transports).

The Rust code does the same thing.

So, we now pick the appropriate transport by:
Searching for the first proxy protocol in the address, if we find one, we use the transport registered to handle that protocol.
If we don't find a proxy protocol, we pick the transport registered to handle the last protocol.

The Rust code currently passes the full multiaddress to all implementations of Transport, and each implementation is responsible for figuring out whether it supports the address as a whole. The transport responsible for p2p-circuit holds an instance of the transport that can be used for communication with the proxy.

For example if you try to dial /ip4/10.11.12.13/tcp/10000/p2p-circuit/ip4/80.81.82.83/tcp/10200, we first try to pass this address to a TcpConfig object, which determines that it cannot handle this multiaddress. We then try to feed the multiaddress to the P2pCircuitConfig transport, which analyses it and determines that it can handle it. The P2pCircuitConfig then passes /ip4/10.11.12.13/tcp/10000 to its inner transport.

In other words, kind of the same as:

Once we've picked the transport, it's up to the transport to deal with the rest of the multiaddr. In the case of the relay transport, we'll have to give the relay transport access to the swarm itself so it can dial the first hop.

Except that we try the transports one by one instead of having a map. There are probably not dozens of transports, so in my opinion any performance difference is negligible. Since the transports are computed at compile-time in the Rust code, it is even possible that the compiler optimizes the lookup.

[Stebalien]

The Rust code does the same thing.

Nice to have a design validated by an independent concurrent implementation 😄.

Except that we try the transports one by one instead of having a map.

So, I considered something like that (I believe this is what js-libp2p does) but couldn't disambiguate /hop1/.../p2p-circuit/hop2/.../p3p-circuit/.../target (where p2p-circuit and p3p-circuit are different "proxy" protocols). That's why I have the "look for the first proxy or the last multiprotocol" logic.