ConnectionPool: Allow connection scoring/preference, for example for a certain EventLoop

[weissi]

Description

A general connection pool works by first narrowing down the available connections to a set of 'acceptable' connections. That is usually done by matching a few required attributes like host/port/TLS configuration. But usually, not all connections in the resulting set are equally desirable for a client.
For example, a client may prefer a connection that is on a particular EventLoop (or not on one). We might call it EventLoop (anti-)affinity. Other attributes like a geographical/DC/... (anti-)affinity or a load factor sound entirely plausible too.

Especially for proxies, EventLoop affinity is very important because they may need to proxy thousands of concurrent requests which shouldn't force thread switches for every request.

Additionally, (thanks @Lukasa for mentioning) a client may have an idea of how long they might be willing to wait for a 'better' connection.

API ideas

• One idea would be for the user of the connection pool to provide a scoring/sorting function about how much they like a certain connection. One idea would be score(Connection) -> Double (1.0: love it; 0.0: cannot use it) or orderConnectionAttractiveness(Connection, Connection) -> Bool

[fabianfett]

I like the idea of allowing preferences (especially for EventLoops) when picking a connections. However – since you mention other attributes (host/port/TLS) – we might need to drill a bit deeper.

Generally I would like to cluster the preferences into two categories:

1. Nice to have preferences: I think this is a prime example for EventLoop affinity
2. Required preferences: I think this is mostly host/port/TLS.

For the required preferences (host/port/TLS), we must ensure that the properties are matched. I wonder if this is something that should be handled inside the connection pool. An alternative might be a pool of pools object that manages the different host/port/TLS configurations. We might want to create an advanced pool that offers these kind of options.

---

One question that has been in the back of my mind for a longer time, how do connection limits (min number of connections, max number of connections) relate to connection properties? How do we handle a situation in which we can create at most 10 connections but have 16 different EventLoops?

@weissi @Lukasa Do you have a hint for me here?