EmbedIO Dual-Stack Localhost Listener Causes Empty Replies

[KazWolfe]

Describe the bug
When using HttpListenerMode.EmbedIO and explicitly listening on both IPv4 (127.0.0.1) and IPv6 ([::1]) interfaces in addition to localhost proper, HTTP calls to localhost may fail. However, appropriately restoring Host may cause connections to succeed again.

It appears as though when localhost is resolved to 127.0.0.1 by the system's DNS, EmbedIO is unable to properly send back a response.

To Reproduce

The following code was sufficient as an MCVE on my system:

void Main() {
    var cts = new CancellationTokenSource();
    var server = new WebServer(o => o
        .WithUrlPrefixes("http://localhost:45454", "http://127.0.0.1:45454", "http://[::1]:45454")
        .WithMode(HttpListenerMode.EmbedIO))
    .WithModule(new ActionModule("/", HttpVerbs.Any, (ctx) => ctx.SendDataAsync(new { Message = "Hello, world!" })));
    
    server.RunAsync(cts.Token).Wait();
}

With the above code, various curl commands were executed, each with different successes:

$ curl "http://localhost:45454"
curl: (52) Empty reply from server
$ curl "http://127.0.0.1:45454"
{"Message": "Hello, world!"}
$ curl "http://[::1]:45454"
{"Message": "Hello, world!"}
$ curl "http://localhost:45454" -H "Host: 127.0.0.1:45454"
{"Message": "Hello, world!"}
$ curl "http://localhost:45454" -H "Host: [::1]:45454"
curl: (52) Empty reply from server
$ curl "http://127.0.0.1:45454" -H "Host: localhost:45454"
curl: (52) Empty reply from server
$ curl "http://[::1]:45454" -H "Host: localhost:45454"
{"Message": "Hello, world!"}

When using cURL's verbose mode:

$ curl -vvvv "http://localhost:45454"
*   Trying 127.0.0.1:45454...
* Connected to localhost (127.0.0.1) port 45454 (#0)
> GET / HTTP/1.1
> Host: localhost:45454
> User-Agent: curl/7.83.1
> Accept: */*
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

$ curl -vvvv "http://localhost:45454" -H "Host: 127.0.0.1:45454"
*   Trying 127.0.0.1:45454...
* Connected to localhost (127.0.0.1) port 45454 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:45454
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Vary: Accept-Encoding
< Content-Encoding: identity
< Content-Type: application/json; charset=utf-8
< Server: EmbedIO/3.5.2
< Date: Sat, 11 Feb 2023 07:32:10 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=15,max=100
<
{"Message": "Hello, world!"}* Connection #0 to host localhost left intact

Notably, if resolution is forced to IPv6, everything works as intended:

$ curl "http://localhost:45454" --ipv6
{"Message": "Hello, world!"}
$ curl "http://localhost:45454" --ipv4
curl: (52) Empty reply from server

Expected behavior
EmbedIO should behave properly when a request is made to localhost when IPv4 is forced.

Screenshots

Wireshark logs from a failed request to localhost:

Desktop:

• OS: Windows 11 Pro, Build 22H2 (OS Build 22621.1105)
• EmbedIO Version 3.5.2 (NuGet)
• curl 7.83.1 (Windows) libcurl/7.83.1 Schannel

Additional context

If WithUrlPrefixes is set to solely http://localhost, EmbedIO doesn't even listen on 127.0.0.1, which may be related (and honestly an easier way to tackle this problem?). My intent for listening to the trifecta was to handle dual-stack systems as easily as possible and not have to rely on systems behaving in odd ways. This may additionally be caused by some odd configuration specific to my system, but I have had a few of my own users confirm this issue in the past as well so I suspect it may be a bit more endemic.

This problem does not exist when using HttpListenerMode.Microsoft, and binding to http://localhost is enough to handle both IPv4 and IPv6 traffic. Ideally, however, I would not need to use the Microsoft listener, as it has caused other issues in the past.

[KazWolfe]

So, finally came back to this and think I figured out the problem. When binding to http://localhost as a URL prefix, EndPointManager will only grab the first resolution from Dns.GetHostAddresses(). On most systems, this will be the IPv6 loopback address of ::1. The actual resolution for 127.0.0.1 goes completely ignored. The EmbedIO listener will then dutifully set up its listener on ::1, as we expect.

Because, however, we are also registering to http://127.0.0.1, EmbedIO binds to both the IPv4 and IPv6 interfaces. When running curl --ipv4 http://localhost:45454/, we connect to the IPv4-only EndPointListener which is not aware of the localhost prefix:

As a result, the server steps through SearchListener, fails to find an appropriate match, and BindContext returns false, killing the connection:

(Of note, while exploring this, I came across issue #165 which had a similar comment - binding to 0.0.0.0 actually creates something that listens on all interfaces, but because the EndPointListener is not aware of 0.0.0.0 being "special", it also gives up. This is doubly made interesting by the fact that EndPointManager.cs#L96 actually has something in place for this edge case, allowing it to bind to [::] instead, but I'm on a tangent so I'll get back to the issue at hand.)

As an experiment, I wrote a (very quick and dirty) patch that changed the behavior of GetEpListener to instead return a list of eligible listeners (assuming more than one is returned by the aforementioned DNS query). This, to my surprise, actually seems to work:

The localhost prefix is added to both the IPv4 and IPv6 listeners, and cURL can connect to either one and receive the expected response. In addition, the actual IP addresses also defined also can connect and get routed appropriately, bringing everything back to what I would have expected. I'm more than happy to formally submit this patch as a PR, but I'm unsure what else would be needed before then (and what other things are lurking in EndPointListener that should be looked at).

(Edit: The above patch diverges from Mono's behavior, which I suspect is not a desired change. Would it make sense to open this issue with the Mono team directly? As far as I can tell from Mono's code, they'd have the same problem - setting prefixes to 127.0.0.1 and [::1] would work, but connects with Host: localhost would be rejected. All this to say EmbedIO v4 can't come soon enough!)

As for why all this came up in the first place: browsers are problematic. Despite the fact that Windows is usually pretty consistent (ha) about returning DNS records for localhost in the same order, browsers will sometimes decide that they want to swap things around or prefer IPv4 to IPv6 at times. This causes the browser to resolve localhost to 127.0.0.1 and hit the EndPointListener with no knowledge that it should be responding to localhost, because all it knows is IPv6.