Using proxy_protocol v2 with h2c backend gives wrong IP address to backend. #6342

CRCinAU · 2024-05-26T03:10:40Z

I recently moved over to Caddy as a frontend for one of my sites.

Extract of the Caddyfile:

example.com {
        header Strict-Transport-Security "max-age=63072000"
        header -Server

        handle_path /forum/* {
                reverse_proxy http://<host2>:8000
        }

        reverse_proxy h2c://<docker_container_name>:80 {
                transport http {
                        proxy_protocol v2
                }
        }
}

When configured as above, after a random number of hits, the source IP addresses logged in the reverse proxy will all be the same. This includes ANY host - IPv4 or IPv6.

Changing to use http:// as the backend as follows seems to report the source IP address correctly:

example.com {
        header Strict-Transport-Security "max-age=63072000"
        header -Server

        handle_path /forum/* {
                reverse_proxy http://<host2>:8000
        }

        reverse_proxy http://<docker_container_name>:80 {
                transport http {
                        proxy_protocol v2
                }
        }
}

Versions:

/srv # caddy --version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

The text was updated successfully, but these errors were encountered:

francislavoie · 2024-05-26T03:40:13Z

Ah that makes sense. The connections to the backend are pooled I think, so subsequent requests might appear to come from the same IP as the first backend. I'm not sure if we have a way to turn off pooling in h2c mode right now.

mohammed90 · 2024-05-26T06:34:02Z

The ntlm-transport does pooling per remote IP address. I wonder if the mechanism can be copied into core for this use case. Now that I said that, I realize the same logic probably covers both NTLM and proxy-protocol + h2c.

WeidiDeng · 2024-05-27T01:07:45Z

@mohammed90 I always thought we should pool connections for the same client ip if proxy protocol is enabled instead of blindly disabling keep-alive. Tried to implement custom pooling but gave up. That package gives me some inspiration.

WeidiDeng · 2024-05-27T06:54:52Z

@CRCinAU Can you try xcaddy build h2c-proxy-protocol to see if it's fixed?

CRCinAU · 2024-05-27T07:17:09Z

I'm not really familiar enough with Caddy to be able to pull this off - I've only ever used the docker container from docker hub to use Caddy. Is there any way to bring this into the existing docker container?

WeidiDeng · 2024-05-27T07:21:10Z

Run following dockerfile:

FROM caddy:2.7.6-builder AS builder

RUN xcaddy build h2c-proxy-protocol

FROM caddy:2.7.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

The resulting image contains this patch. And you can copy the binary.

CRCinAU · 2024-05-27T07:46:08Z

I tried running this with h2c:// - but caddy just seemed to hang when talking to the backend... Nothing seemed to make it through to the client.

WeidiDeng · 2024-05-27T07:56:36Z

Any logs available? Please enable debug in the global option.

CRCinAU · 2024-05-27T12:35:44Z

I'm giving this a go now... I tried it on my main web site - but as it died, I just rolled back straight away.

Here's the logs I see from just random internet traffic hitting the site when using h2c:// to the backend.

{"level":"debug","ts":1716813182.8682232,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"httpd:80","total_upstreams":1}
{"level":"debug","ts":1716813182.868803,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813182.869208,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813183.8701084,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813185.8720198,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813189.8733985,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813190.4139547,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"83.229.76.239","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813197.8786793,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813206.4157789,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"83.229.76.239","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}
{"level":"debug","ts":1716813213.882467,"logger":"http.reverse_proxy.transport.http","msg":"sending proxy protocol header v2","header":{"Version":2,"Command":33,"TransportProtocol":17,"SourceAddr":{"IP":"45.87.9.222","Port":0,"Zone":""},"DestinationAddr":{"IP":"0.0.0.0","Port":0,"Zone":""}}}

francislavoie · 2024-05-27T15:07:03Z

You need to configure it like this (see https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-http-transport):

	reverse_proxy h2c://<docker_container_name>:80 {
		transport http {
			keepalive off
			proxy_protocol v2
		}
	}

CRCinAU · 2024-05-27T15:45:52Z

You need to configure it like this (see https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-http-transport):
	reverse_proxy h2c://<docker_container_name>:80 {
		transport http {
			keepalive off
			proxy_protocol v2
		}
	}

This does actually seem to work - ie the connection doesn't hang - on the caddy:latest image - which seems to be:

docker exec -ti caddy /bin/sh
/srv # caddy --version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
/srv #

However, even in this configuration, the wrong remote IP address is displayed by apache - which is the start of this bug report.

Trying this again with the instructions above ( #6342 (comment) ), the connection still hangs.

Eventually, I get a 502 timeout from the backend:

{
  "level": "error",
  "ts": 1716824790.699319,
  "logger": "http.log.error",
  "msg": "http2: client conn not usable",
  "request": {
    "remote_ip": "<my ipv6 address>",
    "remote_port": "51640",
    "client_ip": "<my ipv6 address>",
    "proto": "HTTP/3.0",
    "method": "GET",
    "host": "<fqdn>",
    "uri": "/",
    "headers": {
      "Sec-Fetch-Dest": [
        "document"
      ],
      "Accept-Language": [
        "en-GB,en;q=0.6"
      ],
      "Sec-Fetch-Mode": [
        "navigate"
      ],
      "Sec-Fetch-User": [
        "?1"
      ],
      "Sec-Ch-Ua-Platform": [
        "\"Linux\""
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
      ],
      "Sec-Gpc": [
        "1"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "Referer": [
        "https://<fqdn>"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br, zstd"
      ],
      "Sec-Ch-Ua": [
        "\"Brave\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""
      ],
      "Sec-Ch-Ua-Mobile": [
        "?0"
      ],
      "Cookie": [
        "REDACTED"
      ],
      "Priority": [
        "u=0, i"
      ],
      "Cache-Control": [
        "max-age=0"
      ],
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4867,
      "proto": "h3",
      "server_name": "<fqdn>"
    }
  },
  "duration": 64.009204212,
  "status": 502,
  "err_id": "07ygq06wq",
  "err_trace": "reverseproxy.statusError (reverseproxy.go:1269)"
}

WeidiDeng · 2024-05-28T00:22:52Z

You need to configure it like this (see https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-http-transport):
	reverse_proxy h2c://<docker_container_name>:80 {
		transport http {
			keepalive off
			proxy_protocol v2
		}
	}

@francislavoie keepalive is disabled if proxy_protocol is in use.

francislavoie · 2024-05-28T00:44:14Z

Ah, I see yeah

caddy/modules/caddyhttp/reverseproxy/httptransport.go

Lines 342 to 348 in 77394f2

    
           // The proxy protocol header can only be sent once right after opening the connection. 
        
           // So single connection must not be used for multiple requests, which can potentially 
        
           // come from different clients. 
        
           if !rt.DisableKeepAlives && h.ProxyProtocol != "" { 
        
           	caddyCtx.Logger().Warn("disabling keepalives, they are incompatible with using PROXY protocol") 
        
           	rt.DisableKeepAlives = true 
        
           }

So just need to use the new build to test I guess.

WeidiDeng · 2024-05-28T01:42:42Z

I think this is a stdlib issue,

https://github.com/golang/net/blob/022530c41555839e27aec3868cc480fb7b5e33d4/http2/transport.go#L1028

However h2c requests start with a streamID of 3.

https://github.com/golang/net/blob/022530c41555839e27aec3868cc480fb7b5e33d4/http2/transport.go#L836

So requests never get sent in this case.

WeidiDeng · 2024-05-30T02:28:05Z

@CRCinAU Can you try with xcaddy build h2c-proxy-protocol --replace golang.org/x/net=github.com/WeidiDeng/net@h2c-disable-keepalive? You'll need to update the caddy image version to 2.8.0 since older version can't build with this method.

CRCinAU · 2024-05-30T03:05:24Z

@WeidiDeng - Sorry, I'm not the best with Caddy or its build process - I tried the following modified Dockerfile based on the above:

FROM caddy:2.8.0-builder AS builder

RUN xcaddy build h2c-proxy-protocol --replace golang.org/x/net=github.com/WeidiDeng/net@h2c-disable-keepalive

FROM caddy:2.8.0

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

However it complains there isn't a tag for caddy:2.8.0 or caddy:2.8.0-builder

ie:

ERROR: failed to solve: caddy:2.8.0-builder: failed to resolve source metadata for docker.io/library/caddy:2.8.0-builder: no match for platform in manifest: not found

Looking at the tags on Docker Hub, I can see that there is a builder and latest tag updated 2 hours ago, and they do have linux/amd64 images listed. Would it be ok to use :builder and :latest for this test?

EDIT: I noticed there are images for caddy:2.8-builder and caddy:2.8 - which I tried, but that errored out with:

 => ERROR [builder 2/2] RUN xcaddy build h2c-proxy-protocol --replace golang.org/x/net=github.com/WeidiDeng/net@h2c-disable-keepalive
[ERROR] missing flag; caddy version already set at h2c-proxy-protocol

WeidiDeng · 2024-05-30T03:23:22Z

That builder has the wrong xcaddy version 0.4.1 instead of the latest 0.4.2. It'll be a while before the docker is ready.

WeidiDeng · 2024-05-30T04:27:53Z

@CRCinAU 2.8.0-builder is ready now, you can try again.

CRCinAU · 2024-05-30T04:33:44Z

Built ok now, and I can confirm I'm seeing HTTP/2.0 requests to the backend, and it looks to be the correct IP address reported to the backend via the proxy protocol.

WeidiDeng linked a pull request May 27, 2024 that will close this issue

reverse_proxy: apply keep-alive setting for h2c requests #6343

Open

WeidiDeng added the upstream ⬆️ Relates to some dependency of this project label May 28, 2024

WeidiDeng mentioned this issue May 28, 2024

x/net/http2: h2c requests can't be sent when *http.Request.Close is true golang/go#67671

Closed

caddyserver deleted a comment from jodeokrae Jun 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using proxy_protocol v2 with h2c backend gives wrong IP address to backend. #6342

Using proxy_protocol v2 with h2c backend gives wrong IP address to backend. #6342

CRCinAU commented May 26, 2024

francislavoie commented May 26, 2024

mohammed90 commented May 26, 2024

WeidiDeng commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

francislavoie commented May 27, 2024

CRCinAU commented May 27, 2024 •

edited

Loading

WeidiDeng commented May 28, 2024

francislavoie commented May 28, 2024 •

edited

Loading

WeidiDeng commented May 28, 2024

WeidiDeng commented May 30, 2024

CRCinAU commented May 30, 2024 •

edited

Loading

WeidiDeng commented May 30, 2024

WeidiDeng commented May 30, 2024

CRCinAU commented May 30, 2024

Using proxy_protocol v2 with h2c backend gives wrong IP address to backend. #6342

Using proxy_protocol v2 with h2c backend gives wrong IP address to backend. #6342

Comments

CRCinAU commented May 26, 2024

francislavoie commented May 26, 2024

mohammed90 commented May 26, 2024

WeidiDeng commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

WeidiDeng commented May 27, 2024

CRCinAU commented May 27, 2024

francislavoie commented May 27, 2024

CRCinAU commented May 27, 2024 • edited Loading

WeidiDeng commented May 28, 2024

francislavoie commented May 28, 2024 • edited Loading

WeidiDeng commented May 28, 2024

WeidiDeng commented May 30, 2024

CRCinAU commented May 30, 2024 • edited Loading

WeidiDeng commented May 30, 2024

WeidiDeng commented May 30, 2024

CRCinAU commented May 30, 2024

CRCinAU commented May 27, 2024 •

edited

Loading

francislavoie commented May 28, 2024 •

edited

Loading

CRCinAU commented May 30, 2024 •

edited

Loading