Add packet conn wrapper support (WIP) #318

vnxme · 2025-08-14T08:46:06Z

This PR adds caddy.PacketConnWrapper implementation (caddyserver/caddy#7180) to support proper UDP multiplexing.

An example of how it could be used:

{
	servers {
		packet_conn_wrappers {
			layer4 {
				@d dns
				route @d {
					proxy udp/one.one.one.one:53
				}
				@w wireguard
				route @w {
					proxy udp/192.168.1.1:51820
				}
			}
		}
	}
}
https://localhost {
	tls {
		issuer internal
	}
	respond "OK" 200
}

For now I've added a backbone code and a few integration tests. But I need help with its core, see TODOs in WrapPacketConn and Provision functions. When I'm trying to imagine a packet path and all the goroutines, channels, etc., it blows my mind, so I must confess it's still too complicated for me to implement. @WeidiDeng, I'd be grateful if you could look at it.

WeidiDeng · 2025-08-14T12:52:04Z

I have given it some thought, something must be changed with layer4.Connection as it asumes there is an underlying connection.

To be fair, the udp traffic matching is also kind of like quic, check if the remote address has an active session, but much less complicated.

I need to add some optional interface that will expose a net.PacketConn. There is also the problem to mark a session as unmatched and all its udp buffers will be read like one, i.e, return the remote address, and discarding buffered data if the read buffer isn't big enough.

WeidiDeng · 2025-08-16T10:01:30Z

@vnxme I finished an implementation of the wrapper, though didn't test it.

vnxme · 2025-08-16T10:12:42Z

@WeidiDeng, thank you! I really appreciate your help. Will test it within a few days.

vnxme · 2025-08-20T12:36:24Z

Since now we use Server fields under the hood of PacketConnWrapper, we have to create it manually at provision and while unmarshalling a caddyfile. I've added a commit fixing this issue.

My testing config:

{
	debug
	layer4 {
		udp/:5300 {
			@d dns
			route @d {
				proxy udp/one.one.one.one:53
			}
		}
	}
	servers {
		packet_conn_wrappers {
			layer4 {
				@d dns
				route @d {
					proxy udp/one.one.one.one:53
				}
				@w wireguard
				route @w {
					proxy udp/192.168.1.1:51820
				}
			}
		}
	}
}
https://localhost {
	tls {
		issuer internal
	}
	respond "OK" 200
}

Steps:

Ensure HTTP/3 still works. Using Curl v8.15.0: curl.exe --http3 --insecure https://localhost/ -v. It's OK.
Ensure we get a DNS response. Using Q v0.19.2: q.exe /all /v @plain://127.0.0.1:443 A google.com. It fails.
Ensure there are no problems with passing through WireGuard traffic. Not tested yet.

Caddy logs the following, but layer4 lines come within ca. 5-10 seconds after Q fails with i/o timeout. Each layer4 line is one attempt in Q. For now I don't know what's wrong, need to trace deeper.

{"level":"info","ts":1755692684.0344846,"msg":"serving initial configuration"}
{"level":"debug","ts":1755692752.9979715,"logger":"caddy.packetconns.layer4","msg":"connection stats","remote":"127.0.0.1:62274","read":0,"written":0,"duration":30.0059349}
{"level":"debug","ts":1755692799.393022,"logger":"caddy.packetconns.layer4","msg":"connection stats","remote":"127.0.0.1:50860","read":0,"written":0,"duration":30.0026523}

DNS works fine when using layer4 directly, i.e. without packen conn wrapper, with q.exe /all /v @plain://127.0.0.1:5300 A google.com.

…o handle connections

WeidiDeng · 2025-08-25T02:51:55Z

@vnxme Can you check if the latest commits work for you?

vnxme · 2025-08-25T14:08:23Z

@WeidiDeng, thank you for keeping working on it.

It seems to have got a bit worse now:

Test 1 (HTTP/3) fails if run after Caddy launch, then passes if not run after the second test;
Test 2 (DNS) ~~fails consistently~~ passes if run after Caddy launch, but fails if run after the first test.

I also got the following panics a few times when stopping Caddy after running the second test and when running the first test after the second one:

panic: send on closed channel

goroutine 45 [running]:
github.com/mholt/caddy-l4/layer4.(*packetConn).Read(0x4000112240, {0x40006c6000, 0x2000, 0x7ff64d6ab934?})
	.../caddy-l4/layer4/server.go:363 +0x480
github.com/mholt/caddy-l4/layer4.(*Connection).Read(0x40002ee540, {0x40006c6000?, 0x7ff65039e740?, 0x3?})
	.../caddy-l4/layer4/connection.go:113 +0xf8
io.(*teeReader).Read(0x4000069020, {0x40006c6000, 0x40006c5e78?, 0x2000})
	C:/Program Files/Go/src/io/io.go:628 +0x30
io.discard.ReadFrom({}, {0x7ff64f3b4700, 0x4000069020})
	C:/Program Files/Go/src/io/io.go:666 +0x70
io.copyBuffer({0x7ff64f3b4740, 0x7ff650436160}, {0x7ff64f3b4700, 0x4000069020}, {0x0, 0x0, 0x0})
	C:/Program Files/Go/src/io/io.go:415 +0x14c
io.Copy(...)
	C:/Program Files/Go/src/io/io.go:388
github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy.func2()
	.../caddy-l4/modules/l4proxy/proxy.go:345 +0x54
created by github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy in goroutine 41
	.../caddy-l4/modules/l4proxy/proxy.go:342 +0x2b4
panic: send on closed channel

goroutine 91 [running]:
github.com/mholt/caddy-l4/layer4.(*packetConn).Read(0x4000093b60, {0x4000610000, 0x2000, 0x7ff64d6ab934?})
	.../caddy-l4/layer4/server.go:363 +0x480
github.com/mholt/caddy-l4/layer4.(*Connection).Read(0x40000c58f0, {0x4000610000?, 0x7ff65039e740?, 0x0?})
	.../caddy-l4/layer4/connection.go:113 +0xf8
io.(*teeReader).Read(0x40004be700, {0x4000610000, 0x400063fe78?, 0x2000})
	C:/Program Files/Go/src/io/io.go:628 +0x30
io.discard.ReadFrom({}, {0x7ff64f3b4700, 0x40004be700})
	C:/Program Files/Go/src/io/io.go:666 +0x70
io.copyBuffer({0x7ff64f3b4740, 0x7ff650436160}, {0x7ff64f3b4700, 0x40004be700}, {0x0, 0x0, 0x0})
	C:/Program Files/Go/src/io/io.go:415 +0x14c
io.Copy(...)
	C:/Program Files/Go/src/io/io.go:388
github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy.func2()
	.../caddy-l4/modules/l4proxy/proxy.go:345 +0x54
created by github.com/mholt/caddy-l4/modules/l4proxy.(*Handler).proxy in goroutine 87
	.../caddy-l4/modules/l4proxy/proxy.go:342 +0x2b4

# Conflicts: # go.mod # go.sum

WeidiDeng · 2025-08-26T00:38:51Z

@vnxme Both tests pass consistently from my testing. And the order don't affect the result. Both can run concurrently without affecting the other.

I can replicate the send on closed channel though.

…ions

…xme/packet-conn-wrapper

vnxme · 2025-08-27T07:13:07Z

@WeidiDeng, thanks for fixing the panic.

I have reproducible failures for the following testing sequences:

Q (pass), Q (pass), Curl (fail), Curl (pass)
After two passing Q tests, the first Curl test fails with ngtcp2_conn_handle_expiry returned error: ERR_IDLE_CLOSE. Caddy logs the following:

{"level":"info","ts":1756277781.6609652,"msg":"serving initial configuration"}
{"level":"info","ts":1756277781.6614707,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:...","instance":"5687cf66-4f26-4518-ae31-a8c01af643e3","try_again":1756364181.6614707,"try_again_in":86400}
{"level":"info","ts":1756277781.6620045,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1756277789.703734,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"127.0.0.1:57338","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756277789.703734,"logger":"caddy.packetconns.layer4","msg":"prefetched","remote":"127.0.0.1:57338","bytes":28}
{"level":"debug","ts":1756277789.703734,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"127.0.0.1:57338","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1756277789.7073565,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"127.0.0.1:57338","upstream":"one.one.one.one:53"}
{"level":"debug","ts":1756277793.3731508,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"127.0.0.1:57340","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756277793.3731508,"logger":"caddy.packetconns.layer4","msg":"prefetched","remote":"127.0.0.1:57340","bytes":28}
{"level":"debug","ts":1756277793.3731508,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"127.0.0.1:57340","matcher":"layer4.matchers.dns","matched":true}
{"level":"debug","ts":1756277793.3742611,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"127.0.0.1:57340","upstream":"one.one.one.one:53"}
{"level":"debug","ts":1756277797.7071056,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:57342","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756277797.7071056,"logger":"caddy.packetconns.layer4","msg":"prefetched","remote":"[::1]:57342","bytes":1200}
{"level":"debug","ts":1756277797.7071056,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:57342","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756277797.7071056,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:57342","matcher":"layer4.matchers.wireguard","matched":false}
{"level":"debug","ts":1756277797.7094665,"logger":"events","msg":"event","name":"tls_get_certificate","id":"aa43ee5f-d6a1-40b8-876d-c9ca51104440","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027],"SupportedProtos":["h3","h3-29"],"SupportedVersions":[772],"RemoteAddr":{"IP":"::1","Port":57342,"Zone":""},"LocalAddr":{"IP":"::","Port":443,"Zone":""}}}}
{"level":"debug","ts":1756277797.710015,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756277797.710015,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756277797.710015,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"57342","subjects":["localhost"],"managed":true,"expiration":1756296164,"hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756277797.7324064,"logger":"caddy.packetconns.layer4","msg":"connection stats","remote":"127.0.0.1:57340","read":28,"written":124,"duration":4.3592537}
{"level":"debug","ts":1756277797.7324064,"logger":"caddy.packetconns.layer4","msg":"connection stats","remote":"127.0.0.1:57338","read":28,"written":124,"duration":8.028674}
{"level":"debug","ts":1756277832.1905231,"logger":"events","msg":"event","name":"tls_get_certificate","id":"45215d53-0cf1-4b58-8190-1dfda06bdee5","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,196,136,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,49170,49160,22,10,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"::1","Port":64796,"Zone":""},"LocalAddr":{"IP":"::1","Port":443,"Zone":""}}}}
{"level":"debug","ts":1756277832.1911163,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756277832.1911163,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756277832.1911163,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"64796","subjects":["localhost"],"managed":true,"expiration":1756296164,"hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"error","ts":1756277832.2316618,"logger":"http.log","msg":"setting HTTP/3 Alt-Svc header","error":"no port can be announced, specify it explicitly using Server.Port or Server.Addr"}

Curl (fail), Curl (pass), Q (fail), Q (fail)
The first Curl test fails with ngtcp2_conn_handle_expiry returned error: ERR_IDLE_CLOSE. Then after the second passing Curl test two Q tests fail consecutively with i/o timeout. Caddy logs the following:

{"level":"info","ts":1756278337.019835,"msg":"serving initial configuration"}
{"level":"debug","ts":1756278348.5251482,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:65372","error":"consumed all prefetched bytes","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756278348.5251482,"logger":"caddy.packetconns.layer4","msg":"prefetched","remote":"[::1]:65372","bytes":1200}
{"level":"debug","ts":1756278348.5261483,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:65372","matcher":"layer4.matchers.dns","matched":false}
{"level":"debug","ts":1756278348.5261483,"logger":"caddy.packetconns.layer4","msg":"matching","remote":"[::1]:65372","matcher":"layer4.matchers.wireguard","matched":false}
{"level":"debug","ts":1756278348.5271487,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c6582d97-ac8a-4621-bdab-87a554080eec","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027],"SupportedProtos":["h3","h3-29"],"SupportedVersions":[772],"RemoteAddr":{"IP":"::1","Port":65372,"Zone":""},"LocalAddr":{"IP":"::","Port":443,"Zone":""}}}}
{"level":"debug","ts":1756278348.5286584,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756278348.5286584,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756278348.5286584,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"65372","subjects":["localhost"],"managed":true,"expiration":1756296164,"hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756278381.0029945,"logger":"events","msg":"event","name":"tls_get_certificate","id":"727c047a-6c0a-4a9f-b282-b0d10d68d64a","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,196,136,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,49170,49160,22,10,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"::1","Port":64997,"Zone":""},"LocalAddr":{"IP":"::1","Port":443,"Zone":""}}}}
{"level":"debug","ts":1756278381.0029945,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756278381.0029945,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"debug","ts":1756278381.0029945,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"::1","remote_port":"64997","subjects":["localhost"],"managed":true,"expiration":1756296164,"hash":"38fdbf0f4efc338899c42514cf2391ff3a21962cf1efbcce678ccb5c5bb0c3e9"}
{"level":"error","ts":1756278381.0360377,"logger":"http.log","msg":"setting HTTP/3 Alt-Svc header","error":"no port can be announced, specify it explicitly using Server.Port or Server.Addr"}

WeidiDeng · 2025-08-27T08:48:54Z

@vnxme As I said, all these tests succeed without any error on my part.

Caddy is running in another terminal, and is restarted between tests.

vnxme · 2025-08-27T12:12:26Z

is restarted between tests

I do restart Caddy between test sequences, but don't restart otherwise. Curl and Q are run from different terminals inside the IDE.

Unless you can reproduce the issues I have, the only way I can imagine is to write automatic tests. Will look into what I can do.

vnxme · 2025-08-31T13:08:12Z

@WeidiDeng, I've added some complex tests for PCW and UDP, but let's begin with the latter.

Pure UDP

TestUDP is designed to conduct a simple UDP message exchange. Sometimes it passes, but usually fails with close of closed channel panic, see below. I tried to experiment with the number of messages sent one by one and the time between sending messages, but it occasionally fails for me even when only one exchange is done (one write and one read).

=== RUN   TestUDP
{"level":"info","ts":1756643099.7956412,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"debug","ts":1756643099.7956412,"logger":"layer4","msg":"listening","address":"udp/[::]:2000"}
{"level":"info","ts":1756643099.7968771,"msg":"autosaved config (load with --resume flag)","file":"...\\autosave.json"}
    udp_test.go:53: [UDP-2000] connecting to localhost:2000, sending ECHO0000, expecting ECHO0000
    udp_test.go:69: [UDP-2000] wrote UDP message ECHO0000, read UDP response ECHO0000
{"level":"debug","ts":1756643100.3006787,"logger":"layer4","msg":"connection stats","remote":"127.0.0.1:49379","read":8,"written":8,"duration":0.0007991}
panic: close of closed channel

goroutine 49 [running]:
github.com/mholt/caddy-l4/layer4.(*Server).servePacket(0x40004058f0, {0x2091deadda0, 0x4000499c00})
	.../caddy-l4/layer4/server.go:153 +0x550
github.com/mholt/caddy-l4/layer4.(*App).Start.func2(...)
	.../caddy-l4/layer4/app.go:86
created by github.com/mholt/caddy-l4/layer4.(*App).Start in goroutine 35
	.../caddy-l4/layer4/app.go:85 +0x558

PacketConnWrapper

TestPCW is intended to test PCW with simultaneous HTTP/3, DNS and UDP requests. I had to comment out the UDP part because of the panic described above. I have to confirm that the current implementation of PCW you suggested works fine most of the time (over 50% for me). But sometimes it fails as well, see below.

I'm not sure whether it's caused by some DNS limitations (the test sends real queries to Cloudflare), that's why I'd like to disable the DNS part and enable the UDP part when it doesn't panic any more -- then I hope we'll be able to either find more issues, or confirm there are none.

Even if the test can't obtain DNS responses Cloudflare, it should have no effect on responding to HTTP/3 requests, but it does, that's why I guess there're some issues to identify and fix.

An extract from a very long Caddy log

=== RUN   TestPCW
...
{"level":"debug","ts":1756644347.1906235,"logger":"events","msg":"event","name":"tls_get_certificate","id":"fab6fbce-7e0f-47e2-b993-f56370f10f0d","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4865,4866],"ServerName":"localhost","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1283,1539],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"127.0.0.1","Port":50082,"Zone":""},"LocalAddr":{"IP":"::","Port":8443,"Zone":""}}}}
{"level":"debug","ts":1756644347.1906235,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756644347.1906235,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
{"level":"debug","ts":1756644347.1906235,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"50082","subjects":["localhost"],"managed":true,"expiration":1756661647,"hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
    common_test.go:216: [HTTP/3-8443.97] received HTTP response body: 97
    packetconn_test.go:151: [HTTP/3-443.98] connecting to localhost:443, requesting https://localhost:443/?98, expecting 98
    packetconn_test.go:127: [DNS-443.78] make DNS query: read udp [::1]:49965->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.49] make HTTP request: Get "https://localhost:443/?49": timeout: no recent network activity
    packetconn_test.go:113: [DNS-443.98] connecting to localhost:443, querying x.com. IN A, expecting at least one
    packetconn_test.go:151: [HTTP/3-8443.98] connecting to localhost:8443, requesting https://localhost:8443/?98, expecting 98
{"level":"debug","ts":1756644347.2939723,"logger":"events","msg":"event","name":"tls_get_certificate","id":"7879fb4f-60da-4deb-b87b-e82c68a172d5","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4865,4866],"ServerName":"localhost","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1283,1539],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"127.0.0.1","Port":50085,"Zone":""},"LocalAddr":{"IP":"::","Port":8443,"Zone":""}}}}
{"level":"debug","ts":1756644347.2939723,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756644347.2939723,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
{"level":"debug","ts":1756644347.2939723,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"50085","subjects":["localhost"],"managed":true,"expiration":1756661647,"hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
    common_test.go:216: [HTTP/3-8443.98] received HTTP response body: 98
    packetconn_test.go:151: [HTTP/3-443.99] connecting to localhost:443, requesting https://localhost:443/?99, expecting 99
    packetconn_test.go:127: [DNS-443.79] make DNS query: read udp [::1]:49968->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.50] make HTTP request: Get "https://localhost:443/?50": context deadline exceeded
    packetconn_test.go:151: [HTTP/3-8443.99] connecting to localhost:8443, requesting https://localhost:8443/?99, expecting 99
    packetconn_test.go:113: [DNS-443.99] connecting to localhost:443, querying wikipedia.org. IN A, expecting at least one
{"level":"debug","ts":1756644347.395006,"logger":"events","msg":"event","name":"tls_get_certificate","id":"98733f2a-d0fd-4872-9970-acf306c19868","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4865,4866],"ServerName":"localhost","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1283,1539],"SupportedProtos":["h3"],"SupportedVersions":[772],"RemoteAddr":{"IP":"127.0.0.1","Port":50087,"Zone":""},"LocalAddr":{"IP":"::","Port":8443,"Zone":""}}}}
{"level":"debug","ts":1756644347.395006,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1756644347.395006,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
{"level":"debug","ts":1756644347.395006,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"50087","subjects":["localhost"],"managed":true,"expiration":1756661647,"hash":"4b62d603aad2e4c53490d44321b8df2383bab38fa09a1481c385197fdfa656fd"}
    common_test.go:216: [HTTP/3-8443.99] received HTTP response body: 99
    packetconn_test.go:127: [DNS-443.80] make DNS query: read udp [::1]:49970->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.51] make HTTP request: Get "https://localhost:443/?51": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.81] make DNS query: read udp [::1]:49974->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.52] make HTTP request: Get "https://localhost:443/?52": context deadline exceeded
    packetconn_test.go:127: [DNS-443.82] make DNS query: read udp [::1]:49977->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.53] make HTTP request: Get "https://localhost:443/?53": context deadline exceeded
    packetconn_test.go:127: [DNS-443.83] make DNS query: read udp [::1]:49980->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.54] make HTTP request: Get "https://localhost:443/?54": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.84] make DNS query: read udp [::1]:49983->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.55] make HTTP request: Get "https://localhost:443/?55": context deadline exceeded
    packetconn_test.go:127: [DNS-443.85] make DNS query: read udp [::1]:49986->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.56] make HTTP request: Get "https://localhost:443/?56": context deadline exceeded
    packetconn_test.go:127: [DNS-443.86] make DNS query: read udp [::1]:49989->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.57] make HTTP request: Get "https://localhost:443/?57": context deadline exceeded
    packetconn_test.go:127: [DNS-443.87] make DNS query: read udp [::1]:49992->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.58] make HTTP request: Get "https://localhost:443/?58": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.88] make DNS query: read udp [::1]:49995->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.59] make HTTP request: Get "https://localhost:443/?59": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.89] make DNS query: read udp [::1]:49997->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.60] make HTTP request: Get "https://localhost:443/?60": context deadline exceeded
    packetconn_test.go:127: [DNS-443.90] make DNS query: read udp [::1]:50060->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.61] make HTTP request: Get "https://localhost:443/?61": context deadline exceeded
    packetconn_test.go:127: [DNS-443.91] make DNS query: read udp [::1]:50063->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.62] make HTTP request: Get "https://localhost:443/?62": context deadline exceeded
    packetconn_test.go:127: [DNS-443.92] make DNS query: read udp [::1]:50066->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.63] make HTTP request: Get "https://localhost:443/?63": context deadline exceeded
    packetconn_test.go:127: [DNS-443.93] make DNS query: read udp [::1]:50069->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.64] make HTTP request: Get "https://localhost:443/?64": context deadline exceeded
    packetconn_test.go:127: [DNS-443.94] make DNS query: read udp [::1]:50072->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.65] make HTTP request: Get "https://localhost:443/?65": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.95] make DNS query: read udp [::1]:50075->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.66] make HTTP request: Get "https://localhost:443/?66": context deadline exceeded
    packetconn_test.go:127: [DNS-443.96] make DNS query: read udp [::1]:50078->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.67] make HTTP request: Get "https://localhost:443/?67": context deadline exceeded
    packetconn_test.go:127: [DNS-443.97] make DNS query: read udp [::1]:50081->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.68] make HTTP request: Get "https://localhost:443/?68": timeout: no recent network activity
    packetconn_test.go:127: [DNS-443.98] make DNS query: read udp [::1]:50084->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.69] make HTTP request: Get "https://localhost:443/?69": context deadline exceeded
    packetconn_test.go:127: [DNS-443.99] make DNS query: read udp [::1]:50088->[::1]:443: i/o timeout
    packetconn_test.go:164: [HTTP/3-443.70] make HTTP request: Get "https://localhost:443/?70": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.71] make HTTP request: Get "https://localhost:443/?71": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.72] make HTTP request: Get "https://localhost:443/?72": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.73] make HTTP request: Get "https://localhost:443/?73": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.74] make HTTP request: Get "https://localhost:443/?74": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.75] make HTTP request: Get "https://localhost:443/?75": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.76] make HTTP request: Get "https://localhost:443/?76": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.77] make HTTP request: Get "https://localhost:443/?77": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.78] make HTTP request: Get "https://localhost:443/?78": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.79] make HTTP request: Get "https://localhost:443/?79": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.80] make HTTP request: Get "https://localhost:443/?80": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.81] make HTTP request: Get "https://localhost:443/?81": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.82] make HTTP request: Get "https://localhost:443/?82": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.83] make HTTP request: Get "https://localhost:443/?83": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.84] make HTTP request: Get "https://localhost:443/?84": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.85] make HTTP request: Get "https://localhost:443/?85": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.86] make HTTP request: Get "https://localhost:443/?86": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.87] make HTTP request: Get "https://localhost:443/?87": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.88] make HTTP request: Get "https://localhost:443/?88": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.89] make HTTP request: Get "https://localhost:443/?89": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.90] make HTTP request: Get "https://localhost:443/?90": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.91] make HTTP request: Get "https://localhost:443/?91": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.92] make HTTP request: Get "https://localhost:443/?92": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.93] make HTTP request: Get "https://localhost:443/?93": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.94] make HTTP request: Get "https://localhost:443/?94": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.95] make HTTP request: Get "https://localhost:443/?95": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.96] make HTTP request: Get "https://localhost:443/?96": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.97] make HTTP request: Get "https://localhost:443/?97": timeout: no recent network activity
    packetconn_test.go:164: [HTTP/3-443.98] make HTTP request: Get "https://localhost:443/?98": context deadline exceeded
    packetconn_test.go:164: [HTTP/3-443.99] make HTTP request: Get "https://localhost:443/?99": context deadline exceeded
{"level":"debug","ts":1756644352.3329525,"logger":"events","msg":"event","name":"stopping","id":"227e8bc6-c597-46dd-9f29-85b9707d8d39","origin":"","data":null}
{"level":"info","ts":1756644352.3329525,"logger":"http","msg":"servers shutting down with eternal grace period"}
--- FAIL: TestPCW (15.64s)

I decided to leave aside Curl and Q tests for a while to focus on pure Go tests. Will repeat those tests once we no longer face issues in automatic tests.

vnxme · 2025-08-31T17:43:02Z

I copied commits f899c48, 9c3c912 and a552a5c into another branch based on the current master, i.e. without any changes in server.go related to the packet conn wrapper implementation, and TestUDP passes for me all the time.

WeidiDeng · 2025-09-01T02:09:56Z

@vnxme Can't you debug this yourself? This is your pr, not mine. I already implemented those channels since you said you can't. The rest is up to you.

WeidiDeng · 2025-09-01T02:10:40Z

I can help you if you're not sure about something. Not do them for you.

vnxme · 2025-09-01T18:27:59Z

@WeidiDeng, fair enough. Thanks again for your time and effort! And sorry, if anything I wrote above seemed to you as an abuse of your kindness and readiness to help that I deeply appreciate.

vnxme · 2025-09-01T19:18:46Z

Just leaving it here for myself and anyone who wants to help with testing and/or development.

TestUDP works fine now, at least I've faced no more panics yet.
TestPCW doesn't seem OK: after a number of packet exchanges or requests there seems to be a congestion inside Caddy blocking any further responses of any kind (HTTP/3, DNS, UDP).
Curl and Q tests still show inconsistent results: they usually pass, but sometimes fail. E.g. the first Curl test after Caddy start fails, Q tests run after a few successful Curl tests do fail as well.

vnxme added 3 commits August 14, 2025 11:27

packet conn wrapper: Adjust requirements (temporarily)

c7fff1f

packet conn wrapper: Backbone

f81fce3

packet conn wrapper: Integration tests

4163c3c

implement packet conn handler

c316a5b

WeidiDeng and others added 3 commits August 18, 2025 08:21

set idle timeout

848137d

Merge branch 'master' into packet-conn-wrapper

1fd23a0

packet conn wrapper: Create Server before populating its fields

af51a04

WeidiDeng added 2 commits August 25, 2025 10:27

create server only if needed

b7bbad4

embed configuration parameter in PacketConnWrapper, use server only t…

1def5ee

…o handle connections

vnxme added 4 commits August 25, 2025 17:55

packet conn wrapper: Bump Go and Caddy

a345fd2

Merge branch 'master' into packet-conn-wrapper

47dae02

# Conflicts: # go.mod # go.sum

packet conn wrapper: Restore field comments

a3a26e0

packet conn wrapper: Rearrange code

970900f

WeidiDeng added 2 commits August 26, 2025 08:52

use atomic to count the number of goroutines handling packet associat…

7900c19

…ions

Merge remote-tracking branch 'vnxme/packet-conn-wrapper' into fork/vn…

6d0c5b6

…xme/packet-conn-wrapper

vnxme added 3 commits August 31, 2025 15:12

packet conn wrapper: Add complex tests

f899c48

Merge branch 'master' into packet-conn-wrapper

6c0830b

packet conn wrapper: Minor test adjustments

9c3c912

packet conn wrapper: Fix UDP response checker

a552a5c

close channel only once

8c9217c

vnxme added 3 commits September 1, 2025 22:01

packet conn wrapper: Fix UDP part of PCW test

503e36a

packet conn wrapper: Disable DNS part, enable UDP part of PCW test

9d0e4f5

packet conn wrapper: Rename wait to delay

ed1d58e

vnxme added the help wanted Extra attention is needed label Sep 1, 2025

Uh oh!

Add packet conn wrapper support (WIP) #318

Are you sure you want to change the base?

Add packet conn wrapper support (WIP) #318

Uh oh!

Conversation

vnxme commented Aug 14, 2025

Uh oh!

WeidiDeng commented Aug 14, 2025

Uh oh!

WeidiDeng commented Aug 16, 2025

Uh oh!

vnxme commented Aug 16, 2025

Uh oh!

vnxme commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WeidiDeng commented Aug 25, 2025

Uh oh!

vnxme commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WeidiDeng commented Aug 26, 2025

Uh oh!

vnxme commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WeidiDeng commented Aug 27, 2025

Uh oh!

vnxme commented Aug 27, 2025

Uh oh!

vnxme commented Aug 31, 2025

Pure UDP

PacketConnWrapper

Uh oh!

vnxme commented Aug 31, 2025

Uh oh!

WeidiDeng commented Sep 1, 2025

Uh oh!

WeidiDeng commented Sep 1, 2025

Uh oh!

vnxme commented Sep 1, 2025

Uh oh!

vnxme commented Sep 1, 2025

Uh oh!

Uh oh!

vnxme commented Aug 20, 2025 •

edited

Loading

vnxme commented Aug 25, 2025 •

edited

Loading

vnxme commented Aug 27, 2025 •

edited

Loading