feature request: implement Endpoint-Independent Mapping for UDP #77

Lanius-collaris · 2024-07-28T03:23:49Z

I know two methods:

use unconnected UDP sockets
bind multiple UDP sockets to one address

This mapping behavior makes NAT traversal between two rethink users possible.

ignoramous · 2024-07-28T16:13:29Z

Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).

This is not going to be trivial though a few projects like SagerNet did once implement this (unsure if it worked as they don't seem to use it anymore).

https://github.com/SagerNet/LibSagerNetCore/blob/1fce969ea5/gvisor/udp.go (old) / https://github.com/SagerNet/sing-tun/blob/aecfc190f4/stack_gvisor_udp.go (new).

Welcome PR, for sure. :D

Also: xjasonlyu/tun2socks#177

ignoramous · 2024-07-29T07:56:48Z

bind multiple UDP sockets to one address

Curious, how does this work?

Lanius-collaris · 2024-07-29T08:59:36Z

bind multiple UDP sockets to one address

Curious, how does this work?

Set two sockopt, SO_REUSEADDR and SO_REUSEPORT.

Lanius-collaris · 2024-07-30T18:44:10Z

Firestack have to give up its reliance on gonet.UDPConn

Why? I am not familiar with gvisor, but its document claims gonet.DialUDP() can create unconnected UDPConn.
https://pkg.go.dev/gvisor.dev/[email protected]/pkg/tcpip/adapters/gonet#DialUDP

ignoramous · 2024-07-30T19:24:51Z

The problem isn't port forwarding (if I may call it that) using gonet.DialUDP, the problem is, the protocol handlers firestack sets up (for TCP and UDP) are not called for unconnected UDP (and listening TCP) sockets.

Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).

Don't think even the SagerNet code I shared above would work. In that,

Either: Android does not forward packets from unconnected sockets to the VPN tunnel:
Testing this may be easy: Listening on unspecifiedIP:port (over both UDP, TCP) using nc+Termux, say, would not generate any relevant logs in firestack (in Rethink, Configure -> Settings -> Log level must be set to Very Verbose).
Or: firestack must handle UDP on its own and not rely on netstack at all. That is, instead of delivering the UDP packet (injecting it in) to netstack:

firestack/intra/netstack/fdbased.go

Line 505 in 7ea0539

d.DeliverNetworkPacket(protocol, pkt)

... deal with it as-is.

Lanius-collaris · 2024-07-30T19:56:12Z

Android does not forward packets from unconnected sockets to the VPN tunnel

This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.

ignoramous · 2024-07-30T20:12:22Z

This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.

Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?

Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, ¹ always announce UDP instead ²?

What could we do for TCP ³? Port forwarding?

https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 ↩
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 ↩
Proxies (severs) currently have to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 ↩

Lanius-collaris · 2024-07-30T20:58:25Z

Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?

On Linux you can use inotify API to monitor /proc/net/udp and /proc/net/tcp, without some permissions this method may not work on Android.

Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?

Yes.

What could we do for TCP 3? Port forwarding?

Footnotes

1. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 [↩](#user-content-fnref-0-044726a13dad48d457abd69cf7035293)

2. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 [↩](#user-content-fnref-1-044726a13dad48d457abd69cf7035293)

3. Proxies (severs) currently _have_ to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 [↩](#user-content-fnref-2-044726a13dad48d457abd69cf7035293)

Allow users to write some port forwarding rules? ( e.g. TCP 0.0.0.0:12346 -> 127.0.0.1:12345 )

ignoramous · 2024-07-30T21:22:34Z

Yes.

Guess, Proxy.Announce would need to be paired with gonet.DialUDP instead of gonet.NewUDPConn?

firestack/intra/netstack/udp.go

Line 149 in 7ea0539

g.c.Store(gonet.NewUDPConn(wq, endpoint))

Linux ... inotify ... procfs

I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).

Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)

SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:

(multicast filter) SagerNet/sing-tun@150b116
(broadcast filter) SagerNet/sing-tun@da350ec

Lanius-collaris · 2024-07-31T09:57:45Z

Guess, Proxy.Announce would need to be paired with gonet.DialUDP instead of gonet.NewUDPConn?

firestack/intra/netstack/udp.go

Line 149 in 7ea0539

g.c.Store(gonet.NewUDPConn(wq, endpoint))

gonet.DialUDP() calls gonet.NewUDPConn() internally.
See https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588

I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).

Ah, I think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes don't have the same effective UID.

Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)

SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:
* (multicast filter) [SagerNet/sing-tun@150b116](https://github.com/SagerNet/sing-tun/commit/150b1162316cb63bbbf5c3c01f3a2b494a56e72d)

* (broadcast filter) [SagerNet/sing-tun@da350ec](https://github.com/SagerNet/sing-tun/commit/da350ecc726daa502b4256a7a70d70a4c6cde6d4)

I guess these two commits allow packets having multicast or broadcast destination address to bypass the userspace netstack.

A off topic question: Why does firestack send SYN+ACK packets immediately after receiving SYN packets?

ignoramous · 2024-07-31T10:39:56Z

gonet.DialUDP() calls gonet.NewUDPConn() internally. See: https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588

I think, firestack needs to create a new gonet.UDPConn without using udp.ForwarderRequest.CreateEndpoint + gonet.NewUDPConn (as that makes a connected UDP socket, per docs):

firestack/intra/netstack/udp.go

Line 143 in 7ea0539

if endpoint, err := g.req.CreateEndpoint(wq); err != nil {

think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes

Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?

Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?

Why does firestack send SYN+ACK packets immediately after receiving SYN packets

netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).

firestack/intra/netstack/tcp.go

Lines 116 to 117 in 7ea0539

    
           func (g *GTCPConn) Connect() (open bool, err error) { 
        
           	rst, err := g.synack(true)

Does this break anything apart from confusing apps into thinking they've connected to remote while they may infact be blocked later due to a firewall rule?

Lanius-collaris · 2024-07-31T14:20:28Z

think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes

Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?

Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?

I don't know if apps on android are in different namespaces, 😂 but on android every app has its own user, you can check with adb shell sh -c "ps -d|cat -" . The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.
Please read https://man.voidlinux.org/man7/socket.7#SO_REUSEPORT

ignoramous · 2024-07-31T15:33:11Z

Thanks. Now, I recall reading a Cloudflare blog post about so_reuseport (mirror).

Possibly the biggest downside, though, is that a service listening on the wildcard INADDR_ANY address claims the port number exclusively for itself. Binding over the wildcard-listening socket with a specific IP and port fails miserably due to the address already being taken (EADDRINUSE).
bind(3, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 
EADDRINUSE (Address already in use)
Unless your service is UDP-only, setting the SO_REUSEADDR socket option, will not help you overcome this restriction. The only way out is to turn to SO_REUSEPORT, normally used to construct a load-balancing socket group. And that is only if you are lucky enough to run the port-conflicting services as the same user (UID). That is a story for another post.

The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.

I don't like (or understand much of) networking in general, but here we are ;)

(but no, I don't hold any special grudge on unconnected sockets)

I don't know if apps on android are in different namespaces

I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only? I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.

Btw, checking if you working on this change (so we both don't end up impl it)?

Lanius-collaris · 2024-07-31T17:58:38Z

I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only?

Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.

I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.

Try this:

package main

import (
	"flag"
	"fmt"
	"syscall"
	"time"
)

const SO_REUSEPORT = 15

func p(e error) {
	if e != nil {
		panic(e)
	}
}
func main() {
	var localAddr syscall.SockaddrInet6
	flag.IntVar(&localAddr.Port, "listen-port", 65535, "")
	flag.Parse()

	tcpSock, err := syscall.Socket(syscall.AF_INET6, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
	p(err)
	defer syscall.Close(tcpSock)
	err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1)
	p(err)
	err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, SO_REUSEPORT, 1)
	p(err)
	err = syscall.Bind(tcpSock, &localAddr)
	p(err)
	err = syscall.Listen(tcpSock, 8)
	p(err)

	nfd, from, err := syscall.Accept(tcpSock)
	p(err)
	defer syscall.Close(nfd)
	fmt.Printf("accept: nfd: %v ; from: %v\n", nfd, from)
	fmt.Println("sleeping...")
	time.Sleep(600 * time.Second)
}

Btw, checking if you working on this change (so we both don't end up impl it)?

I am not working on this change.

ignoramous · 2024-08-06T15:36:26Z

Wireguard cannot access other wireguard peers, despite Network settings rethink-app#1633

ignoramous · 2024-08-23T23:09:10Z

Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.

On Android, unsure if link-local gets sent to TUN (lo isn't, Wifi Calling isn't, Tethering isn't).

hussainmohd-a · 2024-08-30T16:03:07Z

f78f98d522be5fc4 is demuxed to 1.1.1.1, 8.8.8.8, 9.9.9.9, and 8.8.4.4 endpoints from :37288.

2024-08-30 21:32:07.422 28538-2688  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.422 28538-2688  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.8.8:53
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.426 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.426 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.447 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(0.0.0.0:0); err? <nil>
2024-08-30 21:32:07.450 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:497: I udp: mux: f78f98d522be5fc4 new assoc for 10.111.222.1:40413
2024-08-30 21:32:07.450 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.8.8:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 1} {{} 0} {{} 0}}
2024-08-30 21:32:07.450 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.8.8:53
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 9.9.9.9:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(9.9.9.9:53)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 1.1.1.1:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(1.1.1.1:53)
2024-08-30 21:32:07.452 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2695  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-3095  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: f78f98d522be5fc4 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.8.8:53/8.8.8.8:53; mux? true, uid 10671
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 2} {{} 0} {{} 0}}
2024-08-30 21:32:07.453 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 20790a938f64d070 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 9.9.9.9:53/9.9.9.9:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 3} {{} 0} {{} 61}}
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 5c44094c5361f93f (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 1.1.1.1:53/1.1.1.1:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 9.9.9.9:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 1.1.1.1:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 4} {{} 0} {{} 183}}
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: acbd3729b62afe4f (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.506 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:07.507 28538-3095  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 5} {{} 264} {{} 183}}
2024-08-30 21:32:07.507 28538-3095  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.508 28538-3095  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.508 28538-3095  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 6} {{} 396} {{} 183}}
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.511 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.512 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.525 28538-3095  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(9.9.9.9:53); err? <nil>
2024-08-30 21:32:07.527 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(1.1.1.1:53); err? <nil>
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 1.1.1.1:53; err? <nil>
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 0c296b22bbd49481 (proxy? [email protected]:1337) 10.1.204.164:42507 -> 1.1.1.1:53/1.1.1.1:53; mux? false, uid 10671
2024-08-30 21:32:07.529 28538-2692  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 9.9.9.9:53; err? <nil>
2024-08-30 21:32:07.530 28538-2695  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 1f41c4b4d08808b3 (proxy? [email protected]:1337) 10.1.204.164:45931 -> 9.9.9.9:53/9.9.9.9:53; mux? false, uid 10671
2024-08-30 21:32:08.522 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.522 28538-2697  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 7} {{} 396} {{} 244}}
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 0b2a4b8786a59117 (proxy? Exit@127.0.0.127:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:08.545 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 E  udp.go:140: W udp: proxy: 10.111.222.1:40413 ->  8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.546 28538-2695  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.546 28538-2688  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2688  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.574 28538-2695  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:08.589 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 8} {{} 660} {{} 305}}
2024-08-30 21:32:08.589 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.590 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.590 28538-2692  GoLog                   com.celzero.bravedns                 I  common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.609 28538-2692  GoLog                   com.celzero.bravedns                 I  protect.go:67: D control: netbinder: Exit: udp4(8.8.4.4:53); err? <nil>
2024-08-30 21:32:08.614 28538-2697  GoLog                   com.celzero.bravedns                 I  exit.go:53: I proxy: exit: dial(udp) to 8.8.4.4:53; err? <nil>
2024-08-30 21:32:08.614 28538-2697  GoLog                   com.celzero.bravedns                 I  udp.go:315: I udp: connect: 5049098b0031bfbe (proxy? [email protected]:1337) 10.1.204.164:42248 -> 8.8.4.4:53/8.8.4.4:53; mux? false, uid 10671
2024-08-30 21:32:08.843 28538-3107  GoLog                   com.celzero.bravedns                 I  common.go:50: D intra: 8fec78d73c5a1f36 download(792) done(read udp 10.1.204.164:41362->8.8.8.8:53: i/o timeout) b/w a(10.111.222.1:35376->8.8.8.8:53) => b(10.1.204.164:41362<-8.8.8.8:53)
2024-08-30 21:32:09.620 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:10.616 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(1): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(2): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:203: I udp: mux: 78140498e6950c35 read done n(0): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.227 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:387: I udp: mux: 78140498e6950c35 demux from 0.0.0.0:45297 => 100.75.223.185:443 closed
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:276: I udp: mux: 78140498e6950c35 unrouting... 0.0.0.0:45297 => 100.75.223.185:443
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:142: D udp: mux: 78140498e6950c35 stop
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:133: I udp: mux: 78140498e6950c35 awaiter: done
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:164: I udp: mux: 78140498e6950c35 drain: closing 0 demuxed conns
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:153: I udp: mux: 78140498e6950c35 stopped; stats: tx: 0, rx: 6250, conns: 1, dur: 123s
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692  GoLog                   com.celzero.bravedns                 I  udpmux.go:510: I udp: mux: 78140498e6950c35 (Exit) dissoc for 10.111.222.1:36198
2024-08-30 21:32:11.629 28538-2697  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:12.642 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:13.656 28538-3107  GoLog                   com.celzero.bravedns                 I  udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)

egress flows, ie flows to be handled from netstack, already come in socksified with routes setup as needed. as such, they need not be vended again. However, netstack won't necessairly know about ingress flows, ie new flows appearing on our packet conn (dialed out via some ipn.Proxy), and so netstack must be informed about those (ie, a new conn dialed into netstack to handle a new dest)

Lanius-collaris · 2024-10-04T20:44:01Z

netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).

@ignoramous
Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen.
🧐

ignoramous · 2024-10-04T21:15:30Z

Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen.

May work if in "Single Threaded" mode.

firestack/intra/settings/config.go

Lines 82 to 84 in f3a671e

    
           // SingleThreaded is a global flag to run Netstack's packet forwarder 
        
           // in a single-threaded mode. 
        
           var SingleThreaded = atomic.Bool{}

Lanius-collaris · 2024-10-20T10:31:04Z

May work if in "Single Threaded" mode.

I don't know if gvisor is in "Single Threaded" mode by default, but it works.
https://github.com/Lanius-collaris/gvisor-playground
(I use built-in fdbased.New() and built-in stack.New())

ignoramous · 2024-10-20T16:33:53Z

https://github.com/Lanius-collaris/gvisor-playground/blob/ba027236b8a490a6c7c426338d4fa965a1f49708/app/libs/aegis/stack/tcp.go#L75 think that wait might block all other processing in netstack.

It looks to me that by default netstack's fdbased runs a "single processor". The client may have to opt-in (via ProcessorsPerChannel) to run multiple of those. firestack runs 6.

Lanius-collaris · 2024-10-20T20:35:01Z

https://github.com/Lanius-collaris/gvisor-playground/blob/ba027236b8a490a6c7c426338d4fa965a1f49708/app/libs/aegis/stack/tcp.go#L75 think that wait might block all other processing in netstack.

@ignoramous
Wait() is called by another goroutine, can you see go func(){ ... }() ? I guess that you reached maxInFlight ( It is 64 in demo_cli ). Try setting a smaller timeout in DialTCPFn and increasing maxInFlight. firestack called Complete() too early, so it is unlikely that firestack will reach maxInFlight .
https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/transport/tcp/forwarder.go#L135

It looks to me that by default netstack's fdbased runs a "single processor".

default ProcessorsPerChannel >= 1
https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/link/fdbased/endpoint.go#L333-L335

if opts.ProcessorsPerChannel == 0 {
  opts.ProcessorsPerChannel = max(1, runtime.GOMAXPROCS(0)/len(opts.FDs))
}

ignoramous · 2024-10-23T20:48:42Z

Gotcha! Thanks.

Attempt: b139771

Let the handler connect or abort the conns (which depends on onFlow). The net result is that, for tcp, synacks are only sent if the handler wants to pipe the packets to remote. In cases where the conn needs to be dropped (or firewalled), rst is sent instead. For UDP, there's likely no semantic diff.

ignoramous mentioned this issue Jul 28, 2024

LAN access in VPN Lockdown mode celzero/rethink-app#1618

Closed

ignoramous referenced this issue Aug 20, 2024

udp: endpoint-independent mapping mk1

aaef320

ignoramous referenced this issue Aug 20, 2024

common,udpmux: go lint

b9f26ee

ignoramous referenced this issue Aug 22, 2024

udp: impl endpoint-independent filtering

0e8917b

ignoramous referenced this issue Aug 23, 2024

udpmux: announce on any addr not src

1efa7b1

ignoramous referenced this issue Aug 23, 2024

udpmux: assoc dst network; netip as route key

f27b7a7

ignoramous referenced this issue Aug 23, 2024

udpmux: special case, netstack always demuxes 1st conn

ed083d0

ignoramous referenced this issue Aug 23, 2024

udpmux: demux iff on egress proxy parity

b568050

ignoramous referenced this issue Aug 23, 2024

udpmux: post drain() clear all routes

b834b1e

ignoramous closed this as completed Aug 29, 2024

ignoramous referenced this issue Aug 31, 2024

udpmux: rmv vend() fn

d9344f6

ignoramous referenced this issue Aug 31, 2024

netstack/udp: fix cond for demuxing onto existing dst

fef213a

ignoramous mentioned this issue Sep 17, 2024

Complex use case discussion celzero/rethink-app#1698

Closed

ignoramous referenced this issue Oct 23, 2024

tcp,udp: use baseHandler.onInflow for ReverseProxy

3f01c73

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature request: implement Endpoint-Independent Mapping for UDP #77

feature request: implement Endpoint-Independent Mapping for UDP #77

Lanius-collaris commented Jul 28, 2024 •

edited

Loading

ignoramous commented Jul 28, 2024 •

edited

Loading

ignoramous commented Jul 29, 2024

Lanius-collaris commented Jul 29, 2024

Lanius-collaris commented Jul 30, 2024

ignoramous commented Jul 30, 2024 •

edited

Loading

Lanius-collaris commented Jul 30, 2024

ignoramous commented Jul 30, 2024 •

edited

Loading

Lanius-collaris commented Jul 30, 2024

Footnotes

ignoramous commented Jul 30, 2024

Lanius-collaris commented Jul 31, 2024

ignoramous commented Jul 31, 2024 •

edited

Loading

Lanius-collaris commented Jul 31, 2024

ignoramous commented Jul 31, 2024 •

edited

Loading

Lanius-collaris commented Jul 31, 2024

ignoramous commented Aug 6, 2024

ignoramous commented Aug 23, 2024

hussainmohd-a commented Aug 30, 2024 •

edited by ignoramous

Loading

Lanius-collaris commented Oct 4, 2024

ignoramous commented Oct 4, 2024

Lanius-collaris commented Oct 20, 2024

ignoramous commented Oct 20, 2024

Lanius-collaris commented Oct 20, 2024 •

edited

Loading

ignoramous commented Oct 23, 2024

feature request: implement Endpoint-Independent Mapping for UDP #77

feature request: implement Endpoint-Independent Mapping for UDP #77

Comments

Lanius-collaris commented Jul 28, 2024 • edited Loading

ignoramous commented Jul 28, 2024 • edited Loading

ignoramous commented Jul 29, 2024

Lanius-collaris commented Jul 29, 2024

Lanius-collaris commented Jul 30, 2024

ignoramous commented Jul 30, 2024 • edited Loading

Lanius-collaris commented Jul 30, 2024

ignoramous commented Jul 30, 2024 • edited Loading

Footnotes

Lanius-collaris commented Jul 30, 2024

Footnotes

ignoramous commented Jul 30, 2024

Lanius-collaris commented Jul 31, 2024

ignoramous commented Jul 31, 2024 • edited Loading

Lanius-collaris commented Jul 31, 2024

ignoramous commented Jul 31, 2024 • edited Loading

Lanius-collaris commented Jul 31, 2024

ignoramous commented Aug 6, 2024

ignoramous commented Aug 23, 2024

hussainmohd-a commented Aug 30, 2024 • edited by ignoramous Loading

Lanius-collaris commented Oct 4, 2024

ignoramous commented Oct 4, 2024

Lanius-collaris commented Oct 20, 2024

ignoramous commented Oct 20, 2024

Lanius-collaris commented Oct 20, 2024 • edited Loading

ignoramous commented Oct 23, 2024

Lanius-collaris commented Jul 28, 2024 •

edited

Loading

ignoramous commented Jul 28, 2024 •

edited

Loading

ignoramous commented Jul 30, 2024 •

edited

Loading

ignoramous commented Jul 30, 2024 •

edited

Loading

ignoramous commented Jul 31, 2024 •

edited

Loading

ignoramous commented Jul 31, 2024 •

edited

Loading

hussainmohd-a commented Aug 30, 2024 •

edited by ignoramous

Loading

Lanius-collaris commented Oct 20, 2024 •

edited

Loading