[Help] Troubleshooting UDP packet issues: loss / roundtrip delay

[hitech95]

I've discovered a strange issue with UDP traffic in a multiplayer videogame (I'm in ITALY 1:1 port range openwrt stable 23.04).

This is the first game I've found that is completly unable to connect to the game servers and is throwing all sort of non descriptive issues. (cant connect, timeout, etc). Basically I cannot play multiplayer on that specific game unless I use another internet connection.

I'm still digging into this issue as I'm not 100% sure on where the issue lie.

I've tried to figure out if there was some issue on the nat46 implementation but no visible error is thrown.

Setting a log error higher that 2 completly locks the router (its a x86 machine) as it is logging too much traffic to dmesg.
I'm trying to reduce the number of devices connected but a single Windows machine talks so much to the internet.

I did a pcap on the port ranges of the game and the traffic is there, it seems with the "correct" replies.
The packet body is a binary format but the packets seems to be coherent with the documentation of the SDK of such game (ack and reply to events).
The error is thrown by the SDK and not the game itself so I think that if I try with another game suing the same multiplayer library the issue might be there.
Unfortunatly I dont have the knowhow on Unity to spin up some tech demo of such SDK to validate if the issue.

The SDK is this one:
https://doc.photonengine.com/pun/current/connection-and-authentication/regions

The region discovery and initial hadshake seems to be done correctly.
So Master server, (game) nameserver and region server list in loaded succesfully.
I think it is failing on creating the room in a game server.

Looking in the code and the dmesg I cannot find a way to set a suitable log level to only get the src IPs dest IPs, protocol and src port dest port.
I think that the best way to troubleshoot this is to create re-build the module with a custom logger string.

@ayourtch
OT: Was you that were developing a 1wire bike bus/protocol some years ago?

[ayourtch]

Interesting; I would try to stare at two series of pcaps - working and non-working one, doing the same thing;

also on the CPE I would take simultaneously outside + inside pcaps (and/or on the nat46 device - then you should be able to see both legs). Not sure if the extra logging will help, since we don’t know what to look for…

The biggest thing (especially with UDP) is usually MTU; if it is coming from the Internet to you, it might be not possible to capture, but if it is from you to the Internet, then it can be possible to capture on the inside - and, as a quick hack, you can set the MTU on the inside computer itself to something like 1300, this might result in different behavior. As the joke goes, it’s either an MTU or a DNS 😂

another option is to build your own tunnel broker and see if you can reproduce the issue there:

https://stdio.be/blog/2023-08-24-full-1500-mtu-ipv6-lab/

• it’s not quite map-t, but should give you the environment where you can easily control/debug both sides. This setup is specifically aiming to deliver full 1500 end to end by doing very hacky double tunneling, but you can tweak things on both sides and see if you can reproduce with that.

Bike 1-wire protocol: yeah I started at some point, but got distracted by more shiny things 😂

[hitech95]

Interesting; I would try to stare at two series of pcaps - working and non-working one, doing the same thing;

I'll try to do that. It is not eaasy as the only thing that is readable is the info rom the UDP packet header.

The biggest thing (especially with UDP) is usually MTU; if it is coming from the Internet to you, it might be not possible to capture, but if it is from you to the Internet, then it can be possible to capture on the inside - and, as a quick hack, you can set the MTU on the inside computer itself to something like 1300, this might result in different behavior. As the joke goes, it’s either an MTU or a DNS

• I've tried to reduce the MTU to 1100 on the windows machine, the issue is still there same errors.
• I dont think it is DNS as they are using their own DNS server/system to access the right server.

So you suggest to also capture the WAN and the MAP interface on the router side while capturing on the windows amchine to compare the traffic?

[ayourtch]

Yeah, with the black box behavior like this “I don’t know what I am looking for”, the best you can hope for is make a trace of working and non working scenarios at as many points as possible and look at them to try to spot the issue. Try to use the timing to your advantage - eg keep a note when you see the user visible error relative to start of activity, and see how the packets correlate - sometimes there might be some traces. Given the corner case of this behavior, it will be something pretty subtle - maybe some information is lost in v4-v6-v4 translation and it turns out to be critical…

[hitech95]

Unfortunately the captures are quite similar. The only big difference is that on MAP-T network the packet chain is interrupted and after a few seconds a last packet arrives at the "disconnect" event.

Since there is still a bug with the vendor traceroute it doesn't work yet. Also the game servers don't seem to reply to PINGs.
So I cannot figure out more.

I've tried to contact the game studio but no reply so far.

Not sure if @r-pats can help in some way with the debug process.