Fails to connect on MIT wifi, not falling back to secondary port

[waltmck]

Issue

I am trying to connect my laptop (behind a firewall) to my server (not behind a firewall) using zerotier. My laptop is behind MIT's wifi network, which blocks most UDP ports including 9993. Port 4500 is allowed by the firewall, but even when I set secondaryPort = 4500 the devices completely fail to connect.

Diagnostic Info

zerotier-cli info -j output of my server:

{
 "address": "54e1bd21d7",
 "clock": 1755028264468,
 "config": {
  "settings": {
   "allowTcpFallbackRelay": false,
   "concurrency": 4,
   "cpuPinningEnabled": true,
   "defaultBondingPolicy": "balance-aware",
   "forceTcpRelay": false,
   "homeDir": "/var/lib/zerotier-one",
   "interfacePrefixBlacklist": [
    "wg",
    "zt"
   ],
   "listeningOn": [
    "37.27.106.131/9993",
    "37.27.106.131/4500",
    "37.27.106.131/34007"
   ],
   "multicoreEnabled": true,
   "portMappingEnabled": true,
   "primaryPort": 9993,
   "secondaryPort": 4500,
   "softwareUpdate": "disable",
   "softwareUpdateChannel": "release",
   "surfaceAddresses": [
    "37.27.106.131/9993",
    "37.27.106.131/4500",
    "37.27.106.131/34007"
   ],
   "tertiaryPort": 34007
  }
 },
 "online": true,
 "planetWorldId": 149604618,
 "planetWorldTimestamp": 1738848951118,
 "publicIdentity": "54e1bd21d7:0:b08b7ea7a326a57680dbb9ea4abdd4e413f36b0278055ea693dd1301058ce06702da58ca2f16a4182335b44464e29c599b7e112ac49e60af1b29c63e25356dd9",
 "tcpFallbackActive": false,
 "version": "1.14.2",
 "versionBuild": 0,
 "versionMajor": 1,
 "versionMinor": 14,
 "versionRev": 2
}

zerotier-cli info -j output for my laptop:

{
 "address": "53abc8ed10",
 "clock": 1755028020403,
 "config": {
  "settings": {
   "allowTcpFallbackRelay": false,
   "concurrency": 4,
   "cpuPinningEnabled": true,
   "defaultBondingPolicy": "balance-aware",
   "forceTcpRelay": false,
   "homeDir": "/var/lib/zerotier-one",
   "interfacePrefixBlacklist": [
    "wg",
    "zt"
   ],
   "listeningOn": [
    "10.189.97.143/9993",
    "10.189.97.143/33357",
    "10.189.97.143/4500"
   ],
   "multicoreEnabled": true,
   "portMappingEnabled": true,
   "primaryPort": 9993,
   "secondaryPort": 4500,
   "softwareUpdate": "disable",
   "softwareUpdateChannel": "release",
   "surfaceAddresses": [
    "192.54.222.148/22874",
    "192.54.222.148/36632",
    "192.54.222.148/9889",
    "192.54.222.148/13431",
    "192.54.222.148/46838",
    "192.54.222.148/9394",
    "192.54.222.148/47493",
    "192.54.222.148/50122",
    "192.54.222.148/23252",
    "192.54.222.148/52594",
    "192.54.222.148/37364",
    "192.54.222.148/47689",
    "192.54.222.148/15132"
   ],
   "tertiaryPort": 33357
  }
 },
 "online": true,
 "planetWorldId": 149604618,
 "planetWorldTimestamp": 1738848951118,
 "publicIdentity": "53abc8ed10:0:a6ccb9e9c13831758fb5bdb0e01b36f66e5b2c9cecc00b4cfac812c38827ca66166c5f466f68dcb498c485144a76b5c46f49128ec87a91667d7c59d41730d04f",
 "tcpFallbackActive": false,
 "version": "1.14.2",
 "versionBuild": 0,
 "versionMajor": 1,
 "versionMinor": 14,
 "versionRev": 2
}

zerotier-cli peers output of my server:

200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
53abc8ed10 1.14.2 LEAF       0 DIRECT   837      728      192.54.222.148/52594
6ab565387a 1.14.2 LEAF     119 DIRECT   15289    15170    35.209.220.36/33386
778cde7190 -      PLANET   131 DIRECT   276      204363   103.195.103.66/9993
a8f0746744 1.14.0 LEAF       0 DIRECT   6046     6046     192.54.222.136/39309
cafe04eba9 -      PLANET     0 DIRECT   30296    210412   84.17.53.155/9993
cafe80ed74 -      PLANET     0 DIRECT   30296    210274   185.152.67.145/9993
cafefd6717 -      PLANET     0 DIRECT   30296    210195   79.127.159.187/9993

zerotier-cli peers output of my laptop:

200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
54e1bd21d7 1.14.2 LEAF      -1 RELAY
6ab565387a 1.14.2 LEAF      -1 RELAY
778cde7190 -      PLANET    51 DIRECT   53       35042    103.195.103.66/9993
cafe04eba9 -      PLANET   102 DIRECT   5059     34991    84.17.53.155/9993
cafe80ed74 -      PLANET    97 DIRECT   5059     34996    185.152.67.145/9993
cafefd6717 -      PLANET   205 DIRECT   5059     34888    79.127.159.187/9993

Further information

Both port 9993 and port 4500 are open on both devices.

The connection is established successfully if I set primaryPort = 4500 on both devices. However, the connection seems weirdly flaky:

• I can ping my server from my laptop and vice versa
• I can ssh from my server into my laptop, but not vice versa
• I can wget from my server to my laptop, but not vice versa

[laduke]

Hmm

Laptop says it's connecting to the roots ("PLANET") on destination 9993/udp. I'm not sure how if the campus firewall isn't allowing 9993/udp.

It's not connecting to 6ab565387a which is one my.zerotier.com network controllers. It's listening on a random port, not 9993.

If you stop the service on your laptop, clear out the peers cache, and restart the service... does it find the roots again?
https://docs.zerotier.com/config/#system
You might want to do this between every test. Possibly on the server too.

You could disable the secondary and tertiary ports, to help narrow things down.

allowSecondaryPort: false,
portMappingEnabled: false

The campus NAT is "symmetric" nat and creating a new mapping for every attempted connection (see surface addresses). I don't think it's possible to tell from the cli output which port is being mapped. It's hard to tell what's going on.

We don't monitor here for support very closely. If possible and desired contact zerotier sales/support for better help.

The connection is established successfully if I set primaryPort = 4500 on both devices. However, the connection seems weirdly flaky:

usually a symptom of a "relay" connection vs a direct connection.

hope that helps

[waltmck]

@laduke I will do some more debugging next time I am on campus. Just to clarify, deleting /var/lib/zerotier-one/peers.d on each device completely clears the peers cache? (both devices are running Linux)

Thanks.

[laduke]

correct. as long as the service isn't during

[waltmck]

@laduke I was able to do some more troubleshooting (making sure to delete peers.d each time). If I disable port mapping and secondary port, zerotier-cli peers shows:

200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
6ab565387a 1.15.3 LEAF     125 DIRECT   6364     6319     35.209.76.88/59149
778cde7190 -      PLANET    -1 RELAY
cafe04eba9 -      PLANET    -1 RELAY
cafe80ed74 -      PLANET    -1 RELAY
cafefd6717 -      PLANET    -1 RELAY

whereas with the same config my server was able to connect to all of the planets. This seems to confirm that port 9993 is blocked, as expected. If I then set allowSecondaryPort: true and secondaryPort: 4500 (or any other port), I get the same result:

200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
6ab565387a 1.15.3 LEAF     125 DIRECT   5571     5531     35.209.76.88/59149
778cde7190 -      PLANET    -1 RELAY
cafe04eba9 -      PLANET    -1 RELAY
cafe80ed74 -      PLANET    -1 RELAY
cafefd6717 -      PLANET    -1 RELAY

However, this occurs even if I set primaryPort to 4500 or any other allowed UDP port (I also tried 1723, 123, 53, and 59149): regardless, zerotier fails to make any connections.

I'm not really sure what's going on here. One thing I noticed was that my server still uses port 9993 to connect to planets even when primaryPort is set differently---is this typical, and is it possible to change this behavior since MIT blocks outgoing traffic on port 9993?