Tunnel not working after upgrade to 3.12.3

[andrewcoelho]

I updated from 3.9.4 to 3.12.3 which may or may not be related, but since then I can't get the tunnel to work.

// npx sst tunnel --stage production --print-logs

time=2025-04-03T09:46:47.644-04:00 level=INFO msg="checking for pulumi" path="/Users/andrew/Library/Application Support/sst/bin/pulumi"
time=2025-04-03T09:46:48.541-04:00 level=INFO msg="checking for bun" path="/Users/andrew/Library/Application Support/sst/bin/bun"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="starting tunnel"
time=2025-04-03T09:46:49.155-04:00 level=INFO msg="initializing project" version=3.12.3
time=2025-04-03T09:46:49.156-04:00 level=INFO msg="esbuild building" out=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.166-04:00 level=INFO msg="esbuild built" outfile=/Users/andrew/Projects/xxx/.sst/platform/sst.config.1743688009156.mjs
time=2025-04-03T09:46:49.167-04:00 level=INFO msg="evaluating config"
time=2025-04-03T09:46:49.202-04:00 level=INFO msg="config evaluated"
time=2025-04-03T09:46:49.204-04:00 level=INFO msg="checking platform"
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="checking provider" name=command version=1.0.2 compare=1.0.2
time=2025-04-03T09:46:49.205-04:00 level=INFO msg="loading home"
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="aws credentials found" region=us-east-2 profile=xxx-prod
time=2025-04-03T09:46:50.007-04:00 level=INFO msg="fetching bootstrap"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="found existing bootstrap" data="{\"version\":5,\"asset\":\"sst-asset-xxx\",\"assetEcrRegistryId\":\"xxx\",\"assetEcrUrl\":\"xxx.dkr.ecr.us-east-2.amazonaws.com/sst-asset\",\"state\":\"sst-state-xxx\",\"appsyncHttp\":\"\",\"appsyncRealtime\":\"\"}"
time=2025-04-03T09:46:50.251-04:00 level=INFO msg="loaded config" app=xxx stage=production
time=2025-04-03T09:46:50.252-04:00 level=INFO msg="INFO getting passphrase app=xxx stage=production"
time=2025-04-03T09:46:50.432-04:00 level=INFO msg="INFO pulling state app=xxx stage=production out=/Users/andrew/Projects/xxx/.sst/pulumi/xxx/.pulumi/stacks/xxx/production.json"
time=2025-04-03T09:46:51.045-04:00 level=INFO msg="starting tunnel" cmd="[sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host xxx --user ec2-user --print-logs]"
Tunnel

➜  Forwarding ranges
   10.0.4.0/22
   10.0.12.0/22
   10.0.0.0/22
   10.0.8.0/22

Waiting for connections...

time=2025-04-03T09:48:06.371-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="killing process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="process killed with term" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=INFO msg="untracked process" pid=2173
time=2025-04-03T09:48:06.372-04:00 level=ERROR msg="failed to send sigterm" pid=2173

As you can see the tunnel seemingly runs for about 2 minutes before the process is killed. During those 2 minutes, trying to connect to my database from npx sst shell --stage production and then psql -h <hostname> gives me a server closed connection error.

Trying to run tunnel again it exits immediately.

I've tried restarting my computer, tried multiple networks that are working fine, and no luck getting this to work.

additional logs from sst diagnostic:

time=2025-04-03T09:31:49.882-04:00 level=INFO msg="tunnel started"
time=2025-04-03T09:33:04.855-04:00 level=ERROR msg="failed to start tunnel" error="dial tcp xxx:22: connect: operation timed out"
time=2025-04-03T09:33:04.857-04:00 level=INFO msg="running command" command="[ifconfig utun69 destroy]"
time=2025-04-03T09:33:04.865-04:00 level=ERROR msg="failed to execute command" command="[ifconfig utun69 destroy]" 

Not sure if relevant, but for

sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host x.x.x.x --user ec2-user --print-logs

the value printed after --host does not match any of the IP's for EC2 instances that I can see in the console.

my sst.config.ts has

const vpc = new sst.aws.Vpc('Vpc', {
  bastion: true,
  nat: 'ec2',
});

ETA: I renamed my VPC and tried re-deploying. This failed, so I undid the rename on the VPC. This looks like it created some new EC2 instances in my account and the tunnel is now working and outputting an existing EC2 instance IP address after --host.

While the problem is resolved for now, I'm still curious to know why SST was trying to use the wrong IP value for --host after I upgraded to 3.12.3 and re-deployed.

[amineelmoussafer]

Same here !

[Liamandrew]

Glad I'm not the only one - this is quite an important issue to be addressed

[thdxr]

trying to recreate

[thdxr]

did any of you try an sst refresh when it wasn't working? i couldn't recreate this

[andrewcoelho]

Yeah I tried sst refresh a couple times with no luck.

[gmathieu]

I can confirm that the tunnel only works for the personal stage. I tried downgrading to previous versions, 3.11.21 and 3.9.4, running sst refresh, without luck. Help would be greatly appreciated 🙏.

[gmathieu]

I found a workaround by tunneling to your personal stage. The trick is to deploy your personal stage that points to the same VPC used in the production stage.

Here's how to reproduce:

1. Grab the production's VPC ID (skip the sub-steps if you already know it):

   1. Update sst.config.ts to expose your production's VPC ID

      export default $config({
        // ...
        async run() {
          const vpc = new sst.aws.Vpc('vpc', { bastion: true,nat: 'ec2' })
          // ...
          return { vpcId: vpc.id }
        }
      }

   2. Output the VPD ID by running sst refresh --stage production

2. Update your sst.config.ts to reuse production's VPC

   export default $config({
     async run() {
       sst.aws.Vpc.get("vpc", "VPC's ID")
       // I recommend removing all other components since we don't want to deploy to production
     }
   }

3. Deploy your personal stage: sst deploy

4. Undo any changes to sst.config.ts

5. Tunnel using your personal stage sst tunnel

6. Run whatever you need on the production stage sst shell --stage production ...

Note: both the production stage and your personal must be deployed in the same AWS account.

[Thevetat]

Getting the same on a 3.12.3 build.

[Liamandrew]

This is fixed for me after upgrading to 3.12.6 🎉

[ghardin1314]

Still running into this problem with 3.13.1. I tried the workaround to tunnel on a personal stage and my connection gets tunneled but still seems to timeout without reaching the database. Eventually it did start working. I think there may be 2 issues here

1. The tunnel on production stage is not resolving the right IP (no idea why). Seems to work on any non production stage if you use VPC.get(...)
2. I think Aurora may just take a long time to spin up? This was a new stage for me in production and after about an hour or two the connection finally made it through and I could run my migrations

[ghardin1314]

I think I narrowed it down, at least for my case. The issue seems to be with reusing the nat ec2 instance for the bastion. When I remove the nat from the vpc and it redeploys a standalone bastion instance then the tunnel works fine. When I add back the nat, the tunnel seems to not resolve the right IP for either of the nat instances

[pauljasperdev]

Not sure if relevant, but for

sudo -n -E /opt/sst/tunnel tunnel start --subnets 10.0.4.0/22,10.0.12.0/22,10.0.0.0/22,10.0.8.0/22 --host x.x.x.x --user ec2-user --print-logs

the value printed after --host does not match any of the IP's for EC2 instances that I can see in the console.

I've seen this too. Tunneling failed due to wrong IP.

I downgraded to 3.10.7 and tested with personal stage. Only after remove and redeploy the IP changed. Having a 3.12.3 deployment and tunneling with a 3.10.7 without redeploying shows the wrong IP.

EDIT:
Tried with sst tunnel --stage mystage instead of sst dev tunnel today. No luck. IP is wrong again. Tunneling to personal stage via just sst tunnel resolves a correct IP.

[natew]

I tried a ton of things - upgrading, downgrading, re-deploying, managed/nat, etc. None worked, then I restarted my Mac and it worked 🤷

[pauljasperdev]

I think I narrowed it down, at least for my case. The issue seems to be with reusing the nat ec2 instance for the bastion. When I remove the nat from the vpc and it redeploys a standalone bastion instance then the tunnel works fine. When I add back the nat, the tunnel seems to not resolve the right IP for either of the nat instances

This worked for me for production stage.

export const vpc = new sst.aws.Vpc('MyVpc', {
  bastion: true,
  nat: 'managed',
  // nat: 'ec2',
});

Changing to nat: 'managed' redeployed a bastion host and I could tunnel to it. After appling schema changes I swiched back.