Database connection errors for ap-southeast-2

[haydn]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Since maintenance (which I believe was an upgrade to Supavisor 2.0) was done on Supabase's ap-southeast-2 region on January 28 (https://status.supabase.com/incidents/cbzhwr1wjr5b), we have been getting {:shutdown, :db_termination} errors.

I known the upgrade for other regions had to be rolled-back (https://status.supabase.com/incidents/zk493yyz17hy), but it's unclear to me what the status of ap-southeast-2 is. Was it one of the 9 that got upgraded early? Was it rolled-back as part of the incident response?

To Reproduce

I have not been able to reliably reproduce this issue, but it's constantly occurring for our users.

Expected behavior

No errors.

Screenshots

Graph of {:shutdown, :db_termination} errors in our logs (times are using Australia/Sydney time zone):

Graph of 504 responses from our application (times are using Australia/Sydney time zone):

Recent pooler logs in Supabase dashboard:

System information

N/A

Additional context

• #567

I will also log a support request via the Supabase dashboard and link to this issue.

[linear]

POOLER-134 Database connection errors for `ap-southeast-2`

[haydn]

I'm also constantly experiencing errors when connecting to the database via an SQL client (which in the past has had no issues), so I'm pretty confident this is an issue with Supabase/Supavisor and not our app.

@abc3 @hauleth Any word on what's happening? I haven't heard back from customer support.

[hauleth]

I am looking into that. Have you filled support ticket, as we would need the organisation ID to explore the issue further?

[haydn]

@hauleth Thanks! Yep, got this response 18 hours ago:

Thank you for contacting Supabase Support. I'm forwarding this ticket to the pooler team to best assist you.

I assume that's in your inbox somewhere.

[haydn]

@hauleth Any updates? This is a really bad for us at the moment — users are constantly hitting errors and there's nothing I can do. I'm literally spending my time looking at alternatives to Supabase as a way to fix this issue. I'm especially frustrated because this isn't a one-off: #445 and #344 were both problems with the pooler that also impacted our users.

[hauleth]

We are trying our best to find what is the cause as this is one-off case. We have contacted few other teams that were encountering such issue and after the recent deployments they haven't seen that issue anymore. We have some leads, but as this contains non-public data of your project it is handled internally.

[haydn]

@hauleth Thanks, I appreciate the follow-up. You have to understand that from my perspective, the sum total of communication I'd received from Supabase on this (pretty major) issue was:

Thank you for contacting Supabase Support. I'm forwarding this ticket to the pooler team to best assist you.

I am looking into that. Have you filled support ticket, as we would need the organisation ID to explore the issue further?

That hadn't exactly filled me with confidence. The last time I heard from y'all was 5 days ago and there was no acknowledgement that you'd found the support request and had the project ID for further investigation. A bit more communication (either here of via the support request) would've gone a long way to put my mind at ease.

Let me know if there's anything else you need from me to help with the investigation.

[blechatellier]

@hauleth we have been seeing errors coming back again, just less of them to trip our alerts. We are seeing an increased number of timeouts that I suspect are related to the pooler.

[stefansaasen]

I'd like to chime in, we are seeing this a lot too. These are ~ 550 {:shutdown, :db_termination} errors in the last two weeks, starting around January 29th:

This is using aws-0-ap-southeast-2.pooler.supabase.com

[blechatellier]

@stefansaasen @haydn it might be a different issue but I've updated the pool size by +1 (from 96 to 97 of 120 available connections). It forced the poolers to rotate and metrics look healthier now.

[haydn]

As of 2025-02-11T13:00:00+11:00 this issue appears to have been fixed for us: