Cannot upload any file over 6MB to self-hosted supabase storage via dashboard

[fgh-james]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

When uploading a file greater than 6MB in size to storage via the supabase dashboard in a self-hosted instance I receive the following error:

Failed to upload 6_3mb.pdf: tus: failed to resume upload, caused by [object ProgressEvent], originated from request (method: HEAD, url: http://localhost/upload/resumable/cHN5Y2hvc29jaWFsX2NvbnRlbnQvRkdIIGJ1c2luZXNzIGNhcmRfMjAyNDA4MTkgY29weSA2LnppcC9hMjNmYTM5Yy1kZmI0LTRhMjUtYjQzYS1iMDU2OGY2MGM4NGU, response code: n/a, response text: n/a, request id: n/a)

While the file is uploading, I receive this error multiple times in the browser console

HEAD http://localhost/upload/resumable/cHN5Y2hvc29jaWFsX2NvbnRlbnQvRkdIIGJ1c2luZXNzIGNhcmRfMjAyNDA4MTkgY29weSA2LnppcC9hMjNmYTM5Yy1kZmI0LTRhMjUtYjQzYS1iMDU2OGY2MGM4NGU net::ERR_CONNECTION_REFUSED

I made sure to pull the latest docker images. This does not resolve the error.

To Reproduce

1. Create a file that is 6MB in size
2. Go to the self-hosted supabase dashboard: localhost:8000
3. Go to storage, and upload the file
4. It will show its uploading for a moment, then error out with the error above.

Expected behavior

Files up to 50MB should be able to be uploaded.

Screenshots

While uploading

Error

System information

• OS: macOS
• Browser (if applies): chrome
• Version of supabase docker compose: 3.8
• Version of Node.js: v18.19.0

Additional information

• I have tried this with other files, not just the one I am testing with. I receive the same error for anything over 6MB

[cjlryan]

I'm also having this issue, @fgh-james that your console error is also missing the port used for uploading on local, same as mine (usually it would be http://localhost:54321 rather than just http://localhost).

This looks similar to this bug: supabase/supabase#15703

A potential temporary solution might be to downgrade a couple of versions, mine only stopped working after updating to the latest one (CLI version 1.192.5, Storage API 1.11.0)

[cjlryan]

@fgh-james so... I fixed this for me.

It may be a mismatch between different docker image versions. I deleted then reinstalled all the docker images and it works perfectly again.

I hope that works for you!

[yaschet]

I'm experiencing the same problem as mentioned in this issue: #563
I tried the solution suggested by @cjlryan but it didn't work.

[yaschet]

Update: After a thorough investigation, I successfully resolved the issue in my case. The problem was traced back to the supabase/config.toml file, where TLS was enabled. Disabling TLS definitively fixed the issue for me. You can find the solution here: my comment.

[djsisson]

storage is returning an invalid location header

request sent to https://supabase.domain.com/storage/v1/upload/resumable
response contains Location header of
http://supabase.domain.com:8000/upload/resumable/............
this is then set in local storage where uploads are then tried to send to:

this then causes an error either due to content mismatch (trying to post to http from https or 404 since its trying to send to an invalid port plus missing /storage/v1)

i assume the bug is in here?

storage/src/http/routes/tus/lifecycle.ts

Line 86 in f950fc4

export function generateUrl(

the request comes in from kong which is http.. :8000 and no path prefix of /storage/v1

[rickypid]

I also have the same problem reported by @djsisson

[djsisson]

I want to clarify:

This works fine if you call the storage container directly, however if your containers are behind a proxy (like kong, or even a double proxy, then this method of generating a URL doesn't work)
The Simplest solution is to have an optional ENV like TUS_URL where we can set the full url ourselves and it just appends the id onto the end.

However there is a work around, depending on your current setup this is what needs to be done:

if you are behind a proxy, you need to move Kong onto port 80 to prevent the code from appending a port onto the url, so add this to your kong compose:

      - 'KONG_PROXY_LISTEN=0.0.0.0:80 reuseport backlog=16384, 0.0.0.0:443 http2 ssl reuseport backlog=16384'

This is the defaults but moved to 80/443

update the following in the studio section

- 'SUPABASE_URL=http://supabase-kong:80'

since kong is no longer on 8000

Next add the following to the storage compose section

- NODE_ENV=production

This will force the url to be https, regardless if the proxy you are using is http or https

Lastly you need to add a section to the kong yaml file to filter /uploads to storage as well

  ## Storage routes: the storage server manages its own auth
  - name: storage-tus
    _comment: 'Storage: /upload/* -> http://supabase-storage:5000/upload/*'
    url: http://supabase-storage:5000/
    routes:
      - name: storage-tus-all
        strip_path: false
        paths:
          - /upload/
    plugins:
      - name: cors

with this, storage now generates valid location header urls (albeit missing /storage/v1) but that is picked up by the added section in kong.yml

[rickypid]

Thank you @djsisson for your quick reply!!!

My situation is as follows:
Client Dart -> Ngnix Rever Proxy -> Supabase Self-Hosted [ Kong -> Storage ]

I have the same problem as described above: I make a request on https://myhost/storage/v1/upload/resumable, in the Location Header of the response I get http://myhost/storage/v1/upload/resumable/dGVzdC1leHBsb3Jlci90ZXN0LzQxMTQ3OTctdWhkXzM4NDBfMjE2MF8yNWZwcy5tcDQvODdlZDFkODYtYjZhMS00NTJhLWJkOGEtMmY3MTdlYzk1NTAx

[rickypid]

I solved it by inserting in my proxy rever configuration 🍾 🍾 🍾 :

proxy_redirect http://myhost:8000/ https://myhost/;

[rallisf1]

I spend a full day on this, tried every possible fix I could find. I couldn't get rid of the 8000 port number in the request, so I used it! My dirty hack:

kong.yml

  ## Storage routes: the storage server manages its own auth
  - name: storage-v1
    _comment: 'Storage: /storage/v1/* -> http://storage:5000/*'
    url: http://storage:5000/
    routes:
      - name: storage-v1-all
        strip_path: true
        paths:
          - /storage/v1/
    plugins:
      - name: cors
      - name: request-transformer
        config:
          add:
            headers:
              - 'Forwarded: host=\$(headers.host)/storage/v1;proto=https'

  ## Fix Storage large uploads
  - name: storage-fix
    _comment: 'Storage: /storage/v1:8000/* -> http://storage:5000/*'
    url: http://storage:5000/
    routes:
      - name: storage-fix-all
        strip_path: true
        paths:
          - /storage/v1:8000/
    plugins:
      - name: cors

It's ugly AF but it works and I can finally go to sleep.

P.S. Kong might be a great Enterprise choice but the FOSS version is limited and most people are not familiar with it. I'm pretty sure this bug is left in here on purpose (you can find issues over a year ago!) while the SaaS uses request-transformer-advanced to fix this in a single rule...

[GrizliK1988]

looks like it's already fixed in storage, but has not yet been fully updated to docker-compose of studio.

which this patch upload works for me locally

diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index 847aae8cef..47ea2b7465 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -225,7 +225,7 @@ services:
   # To use S3 backed storage: docker compose -f docker-compose.yml -f docker-compose.s3.yml up
   storage:
     container_name: supabase-storage
-    image: supabase/storage-api:v1.11.13
+    image: supabase/storage-api:v1.13.3
     depends_on:
       db:
         # Disable this if you are using an external Postgres database
@@ -263,6 +263,7 @@ services:
       GLOBAL_S3_BUCKET: stub
       ENABLE_IMAGE_TRANSFORMATION: "true"
       IMGPROXY_URL: http://imgproxy:5001
+      REQUEST_ALLOW_X_FORWARDED_PATH: "true"
     volumes:
       - ./volumes/storage:/var/lib/storage:z

Hope it works for you.