Buffering capacity 16777216 exceeded

[Chaho12]

Related Code

• https://github.com/airlift/airlift/blob/master/http-client/src/main/java/io/airlift/http/client/HttpClientConfig.java#L76
• https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/proxyserver/ProxyResponseHandler.java

If result from Trino is large, larger than default 16mb, a buffering capacity error would occur by the http client.
A hard limit was imposed for logging, but this does not solve the issue.

Suggested Solution

1. Increase proxy.http-client.max-content-length beyond your largest expected response. Ensure heap sizing matches.

Pro. Simple http client option to use.
Con. Gateway uses great amount of memory.

2. Re-enable compression (keep/forward Accept-Encoding: gzip) to reduce payloads.

Add Gzip(or custom) format when making request.
https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/proxyserver/ProxyRequestHandler.java#L168-L176

3. Change the proxy to stream responses instead of buffering, so large result sets don’t need to fit in memory.

@oneonestar Wdyt? On the long run, option 3 sounds best but I think option 2 is also needed.

[andythsu]

I think overall this makes sense

[Chaho12]

Previously, enconding/decoding was removed because we had to migrate to Airlift quickly.
If we add compression, we need to test for various encoding.

Also, Trino should not response with that large size. It maybe issue with Trino conf so gotta checkout that part.

[RoeyoOgen]

@Chaho12 did you check :?

Also, Trino should not response with that large size. It maybe issue with Trino conf so gotta checkout that part.

we didn't manage to find anything from Trino's side :(

after encountering this problem ourselves, we were thinking to patch with proxy.http-client.max-content-length=50mb

[Chaho12]

Default 16mb is not that big and we increased to 128mb, which lead to increase in Trino Gateway's memory.
Simple and fast fix, but does not resolve the root cause. But we may need to increase default too.
When you encounted the problem, was your query output large and/or run query in trino with large memory?

[RoeyoOgen]

We had a column value which returned lots of data (above the 16mb)