High xray client memory consumption for xhttp/http2 server

[darkk]

Integrity requirements

• I have read all the comments in the issue template and ensured that this issue meet the requirements.
• I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
• I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
• I searched issues and did not find any similar issues.
• The problem can be successfully reproduced in the latest Release

Description

I've investigated high memory consumption of an iOS Network Extension (NE) for an app based on xray-core and found an interesting quirk specific to HTTP/2.

The overall stack pins significant amount of RAM, 0.5+ MiB per conn, to maintain idle(?) connections.

The largest heap user (inuse_space per pprof) was http2.(*clientStream).writeRequestBody function:

or (another crash of the same kind)

These goroutines were blocked on io.(*pipe).Read(). OOM was happening as soon as there were ≈25 of them. I suspect, that's the Pipe() from splithttp.Dial():

Xray-core/transport/internet/splithttp/dialer.go

Line 441 in fdb9b61

reader, writer := io.Pipe()

The writeRequestBody() comes from http2 client in x/net/http2/transport.go. It allocates some memory for scratch buffer to read from Pipe:
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/transport.go#L1506-L1515

writeRequestBody() does a reasonable thing. The API author probably assumed, that the pipe is a "local" (e.g. file), so it was unexpected for the x/net/http2 code to block for a long time, waiting for Pipe to be written to.

Content-Length is not known in advance in xHTTP, so the frameScratchBufferLen() defaults to min(512KiB, SETTINGS_MAX_FRAME_SIZE):
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/transport.go#L1451

Default SETTINGS_MAX_FRAME_SIZE seems to be 1MiB Go x/net/http2. It's http2.defaultMaxReadFrameSize and server sends {SettingMaxFrameSize, conf.MaxReadFrameSize} as a part of SETTINGS frame to avoid default 16 KiB limit:
https://github.com/golang/net/blob/8ecbaa95fea823c19fa74c5c3b53e0bccd473828/http2/http2.go#L85

So, if I understand the case correctly, each idle connection pins at least 512 KiB of buffers on the client side just to maintain possibility to upload data via possibly-idle HTTP/2 stream.

#4749 expressed concerns about MaxReadFrameSize tuning in high-bandwidth scenarios. I've done two tests with iPhone 12 running xray-core with frameScratchBufferLen reduced from 512 KiB to 16 KiB. Both tests were using landline connections, with Wi-Fi 6, running on top of Gigabit Ethernet

• link soft-capped at 100 Mbit/s - iPhone was still able to saturate 100 Mbit/s just fine, OOKLA's Speedtest resulted in 110 Mbit/s (upload)
• link soft-capped at 800 Mbit/s - OOKLA's Speedtest uploads as 215...230 Mbit/s

The 512-to-16 KiB patch changed heap profile:

The thing that changes the most is the number of goroutines waiting in writeRequestBody right before OOM: the number went from ≈25 to ≈200. Yet, this number looks strange on its own and it's another thing I'm going to investigate.

The takeaways are the following:

• xray-core might provide an option to tune MaxReadFrameSize for the h2 listener and, maybe, other options of http2.Server/http2.Transport, it'll not help when h2 is terminated at a CDN, but will help self-hosters with REALITY
• iOS apps maintainers may want to patch frameScratchBufferLen() in Go http2 library to reduce per-connection overhead
• Go http2 library might provide a setting to cap frameScratchBufferLen and/or provide a better platform-specific default
• mux seems to have potential to help with such setups

Reproduction Method

My test-case was specific:

• feed refresh in VK app triggered a small connection spike
• it was enough to hit 50 MiB limit after a two-three refreshes
• I polled /debug/pprof/heap?gc=1 every two seconds and investigated the profile

Client config

xray-core versoin: v26.6.1, go1.26.3

Details

{
  "inbounds": [
    {
      "settings": {
        "udp": true
      },
      "port": NNN,
      "listen": "127.0.0.1",
      "protocol": "socks"
    }
  ],
  "outbounds": [
    {
      "streamSettings": {
        "xhttpSettings": {
          "path": "/"
        },
        "security": "reality",
        "realitySettings": {
          "publicKey": "...........................................",
          "serverName": "example.net",
          "spiderX": "/",
          "shortId": "",
          "fingerprint": "..."
        },
        "network": "xhttp"
      },
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "a.b.c.d",
            "users": [
              {
                "id": "ffffffff-ffff-ffff-ffff-ffffffffffff",
                "encryption": "none"
              }
            ],
            "port": 443
          }
        ]
      }
    }
  ]
}

Server config

n/a

Client log

n/a

Server log

n/a

[Fangliding]

我觉得根源在于 go 的 512 KiB 单块 buffer 对于一个 copier 太巨大了 通用的 io.Copy 都才使用 32 KiB
我开了一个 issue golang/go#79831 看 go team 是否接受减小最大 buffer size 的请求 如果不行可以在核心开这个口子

[darkk]

@Fangliding following io.Copy() lead and chaning 512 KiB to 32 KiB sounds like an excellent idea to me!

I thought about writing a proposal, suggesting MaxWriteFrameSize* knob, but yours feels more elegant.

*) as MaxReadFrameSize comes from the server and h2 doesn't really "negotiate" a common value for it.

[Fangliding]

不长时间持有大块buffer其实和发出大FrameSize并不冲突(RFC的考虑) 比如xray中大量使用的向量化IO是通过传递数个小的buffer实现的 这就允许传递大块payload而中间件不必长时间持有那么大的buffer 只是这和go选择的IO模式不一样

[darkk]

Pardon me being ambiguous. I was talking about hypothetical proposal of MaxWriteFrameSize knob for x/net/http2, not for xray-core.

You're right that scatter-gather I/O and explicit "readable" / "writable" signaling does not need pinned buffer at all. That's curse of blocking io.Reader() interface.