Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: bufio.Writer aren't garbage collected after Hijack #59567

Closed
lxzan opened this issue Apr 12, 2023 · 6 comments · May be fixed by #70756
Closed

net/http: bufio.Writer aren't garbage collected after Hijack #59567

lxzan opened this issue Apr 12, 2023 · 6 comments · May be fixed by #70756
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@lxzan
Copy link

lxzan commented Apr 12, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20.3 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/caster/Library/Caches/go-build"
GOENV="/Users/caster/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/caster/go/pkg/mod"
GONOPROXY=""
GONOSUMDB="gitlab.ustock.cc/*"
GOOS="darwin"
GOPATH="/Users/caster/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="0"
GOMOD="/Users/caster/go/src/github.com/lxzan/go-websocket-testing/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/q0/5vn9xlns1cd70d3mzcdg3jv00000gn/T/go-build966805946=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

upgrade http to websocket

What did you expect to see?

After hijacking the http connection, the bufio.Writer in the connection is garbage collected

What did you see instead?

They have never been released

@seankhliao
Copy link
Member

which bufio Reader / Writer are you referring to?
Do you have code to demonstrate the issue?

@seankhliao seankhliao changed the title affected/package: net/http net/http: bufio.Reader/Writer aren't garbage collected after Hijack Apr 12, 2023
@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 12, 2023
@lxzan
Copy link
Author

lxzan commented Apr 12, 2023

code

package main

import (
	"github.com/lxzan/gws"
	"log"
	"net/http"
	_ "net/http/pprof"
)

func main() {
	handler := new(Handler)

	upgrader := gws.NewUpgrader(handler, nil)

	http.HandleFunc("/connect", func(writer http.ResponseWriter, request *http.Request) {
		socket, err := upgrader.Accept(writer, request)
		if err != nil {
			log.Println(err.Error())
			return
		}
		go socket.Listen()
	})

	if err := http.ListenAndServe(":8000", nil); err != nil {
		log.Panic(err.Error())
	}
}

type Handler struct {
	gws.BuiltinEventHandler
}

func (c *Handler) OnPing(socket *gws.Conn, payload []byte) {
	socket.WritePong(payload)
}

func (c *Handler) OnMessage(socket *gws.Conn, message *gws.Message) {
	defer message.Close()
	socket.WriteMessage(message.Opcode, message.Data.Bytes())
}

Test Command

tcpkali -c 10000 --connect-rate 500 -r 1 -T 3000s -m 'hello' --ws 127.0.0.1:8000/connect

pprof

image

@lxzan lxzan changed the title net/http: bufio.Reader/Writer aren't garbage collected after Hijack net/http: bufio.Writer aren't garbage collected after Hijack Apr 12, 2023
@seankhliao
Copy link
Member

I believe this is working as intended.
The bufio.Writer is placed in a sync.Pool for reuse and may be cleared at a later time.
https://github.com/golang/go/blob/master/src/net/http/server.go#L2086

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale May 19, 2023
@lxzan
Copy link
Author

lxzan commented May 19, 2023

Never mind, I have implemented a websocket server based on tcp independently, and the memory footprint is significantly reduced.

@das7pad
Copy link

das7pad commented Dec 19, 2023

Hi @seankhliao, the returning of a write buffer you likely pointed at, http.response.w, is for the buffer used in chunked-encoding. It is one of two write buffers that are used as part of the request/response life cycle.

putBufioWriter(w.w)

The other write buffer is attached to the connection, http.conn.bufw, and it remains unused after hijacking.

go/src/net/http/server.go

Lines 288 to 289 in 35222ee

// bufw writes to checkConnErrorWriter{c}, which populates werr on error.
bufw *bufio.Writer

As part of the hijack process, the connection read buffer, http.conn.bufr, will be re-used and returned to the hijack caller. A new write buffer will be allocated and returned to the caller.

buf = bufio.NewReadWriter(c.bufr, bufio.NewWriter(rwc))

I do not see a need for allocating a new write buffer as part of the hijack process. After hijacking a connection, no further writes can be made via the http.response methods.
So we might as well re-use the connection write buffer and return it to the hijack caller. WDYT @seankhliao?

The net/http unit tests are happy with the following patch and BenchmarkServerHijack records a reduction of two allocations worth 4172 bytes per operation (the 4096B buffer slice and a struct for the buffer).

Patch: 25b0691

diff --git a/src/net/http/server.go b/src/net/http/server.go
index 9245778590..0ba55e8ee5 100644
--- a/src/net/http/server.go
+++ b/src/net/http/server.go
@@ -322,7 +322,7 @@ func (c *conn) hijackLocked() (rwc net.Conn, buf *bufio.ReadWriter, err error) {
        rwc = c.rwc
        rwc.SetDeadline(time.Time{})
 
-       buf = bufio.NewReadWriter(c.bufr, bufio.NewWriter(rwc))
+       buf = bufio.NewReadWriter(c.bufr, c.bufw)
        if c.r.hasByte {
                if _, err := c.bufr.Peek(c.bufr.Buffered() + 1); err != nil {
                        return nil, nil, fmt.Errorf("unexpected Peek failure reading buffered byte: %v", err)
BenchmarkServerHijack stats
goos: linux
goarch: amd64
pkg: net/http
cpu: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
               │   old.txt   │               new.txt               │
               │   sec/op    │   sec/op     vs base                │
ServerHijack-8   13.28µ ± 2%   10.84µ ± 3%  -18.36% (p=0.000 n=10)

               │   old.txt    │               new.txt                │
               │     B/op     │     B/op      vs base                │
ServerHijack-8   15.84Ki ± 0%   11.77Ki ± 0%  -25.70% (p=0.000 n=10)

               │  old.txt   │              new.txt              │
               │ allocs/op  │ allocs/op   vs base               │
ServerHijack-8   50.00 ± 0%   48.00 ± 0%  -4.00% (p=0.000 n=10)

I can open a PR with the patch in case you are happy with the approach, @seankhliao.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/634855 mentions this issue: net/http: reduce memory usage when hijacking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants