Upload performance is less impressive than download

[jgmission]

I deployed one chat server and opened one chat client in different countries. I used this single client to profile download and upload performance. The download speed could reach about 15Mbps, which is OK. But the upload speed never reached 1Mbps. The server continued processing messages for some time after client finished sending all the messages. No message was lost on the server side. It seems there is some throttling for upload.

If there is throttling, where can I find out the code for throttling? If there is not, how can I improve the upload speed?

Thank you.

[PhilipRoman]

There are a lot of factors that could cause this. Does your server do any significant work in onMessage? If yes, try offloading the work to another thread using a blocking queue or an executor service (or just comment it out for sake of testing).

If that doesn't fix the issue, you should create a minimal example that demonstrates the issue and post it here so we can investigate it.

There shouldn't be any throttling done in this library.

[jgmission]

Here is the example. I made minimal changes to ChatServer and ChatClient. If the server and client are in different countries, you will see the upload speed is less than 1Mbps. The sending call runs very quickly, but the messages are kept in memory on Client side and processed slowly.

example

I used WireShark to compare this example with some online speed test. It seems ChatClient gets ACK packets from ChatServer much more frequently than online speed test.

[PhilipRoman]

If my measurements are correct, the example sends 60 Mbps of websocket data, which is... a lot. Keep in mind that each message on console represents 120000 bytes sent.

The only way to see if the bottleneck is in this library is to check the TCP upload speed between both machines using something like iperf and compare it with websocket throughput.

For me, the TCP upload speed between my laptop and server is 14 Mbps while the websocket server can keep up with 6 Mbps.

[jgmission]

Yes, console prints out one line per 100 send() calls, which send 120000 bytes. In my testing, the client console printed out the lines very quickly, which gave me the impression that the upload was very fast. But in fact, most messages were kept in memory by the client and the memory usage kept going up. The server received about 100 messages per second and the server console printed out the lines at a much slower speed - about one line per second. You can check out the screenshot of my testing.

Timestamp is printed out in console now after I made minor code change: example

[PhilipRoman]

The send call on client side doesn't block. It creates the frame and pushes it to a linked blocking queue.

[jgmission]

So the blocking queue is not consumed in a timely manner, is it? How can I improve the throughput?

[PhilipRoman]

Have you checked the actual TCP throughput? Are you sure you aren't simply limited by internet speed between the two machines?

[jgmission]

If I remove line 817 (ostream.write(buffer.array(), 0, buffer.limit());) in WebSocketClient.java, the blocking queue is consumed fast enough. So it seems the bottleneck is in the Socket.

The upload speed of my internet connection is 500Mbps, and the download speed of the other machine is also hundreds of Mbps.

I run speed test in web browser and the upload speed is 300Mbps. Wireshark shows just one tcp connection to the server during upload testing.

[jgmission]

I deployed a netty server, then used the same ChatClient, the upload speed was 30Mbps.

Everything is the same except the websocket server.

[PhilipRoman]

Okay, that sounds pretty bad. I'll try to profile the library and see what I can find.

[dpingg]

@PhilipRoman did you find anything? I also encounter this issue.

[dpingg]

It turns out for the client -> server message, the data needs to be masked (XOR), so the upload performance is not as good as download performance.

[MouseKing1993]

Has the problem been solved now

[soylomass]

Any news about this?

[PhilipRoman]

I have reproduced the issue. Still investigating the root cause.
Some observations:
sending kilobyte-sized messages results in exactly one write(2) syscall with slightly more than 1000 bytes.
When the server receives the upload, read(2) returns relatively small chunks - about 1400 bytes.
However when the client receives the download, read(2) returns much more data - 16kb per read.
This should not be enough to cause the huge difference in speed but I'm still not sure yet.
Neither server nor client are cpu bound.
It also happens on localhost, although the effect is less extreme.

[marci4]

@PhilipRoman you are looking for something like

Java-WebSocket/src/main/java/org/java_websocket/server/WebSocketServer.java

Line 565 in bdf928b

socket.setReceiveBufferSize(WebSocketImpl.RCVBUF);

?

[marci4]

@PhilipRoman could you check if https://github.com/marci4/Java-WebSocket-Dev/tree/Issue1117 helps?

[claraccio]

Hello, has there been any update on this problem?