Response times increasing as number of concurrent requests increase, even below thread count

[jwodder]

Summary

We have a web server written using axum that was initially implemented in the hopes of improving on the performance of a related service. While measuring the axum server's performance (by making repeated concurrent requests to the server and timing the request send + response read times), we have found that the median and average request+response times increase as the number of workers used by the client for concurrent requests increases, and this happens even when this number is below the number of CPUs on the server (and thus the slowdown is not due to there being more requests than tokio workers).

I have a work-in-progress MVCE of an axum server and a request-timing client at https://github.com/jwodder/axum-hammer. Most of the endpoints in the server involve services (registered with Router::nest_service()) that parse the request path to determine what page to reply with, as this is the basic format of our production server. Running the client against the server shows the abovementioned increase in response times on both macOS and Debian Linux, but when I attempted to strip away superfluous endpoints to get a truly minimal example, the behavior disappeared on Linux but not macOS. I am therefore holding off on providing actual timing data at this point.

At the moment, I'd like to know whether this is expected and/or easily-explainable behavior of axum. The only request bottleneck on the server side I can think of would be the fact that only a single task (I assume) can listen for & respond to incoming connections, but I wouldn't expect the slowdown from that to be more than a blip. If the slowdown we're seeing is anomalous, I can provide more information in the hopes of getting to the bottom of this.

axum version

0.7.5

[mladedav]

Hi, thanks for the writeup.

The only thing I believe should be synchronous is accepting a TCP connection and cloning the Router. Path resolution (routing itself), extractors and so on should happen in a spawned task so that depends on Tokio but should happen in new threads if there isn't other work to be done. Sending response should be possible to do in parallel (well, obviously the data is sent over the same wire, but the OS should handle that part).

If you're able to reproduce it consistently, someone can surely take a look. I can't really think of any specifics to simply explain it away easily.

[jwodder]

It does seem consistently reproduceable, though I've only run the most recent version of the MVCE less than a dozen times. Is there any specific information you need from me?

[mladedav]

Oh,I thought it went away when you deleted superfluous handlers so I thought you were still looking for a more minimal reproducible example.

If I get some time I'll try to take a look.

[jwodder]

I mean, I am still looking for a more minimal MVCE, but I do have a somewhat-minimal MVCE. Other than just testing deleting every combination of endpoints from it, I don't know where to go next with it.