benchmark: add scenarios for the request shapes the suite never covered - #368
Closed
nigrosimone wants to merge 1 commit into
Closed
benchmark: add scenarios for the request shapes the suite never covered#368nigrosimone wants to merge 1 commit into
nigrosimone wants to merge 1 commit into
Conversation
grep -r "res.json\|req.params\|req.query" over benchmark/scenarios returned nothing. The three most common things an Express application does were not measured at all, while five of the twelve rows sat on workloads dominated by zlib, sha256, JSON.parse or loopback bandwidth, where the ratio is capped by arithmetic no matter what either framework does. routing/api-endpoint: GET /api/users/:userId/posts?fields=...&limit=10 through a mounted router, answering with res.json(). Param extraction, query parsing and serialisation - the shape where the framework's own work is a real share of the request. middlewares/realistic-stack: helmet + cors + cookie-parser + json + morgan, which is what a service actually mounts, as opposed to middlewares-100's 100 no-ops. morgan writes to a sink so the row measures the formatting every request pays rather than the runner's terminal, and 'combined' is used because that is the production default. middlewares/body-json-4kb: a body-parser row at a size an API actually receives. The 512 KiB one is kept as the stress case - it measures JSON.parse, this one measures getting the bytes to it. connections/high-concurrency: 1000 connections against a trivial handler. Every other scenario runs at 50-200, where connection handling is free, so nothing exercised the part where uWS differs from node:http structurally. All four verified to return identical status and body on both frameworks. Local preview, 5 paired runs on a machine whose ratios run about a third of CI's: api-endpoint 1.88x, body-json-4kb 1.19x, realistic-stack 1.12x.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three most common things an Express application does were not measured at all. Meanwhile five of the twelve rows sit on workloads dominated by
node:zlib,node:sha256,JSON.parseor loopback bandwidth, where the ratio is capped by arithmetic, roughly 1.01x for the streaming rows and 1.02x for the 512 KiB JSON one, no matter what either framework does. The table currently reads as "these frameworks are mostly equivalent", when what it mostly measures is work neither of them performs.This adds four rows for shapes that were missing, chosen for how common they are rather than for how they score.
routing/api-endpointGET /api/users/:userId/posts?fields=id,title,author&limit=10through a mounted router, answering withres.json(). Param extraction, query parsing and serialisation, the shape where the framework's own work is a real share of the request rather than a rounding error.middlewares/realistic-stackhelmet + cors + cookie-parser +
express.json()+ morgan, which is what a service actually mounts, as opposed tomiddlewares-100's hundred no-ops. morgan writes to a sink so the row measures the formatting every request pays rather than the runner's terminal, andcombinedis used because that is the production default, it includes the remote address, which each framework resolves its own way.Worth saying plainly: this row is not favourable, and that is the point. Locally it comes out at 1.12x, because helmet, cors, cookie-parser and morgan are the same code on both sides. A realistic middleware stack is largely shared work, the same way zlib is. It is the row most relevant to somebody deciding whether to adopt, and it should say what it says.
middlewares/body-json-4kbA body-parser row at a size an API actually receives. The 512 KiB one is kept as the stress case, that one measures
JSON.parse, this one measures getting the bytes to it.connections/high-concurrency1000 connections against a trivial handler. Every other scenario runs at 50–200, where connection handling is free, so nothing in the suite exercised the part where uWS differs from
node:httpstructurally rather than in the request path. Two wrk threads on a 4 vCPU runner leaves the server two; at this connection count some of what is measured is the generator, and the comment in the scenario says so.Benchmark Comparison