Skip to content

Commit 620e96a

Browse files
preParsing hook payload manipulation and consistent Content Type Parser API (fastify#2286)
* feat: Allow preParsing to change the request payload stream. * docs: Updated documentation and examples. * test: Updated tests. * feat: Update types. * docs: Updated migration guide. * chore: Use local symbols for test internals. * docs: Typo fix. Co-authored-by: James Sumners <[email protected]> * docs: Typo fix. Co-authored-by: James Sumners <[email protected]> * docs: Markup fix. Co-authored-by: James Sumners <[email protected]> * docs: Markup fix. Co-authored-by: James Sumners <[email protected]> Co-authored-by: James Sumners <[email protected]>
1 parent 86f56ed commit 620e96a

28 files changed

+910
-245
lines changed

docs/ContentTypeParser.md

+21-17
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,23 @@ Fastify automatically adds the parsed request payload to the [Fastify request](.
99

1010
### Usage
1111
```js
12-
fastify.addContentTypeParser('application/jsoff', function (req, done) {
13-
jsoffParser(req, function (err, body) {
12+
fastify.addContentTypeParser('application/jsoff', function (request, payload, done) {
13+
jsoffParser(payload, function (err, body) {
1414
done(err, body)
1515
})
1616
})
1717

1818
// Handle multiple content types with the same function
19-
fastify.addContentTypeParser(['text/xml', 'application/xml'], function (req, done) {
20-
xmlParser(req, function (err, body) {
19+
fastify.addContentTypeParser(['text/xml', 'application/xml'], function (request, payload, done) {
20+
xmlParser(payload, function (err, body) {
2121
done(err, body)
2222
})
2323
})
2424

2525
// Async is also supported in Node versions >= 8.0.0
26-
fastify.addContentTypeParser('application/jsoff', async function (req) {
27-
var res = await new Promise((resolve, reject) => resolve(req))
26+
fastify.addContentTypeParser('application/jsoff', async function (request, payload) {
27+
var res = await jsoffParserAsync(payload)
28+
2829
return res
2930
})
3031
```
@@ -33,12 +34,16 @@ You can also use the `hasContentTypeParser` API to find if a specific content ty
3334

3435
```js
3536
if (!fastify.hasContentTypeParser('application/jsoff')){
36-
fastify.addContentTypeParser('application/jsoff', function (req, done) {
37-
// Code to parse request body/payload for the given content type
37+
fastify.addContentTypeParser('application/jsoff', function (request, payload, done) {
38+
jsoffParser(payload, function (err, body) {
39+
done(err, body)
40+
})
3841
})
3942
}
4043
```
4144

45+
**Notice**: The old syntaxes `function(req, done)` and `async function(req)` for the parser are still supported but they are deprecated.
46+
4247
#### Body Parser
4348
You can parse the body of a request in two ways. The first one is shown above: you add a custom content type parser and handle the request stream. In the second one, you should pass a `parseAs` option to the `addContentTypeParser` API, where you declare how you want to get the body. It could be of type `'string'` or `'buffer'`. If you use the `parseAs` option, Fastify will internally handle the stream and perform some checks, such as the [maximum size](./Server.md#factory-body-limit) of the body and the content length. If the limit is exceeded the custom parser will not be invoked.
4449
```js
@@ -52,7 +57,6 @@ fastify.addContentTypeParser('application/json', { parseAs: 'string' }, function
5257
}
5358
})
5459
```
55-
As you can see, now the function signature is `(req, body, done)` instead of `(req, done)`.
5660

5761
See [`example/parser.js`](../examples/parser.js) for an example.
5862

@@ -63,10 +67,10 @@ See [`example/parser.js`](../examples/parser.js) for an example.
6367
#### Catch-All
6468
There are some cases where you need to catch all requests regardless of their content type. With Fastify, you can just use the `'*'` content type.
6569
```js
66-
fastify.addContentTypeParser('*', function (req, done) {
70+
fastify.addContentTypeParser('*', function (request, payload, done) {
6771
var data = ''
68-
req.on('data', chunk => { data += chunk })
69-
req.on('end', () => {
72+
payload.on('data', chunk => { data += chunk })
73+
payload.on('end', () => {
7074
done(null, data)
7175
})
7276
})
@@ -77,7 +81,7 @@ Using this, all requests that do not have a corresponding content type parser wi
7781
This is also useful for piping the request stream. You can define a content parser like:
7882

7983
```js
80-
fastify.addContentTypeParser('*', function (req, done) {
84+
fastify.addContentTypeParser('*', function (request, payload, done) {
8185
done()
8286
})
8387
```
@@ -86,7 +90,7 @@ and then access the core HTTP request directly for piping it where you want:
8690

8791
```js
8892
app.post('/hello', (request, reply) => {
89-
reply.send(request.req)
93+
reply.send(request.raw)
9094
})
9195
```
9296

@@ -96,8 +100,8 @@ Here is a complete example that logs incoming [json line](http://jsonlines.org/)
96100
const split2 = require('split2')
97101
const pump = require('pump')
98102

99-
fastify.addContentTypeParser('*', (req, done) => {
100-
done(null, pump(req, split2(JSON.parse)))
103+
fastify.addContentTypeParser('*', (request, payload, done) => {
104+
done(null, pump(payload, split2(JSON.parse)))
101105
})
102106

103107
fastify.route({
@@ -109,4 +113,4 @@ fastify.route({
109113
})
110114
```
111115

112-
For piping file uploads you may want to checkout [this plugin](https://github.com/fastify/fastify-multipart).
116+
For piping file uploads you may want to checkout [this plugin](https://github.com/fastify/fastify-multipart).

docs/Hooks.md

+16-7
Original file line numberDiff line numberDiff line change
@@ -54,23 +54,34 @@ fastify.addHook('onRequest', async (request, reply) => {
5454
**Notice:** in the [onRequest](#onRequest) hook, `request.body` will always be `null`, because the body parsing happens before the [preValidation](#preValidation) hook.
5555

5656
### preParsing
57+
58+
If you are using the `preParsing` hook, you can transform the request payload stream before it is parsed. It receives the request and reply objects as other hooks, and a stream with the current request payload.
59+
60+
If it returns a value (via `return` or via the callback function), it must return a stream.
61+
62+
For instance, you can uncompress the request body:
63+
5764
```js
58-
fastify.addHook('preParsing', (request, reply, done) => {
65+
fastify.addHook('preParsing', (request, reply, payload, done) => {
5966
// Some code
60-
done()
67+
done(null, newPayload)
6168
})
6269
```
6370
Or `async/await`:
6471
```js
65-
fastify.addHook('preParsing', async (request, reply) => {
72+
fastify.addHook('preParsing', async (request, reply, payload) => {
6673
// Some code
6774
await asyncMethod()
68-
return
75+
return newPayload
6976
})
7077
```
7178

7279
**Notice:** in the [preParsing](#preParsing) hook, `request.body` will always be `null`, because the body parsing happens before the [preValidation](#preValidation) hook.
7380

81+
**Notice:** you should also add `receivedEncodedLength` property to the returned stream. This property is used to correctly match the request payload with the `Content-Length` header value. Ideally, this property should be updated on each received chunk.
82+
83+
**Notice**: The old syntaxes `function(request, reply, done)` and `async function(request, reply)` for the parser are still supported but they are deprecated.
84+
7485
### preValidation
7586
```js
7687
fastify.addHook('preValidation', (request, reply, done) => {
@@ -86,8 +97,6 @@ fastify.addHook('preValidation', async (request, reply) => {
8697
return
8798
})
8899
```
89-
**Notice:** in the [preValidation](#preValidation) hook, `request.body` will always be `null`, because the body parsing happens before the [preValidation](#preHandler) hook.
90-
91100
### preHandler
92101
```js
93102
fastify.addHook('preHandler', (request, reply, done) => {
@@ -114,7 +123,7 @@ fastify.addHook('preSerialization', (request, reply, payload, done) => {
114123
done(err, newPayload)
115124
})
116125
```
117-
Or `async/await`
126+
Or `async/await`:
118127
```js
119128
fastify.addHook('preSerialization', async (request, reply, payload) => {
120129
return { wrapped: payload }

docs/Migration-Guide-V3.md

+24
Original file line numberDiff line numberDiff line change
@@ -118,13 +118,37 @@ fastify.setValidatorCompiler(({ schema, method, url, httpPart }) =>
118118
);
119119
```
120120

121+
### Changed preParsing hook behaviour ([#2286](https://github.com/fastify/fastify/pull/2286))
122+
123+
From Fastify v3, the behavior of `preParsing` hook will change slightly in order to support request payload manipulation.
124+
125+
The hook now takes an additional argument, `payload`, and therefore the new hook signature is `fn(request, reply, payload, done)` or `async fn(request, reply, payload)`.
126+
127+
The hook can optionally return a new stream via `done(null, stream)` or returning the stream in case of async functions.
128+
129+
If the hook returns a new stream, it will be used instead of the original one in following hooks. A sample use case for this is handling compressed requests.
130+
131+
The new stream should add the `receivedEncodedLength` property to the stream that should reflect the actual data size received from the client. For instance, in compressed request it should be the size of the compressed payload.
132+
This property can (and should) be dynamically updated during `data` events.
133+
134+
The old syntax of Fastify v2 without payload it is supported but it is deprecated.
135+
121136
### Changed hooks behaviour ([#2004](https://github.com/fastify/fastify/pull/2004))
122137

123138
From Fastify v3, the behavior of `onRoute` and `onRegister` hooks will change slightly in order to support hook encapsulation.
124139

125140
- `onRoute` - The hook will be called asynchronously, in v1/v2 it's called as soon as a route is registered. This means that if you want to use it, you should register this hook as soon as possible in your code.
126141
- `onRegister` - Same as the onRoute hook, the only difference is that now the very first call will no longer be the framework itself, but the first registered plugin
127142

143+
### Changed Content Type Parser syntax ([#2286](https://github.com/fastify/fastify/pull/2286))
144+
145+
In Fastify v3 the Content Type Parsers have now a single signature for parsers.
146+
147+
The new signatures is `fn(request, payload, done)` or `async fn(request, payload)`. Note that `request` is now a fastify request, not an `IncomingMessage`.
148+
The payload is by default a stream. If the `parseAs` option is used in `addContentTypeParser`, then `payload` reflects the option value (string or buffer).
149+
150+
The old signatures `fn(req, [done])` or `fn(req, payload, [done])` (where `req` is `IncomingMessage`) are still supported but deprecated.
151+
128152
### Changed TypeScript support
129153

130154
The type system was changed in Fastify version 3. The new type system introduces generic constraining and defaulting, plus a new way to define schema types such as a request body, querystring, and more!

docs/TypeScript.md

+2-4
Original file line numberDiff line numberDiff line change
@@ -954,18 +954,16 @@ Notice: in the `onRequest` hook, request.body will always be null, because the b
954954
955955
preParsing` is the second hook to be executed in the request lifecycle. The previous hook was `onRequest`, the next hook will be `preValidation`.
956956

957-
Notice: in the `preParsing` hook, request.body will always be null, because the body parsing happens before the `preHandler` hook.
957+
Notice: in the `preParsing` hook, request.body will always be null, because the body parsing happens before the `preValidation` hook.
958958

959+
Notice: you should also add `receivedEncodedLength` property to the returned stream. This property is used to correctly match the request payload with the `Content-Length` header value. Ideally, this property should be updated on each received chunk.
959960

960961
##### fastify.preValidationHookhandler<[RawServer][RawServerGeneric], [RawRequest][RawRequestGeneric], [RawReply][RawReplyGeneric], [RequestGeneric][FastifyRequestGenericInterface], [ContextConfig][ContextConfigGeneric]>(request: [FastifyRequest][FastifyRequest], reply: [FastifyReply][FastifyReply], done: (err?: [FastifyError][FastifyError]) => void): Promise\<unknown\> | void
961962

962963
[src](../types/hooks.d.ts#L53)
963964

964965
`preValidation` is the third hook to be executed in the request lifecycle. The previous hook was `preParsing`, the next hook will be `preHandler`.
965966

966-
Notice: in the `preValidation` hook, request.body will always be null, because the body parsing happens before the `preHandler` hook.
967-
968-
969967
##### fastify.preHandlerHookhandler<[RawServer][RawServerGeneric], [RawRequest][RawRequestGeneric], [RawReply][RawReplyGeneric], [RequestGeneric][FastifyRequestGenericInterface], [ContextConfig][ContextConfigGeneric]>(request: [FastifyRequest][FastifyRequest], reply: [FastifyReply][FastifyReply], done: (err?: [FastifyError][FastifyError]) => void): Promise\<unknown\> | void
970968

971969
[src](../types/hooks.d.ts#L70)

examples/parser.js

+6-6
Original file line numberDiff line numberDiff line change
@@ -8,27 +8,27 @@ const qs = require('qs')
88
// curl -X POST -d '{"hello":"world"}' -H'Content-type: application/json' http://localhost:3000/
99

1010
// curl -X POST -d '{"hello":"world"}' -H'Content-type: application/jsoff' http://localhost:3000/
11-
fastify.addContentTypeParser('application/jsoff', function (req, done) {
12-
jsonParser(req, function (err, body) {
11+
fastify.addContentTypeParser('application/jsoff', function (request, payload, done) {
12+
jsonParser(payload, function (err, body) {
1313
done(err, body)
1414
})
1515
})
1616

1717
// curl -X POST -d 'hello=world' -H'Content-type: application/x-www-form-urlencoded' http://localhost:3000/
18-
fastify.addContentTypeParser('application/x-www-form-urlencoded', function (req, done) {
18+
fastify.addContentTypeParser('application/x-www-form-urlencoded', function (request, payload, done) {
1919
var body = ''
20-
req.on('data', function (data) {
20+
payload.on('data', function (data) {
2121
body += data
2222
})
23-
req.on('end', function () {
23+
payload.on('end', function () {
2424
try {
2525
const parsed = qs.parse(body)
2626
done(null, parsed)
2727
} catch (e) {
2828
done(e)
2929
}
3030
})
31-
req.on('error', done)
31+
payload.on('error', done)
3232
})
3333

3434
fastify

fastify.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -404,7 +404,7 @@ function fastify (options) {
404404
if (fn.constructor.name === 'AsyncFunction' && fn.length !== 0) {
405405
throw new Error('Async function has too many arguments. Async hooks should not use the \'done\' argument.')
406406
}
407-
} else {
407+
} else if (name !== 'preParsing') {
408408
if (fn.constructor.name === 'AsyncFunction' && fn.length === 3) {
409409
throw new Error('Async function has too many arguments. Async hooks should not use the \'done\' argument.')
410410
}

0 commit comments

Comments
 (0)