Fix UnicodeDecodeError when filtering a binary POST body (#660)#1017
Open
uttam12331 wants to merge 1 commit into
Open
Fix UnicodeDecodeError when filtering a binary POST body (#660)#1017uttam12331 wants to merge 1 commit into
uttam12331 wants to merge 1 commit into
Conversation
replace_post_data_parameters() treats any non-dict, non-JSON POST body as
form-urlencoded data and decodes it as UTF-8 to apply
filter_post_data_parameters. When the body is raw binary bytes (e.g. an image
upload passed directly as bytes rather than wrapped in a BytesIO), any 0x3D
('=') byte makes the preceding binary run be treated as a form key, and
k.decode('utf-8') raises UnicodeDecodeError.
Wrap the form-parameter parsing so a body that is not valid UTF-8 form data is
left untouched: a binary body has no form parameters to replace, so there is
nothing to filter. Text/urlencoded bodies keep working exactly as before.
Fixes kevin1024#660
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #660
Problem
When
filter_post_data_parametersis configured and a request has a raw binary POST body passed asbytes(e.g. an image/file upload, rather than aBytesIO),replace_post_data_parameterscrashes:The
elsebranch treats any non-dict / non-JSON body as form-urlencoded, splits onb"&"/b"=", and doesk.decode("utf-8"). Any0x3D(=) byte inside binary data makes the preceding binary run be parsed as a form "key", so the decode blows up. The existingisinstance(request.body, BytesIO)guard doesn't help because a rawbytesbody isn't aBytesIO.Fix
Wrap the form-parameter parsing in a
try/except UnicodeDecodeError. A body that isn't valid UTF-8 form data has no form parameters to replace, so it's left untouched instead of crashing. Text / form-urlencoded bodies behave exactly as before (the new body is only assigned on success, so there's no partial mutation).Verification
b"id=secret&keep=1"→b"keep=1").test_replace_post_data_parameters_binary_body, which fails onmasterand passes with this change; the fulltests/unit/test_filters.pysuite passes (25 passed).ruffclean. Changelog updated.