Skip to content

Fix UnicodeDecodeError when filtering a binary POST body (#660)#1017

Open
uttam12331 wants to merge 1 commit into
kevin1024:masterfrom
uttam12331:fix/binary-post-body-660
Open

Fix UnicodeDecodeError when filtering a binary POST body (#660)#1017
uttam12331 wants to merge 1 commit into
kevin1024:masterfrom
uttam12331:fix/binary-post-body-660

Conversation

@uttam12331

Copy link
Copy Markdown

Fixes #660

Problem

When filter_post_data_parameters is configured and a request has a raw binary POST body passed as bytes (e.g. an image/file upload, rather than a BytesIO), replace_post_data_parameters crashes:

from vcr.request import Request
from vcr.filters import replace_post_data_parameters

body = b"\xff\xd8\xff\xe1\x00\x08Exif=\x00\x00MM*"   # raw JPEG-ish bytes
req = Request("POST", "http://x/upload", body, {"Content-Type": "image/jpeg"})
replace_post_data_parameters(req, [("session", "REDACTED")])
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0

The else branch treats any non-dict / non-JSON body as form-urlencoded, splits on b"&" / b"=", and does k.decode("utf-8"). Any 0x3D (=) byte inside binary data makes the preceding binary run be parsed as a form "key", so the decode blows up. The existing isinstance(request.body, BytesIO) guard doesn't help because a raw bytes body isn't a BytesIO.

Fix

Wrap the form-parameter parsing in a try/except UnicodeDecodeError. A body that isn't valid UTF-8 form data has no form parameters to replace, so it's left untouched instead of crashing. Text / form-urlencoded bodies behave exactly as before (the new body is only assigned on success, so there's no partial mutation).

Verification

  • The reproduction above now leaves the body unchanged instead of raising.
  • Normal form filtering is unaffected (b"id=secret&keep=1"b"keep=1").
  • Added test_replace_post_data_parameters_binary_body, which fails on master and passes with this change; the full tests/unit/test_filters.py suite passes (25 passed). ruff clean. Changelog updated.

replace_post_data_parameters() treats any non-dict, non-JSON POST body as
form-urlencoded data and decodes it as UTF-8 to apply
filter_post_data_parameters. When the body is raw binary bytes (e.g. an image
upload passed directly as bytes rather than wrapped in a BytesIO), any 0x3D
('=') byte makes the preceding binary run be treated as a form key, and
k.decode('utf-8') raises UnicodeDecodeError.

Wrap the form-parameter parsing so a body that is not valid UTF-8 form data is
left untouched: a binary body has no form parameters to replace, so there is
nothing to filter. Text/urlencoded bodies keep working exactly as before.

Fixes kevin1024#660
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

POSTing binary data as bytes or bytearray instead of BytesIO breaks assumptions baked into replace_post_data_parameters

1 participant