Logs with a lone surrogate make Relay drop the whole log batch #8144

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.73.0 (master at 63707d7)

Steps to Reproduce

I found this while working on getsentry/relay#5837. On Linux and macOS, file names, environment variables and command line arguments that are not valid UTF-8 are decoded with surrogateescape, so they end up as strings with lone surrogates. Logging one of them is enough:

import logging
import sentry_sdk

sentry_sdk.init(dsn="<your DSN>", enable_logs=True)

# What os.fsdecode returns on Linux and macOS: 'report-\udcff.csv'
name = b"report-\xff.csv".decode("utf-8", "surrogateescape")

sentry_sdk.logger.info("first log")
sentry_sdk.logger.info("opened {name}", name=name)
logging.getLogger("app").warning("could not read %s", name)
sentry_sdk.logger.info("last log")

json_dumps (sentry_sdk/utils.py:125) calls json.dumps with the default ensure_ascii=True, which writes the lone surrogate as the escape \udcff. With a capturing transport, the log container item has item_count: 4 and contains \udcff four times: in the two bodies and in the sentry.message.parameter.name and sentry.message.parameter.0 attributes.

Expected Result

All four logs arrive. The two logs with the file name show a replacement character in its place.

Actual Result

Relay's JSON parser rejects a lone surrogate escape, so it rejects the whole container and every log flushed with it, including the two healthy ones. I have not sent this to sentry.io; I checked it by feeding the captured payload into Relay's container parser (ItemContainer::<OurLog>::parse on Relay master), which fails with lone leading surrogate in hex escape. With the change proposed below, the same script's payload parses with all 4 logs.

Relay's position is that this should be fixed in the SDKs (getsentry/relay#5833: "We require payloads to be valid UTF-8 after JSON decoding in Relay. SDKs sending non-compliant data should be fixed at the SDK level.", tracked on the Relay side in getsentry/relay#5837). The JavaScript SDK did this for logs in getsentry/sentry-javascript#20245 by replacing lone surrogates with U+FFFD at capture time.

Proposed fix

Do the same in LogBatcher._to_transport_format: replace lone surrogates with U+FFFD in the body, in attribute keys and in string attribute values (including string arrays), using s.encode("utf-16", "surrogatepass").decode("utf-16", "replace"), which keeps valid surrogate pairs. Only strings that fail s.encode("utf-8") are rewritten. MetricsBatcher and SpanBatcher call serialize_attribute the same way, so trace metric and span containers are likely affected too. I would leave them for a follow-up.

I'd be happy to open a PR for this if the approach sounds right.