ENH: read_csv raises ParserError or hangs on certain malformed inputs

[rmhowe425]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

When reading certain malformed CSV bytes, pandas.read_csv either raises a ParserError or takes a long time to process. While this does not crash Python or cause a security issue, it may indicate a performance edge case or area where the parser could handle extreme inputs more gracefully.

Reproduction steps
from io import BytesIO
from pandas import read_csv

data = b"\x09\x00\x31\x2d\x2e\x23\x51\x00\x61\x2c\x00\x22\x0d\x31\x0d\x0d\x20\x0d\x3a\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0a\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x2c\x21\x0d\x0d\x0d\x0d\x0d\x0d\x69\x6e\x66\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x0d\x2c\x0d\x0d\x20\x20\x2c\x22\x22\x22\x2c\x0a\x2c\x2c"

read_csv(BytesIO(data), encoding='latin1')

Observed behavior
ParserError: Buffer overflow caught - possible malformed input file is raised.
In fuzz testing, the parser may take a long time on certain inputs (timeout).

Expected behavior / suggestion
It’s acceptable to raise ParserError on malformed inputs.

Consider whether extremely long sequences of control characters could be handled more efficiently or rejected faster to improve robustness.

Impact
No crash, no memory corruption.
Not security-relevant (not CVE-worthy).
Mostly affects fuzzing / extreme edge-case inputs.

Feature Description

1. Define thresholds for pathological sequences:

   • Max consecutive control characters (e.g., \0, \r, \n, \t) in a row.
   • Max line length (already exists in many parsers, but can be tuned).
   • Max number of embedded quotes without separator.

2. Scan input before tokenization (optional fast path):

   • Walk the input buffer, count consecutive control characters or other unusual patterns.
   • If thresholds exceeded → raise ParserError immediately.

3. Modify _tokenize_helper / tokenizer logic:

   • Track how long tokenization has taken per unit (for fuzzing / long input detection).
   • If time exceeds a configurable limit → raise a timeout ParserError.
   • Optionally: track recursion depth for nested quotes/escapes.

Alternative Solutions

N/A

Additional Context

No response

[Liam3851]

I ran the demonstration code on pandas 2.3.3 and it returns a ParserError in 55 microseconds that correctly notes that the file has malformed input. What is the performance edge case you are concerned with?

[rmhowe425]

For the specific byte sequence in my above example, current pandas (2.3.3) correctly raises ParserError very quickly, and there’s no issue with that behavior.

The performance concern comes from fuzzing rather than this single minimized repro. When I was running libFuzzer against the Pandas C tokenizer I saw multiple similar inputs (long runs of control characters, embedded NULs, irregular quoting) where the parser does not fail immediately and instead spends a disproportionate amount of time in tokenization before a ParserError is finally raised.

The snippet in the issue is intended as a concrete, deterministic example of malformed input hitting the same code paths, not a claim that this specific input is slow in current releases.

In other words, the “edge case” is about worst-case behavior under adversarial or fuzz-generated inputs, not typical malformed CSVs and not this particular repro on its own.

[Liam3851]

If there are other cases you have found in your fuzzing that reproduce pathological behavior, please list those instead of the current reproduction snippet above that generates expected behavior.

[rishii-19-works]

Thanks for the detailed report — I understand that for certain malformed CSV bytes, read_csv currently either raises a ParserError or experiences very long parse times due to inefficient handling of pathological sequences.
I’m drafting a PR to add early checks and thresholds to catch these extreme cases more efficiently and improve robustness. I’ll post the PR here shortly for review.

[Liam3851]

@rishii-19-works I am not entirely convinced there is an issue. I would suggest waiting for a reproducer that produces behavior that needs changing before proposing a PR.