BUG: pd.read_json fails with "Value is too big!" on large integers, while json.load + DataFrame works

[wongzhenhao]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

### Minimal reproducible example


import pandas as pd
import json
from io import StringIO

json_str = """
[
  {
    "composition": "Nb:100",
    "n_atoms": 128000,
    "space_group": 229,
    "time": 1000000000000000000000000
  }
]
"""

# This fails
pd.read_json(StringIO(json_str))

Issue Description

Problem description

pd.read_json fails with a ValueError: Value is too big! when parsing JSON files that contain very large integers (e.g. > 1e18).

However, loading the same file using Python's built-in json.load and then constructing a pd.DataFrame works correctly.

I'm happy to help test a fix or provide additional examples if needed.

Expected Behavior

pd.read_json should be able to parse large integers in JSON files in the same way as Python's built-in json.load, and construct a DataFrame without raising an overflow error.

Installed Versions

Details

Environment

• pandas version: 2.x.x
• Python version: 3.10

[rhshadrach]

Thanks for the report. Currently pandas uses a vendored copy of ujson, and an old version of it I believe. Testing the latest release, ujson is able to load large integers. However ujson is also in maintenance mode, and pandas is looking to switch to use third party tools directly rather than vendored copies. I think that would be the right approach for this issue.

See #62464 for more details.

[rhshadrach]

Related: #44684