[Python] Hive partition columns being forced to dict type

[JasonTam]

Describe the bug, including details regarding any error messages, version, and platform.

version: 21.0.0
platform: arm64

When reading data with:

• hive-style partioning
• integer dataframe columns

The partition columns get inferred as int type.

from pyarrow.parquet import read_table

path = "gs://project/data/run_date=2025-09-17/job_id=abc123/0.pq"
dataset = read_table(path)
dataset.read()

pyarrow.Table
0: uint32
1: uint32
2: uint32
3: uint32
...
99: uint32
run_date: dictionary<values=string, indices=int32, ordered=0>
job_id: dictionary<values=string, indices=int32, ordered=0>

This causes issues when trying to cast "2025-09-17" to int32 for example

Component(s)

Python

[AlenkaF]

When using read_table from the parquet module, you access the dataset functionality under the hood if it is available. In this case hive partitioning will be used by default with partitioned fields inferred as dictionary types. See infer_dictionary=True in:

arrow/python/pyarrow/parquet/core.py

Lines 1419 to 1422 in 479662e

	# check partitioning to enable dictionary encoding
	if partitioning == "hive":
	partitioning = ds.HivePartitioning.discover(
	infer_dictionary=True)

See also: https://arrow.apache.org/docs/python/generated/pyarrow.dataset.HivePartitioning.html#pyarrow.dataset.HivePartitioning.discover

If you want partition fields as regular string/int columns, you can use the dataset module directly. Then, Arrow will infer partition column types instead of dictionary types:

>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> import os

>>> t = pa.table({"col1": [1, 2, 3], "col2": [4, 5, 6]})
>>> partition_dir = os.path.join("data", "run_date=2025-09-17", "job_id=abc123")
>>> os.makedirs(partition_dir, exist_ok=True)
>>> pq.write_table(t, os.path.join(partition_dir, "0.parquet"))

>>> from pyarrow.dataset import dataset
>>> ds = dataset("data", format="parquet", partitioning="hive")
>>> table = ds.to_table()
>>> print(table.schema)
col1: int64
col2: int64
run_date: string
job_id: string

>>> table
pyarrow.Table
col1: int64
col2: int64
run_date: string
job_id: string
----
col1: [[1,2,3]]
col2: [[4,5,6]]
run_date: [["2025-09-17","2025-09-17","2025-09-17"]]
job_id: [["abc123","abc123","abc123"]]

vs using the parquet module:

>>> pq.read_table("data")
pyarrow.Table
col1: int64
col2: int64
run_date: dictionary<values=string, indices=int32, ordered=0>
job_id: dictionary<values=string, indices=int32, ordered=0>
----
col1: [[1,2,3]]
col2: [[4,5,6]]
run_date: [  -- dictionary:
["2025-09-17"]  -- indices:
[0,0,0]]
job_id: [  -- dictionary:
["abc123"]  -- indices:
[0,0,0]]

[JasonTam]

@AlenkaF Thanks for the details.

I'm worried that I might have misdiagnosed my issue. Perhaps it's more of a pandas conversion problem. I tried to make a minimal example here:

import pandas as pd
df = pd.DataFrame({
    0: [1, 2, 3],
    1: [4, 5, 6]
})
df.to_parquet("./test-pd-data/run_date=2025-09-17/0.parquet")

from pyarrow.dataset import dataset
ds = dataset("test-pd-data", format="parquet", partitioning="hive")
table = ds.to_table()
print(table.schema)

table.to_pandas()

table.schema:

0: int64
1: int64
run_date: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 437

Error:

ValueError                                Traceback (most recent call last)
table.to_pandas()

File .venv/lib/python3.12/site-packages/pyarrow/array.pxi:1020, in pyarrow.lib._PandasConvertible.to_pandas()

File .venv/lib/python3.12/site-packages/pyarrow/table.pxi:5177, in pyarrow.lib.Table._to_pandas()

File .venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py:803, in table_to_dataframe(options, table, categories, ignore_metadata, types_mapper)
    798     ext_columns_dtypes = _get_extension_dtypes(
    799         table, [], types_mapper, options, categories
    800     )
    802 _check_data_column_metadata_consistency(all_columns)
--> .venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py:803) columns = _deserialize_column_index(table, all_columns, column_indexes)
    805 column_names = table.column_names
    806 result = pa.lib.table_to_blocks(options, table, categories,
    807                                 list(ext_columns_dtypes.keys()))

File .venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py:967, in _deserialize_column_index(block_table, all_columns, column_indexes)
    965 # if we're reconstructing the index
    966 if len(column_indexes) > 0:
--> .venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py:967)     columns = _reconstruct_columns_from_metadata(columns, column_indexes)
    969 return columns
...
    132     # Explicit copy, or required since NumPy can't view from / to object.
--> .venv/lib/python3.12/site-packages/pandas/core/dtypes/astype.py:133)     return arr.astype(dtype, copy=True)
    135 return arr.astype(dtype, copy=copy)

ValueError: invalid literal for int() with base 10: 'run_date'

The thing is, if I create & write the parquet file with pyarrow, I don't run into this issue.

Pandas version: 2.2.3

[AlenkaF]

Here you are mixing a single-file metadata write (which also stores pandas metadata) with a hive-partitioned read, and that is causing the issue. To make the example work, both the write and the read need to use partitions:

>>> df = pd.DataFrame({
...     "0": [1, 2, 3],
...     "1": [4, 5, 6],
...     "run_date": ["2025-09-17","2025-09-17","2025-09-17"]
... })
... df.to_parquet("./test-pd-data", partition_cols=["run_date"])
... 
>>> 
>>> from pyarrow.dataset import dataset
... ds = dataset("test-pd-data", format="parquet", partitioning="hive")
... table = ds.to_table()
... print(table.schema)
... 
0: int64
1: int64
run_date: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 620
>>> table.to_pandas()
... 
   0  1    run_date
0  1  4  2025-09-17
1  2  5  2025-09-17
2  3  6  2025-09-17

Just to confirm, are you only seeing issues on the read side? I’m a little confused about the exact problem, so a clearer description and the error output would really help.

[JasonTam]

Yes, only seeing issues on the read side. I have the error/stacktrace in the previous message from the minimal example.

Maybe I'm trying to do something that's not the intended way. The goal is to write some pq partitions with pandas where the partition information only exists in the filepath. (Just like my previous example)

import pandas as pd
df = pd.DataFrame({
    0: [1, 2, 3],
    1: [4, 5, 6]
})
df.to_parquet("./test-pd-data/run_date=2025-09-17/0.parquet")
df.to_parquet("./test-pd-data/run_date=2025-09-18/0.parquet")
df.to_parquet("./test-pd-data/run_date=2025-09-18/1.parquet")
# etc

A separate system will be reading this dataset test-pd-data in and inferring the hive partitions.

pd.read_parquet("./test-pd-data/", engine="pyarrow", partitioning="hive")

But are you saying that both writer and reader need to be in sync with metadata (especially for the partition fields)? In my use-case, the writer doesnt have any information about the partition field aside from being in the filepath.

And I only run into this error (ValueError: invalid literal for int() with base 10: 'run_date') when the whole dataframe is of int type.