iceberg_scan() is slower than read_parquet() if multithreaded

[alippai]

What happens?

Using the same single file, the iceberg_scan() is slower than read_parquet().

The file is about 4GB with ~35 row groups.
A simple select * from iceberg_scan():

Run Time (s): real 8.250 user 4.033364 sys 4.176462

The same parquet file:

Run Time (s): real 0.918 user 6.176052 sys 12.617458

Only 40 rows are shown as usual, the CLI output is identical. Maybe it's as simple as not skipping row groups, not pushing down the limit+offest when displaying the sample.

To Reproduce

select * from iceberg_scan('an iceberg table with the same single parquet')

OS:

linux

DuckDB Version:

1.3.2

DuckDB Client:

CLI

Hardware:

No response

Full Name:

Adam Lippai

Affiliation:

N/A

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[Tishj]

Sanity check: you are closing and reopening the CLI between these tests?
If you're not, the faster result is likely explained by caching

[alippai]

Good point. I checked repeatedly and it’s consistent. The explain plans are practically the same too aside from the scan name. Also checked select sum(double_value) from read_ to ensure it’s not about some schema casting, that shows the 8-10x difference too. Doing sum(col1+col2) doubles the query time for both iceberg and parquet, so it shouldn’t be the static overhead of reading json+avro.

Do you have a public parquet+iceberg pair where it’s easy compare?

[alippai]

Found something.

• Limiting the threads to 1 brings the parquet speed to the iceberg level.
• Increasing the threads to 4 doesn’t change the iceberg numbers, but it improves the parquet numbers.

Feels like iceberg is always single threaded.

[alippai]

Vibe coded benchmark (sorry):

import shutil
import time
from pathlib import Path
import statistics

import duckdb
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import DoubleType, NestedField

# -- Configuration
PARQUET_PATH = Path("/tmp/benchmark/value_data.parquet")
TOTAL_ROWS = 125_000_000
WAREHOUSE_PATH = Path("/tmp/warehouse")
ICEBERG_TABLE_PATH = WAREHOUSE_PATH / "iceberg_value_table"
shutil.rmtree(WAREHOUSE_PATH, ignore_errors=True)

# -- Generate Parquet data
PARQUET_PATH.parent.mkdir(parents=True, exist_ok=True)
duckdb.sql(
    f"""
    COPY (
        SELECT RANDOM()::DOUBLE AS value
        FROM range({TOTAL_ROWS})
    ) TO '{PARQUET_PATH}' (FORMAT 'parquet', compression 'ZSTD');
    """
)
print(f"Generated {TOTAL_ROWS:,} rows of test data...")

# -- Create Iceberg table
WAREHOUSE_PATH.mkdir(parents=True, exist_ok=True)
catalog = load_catalog(
    "default",
    **{
        "type": "sql",
        "uri": f"sqlite:///{WAREHOUSE_PATH}/pyiceberg_catalog.db",
        "warehouse": f"file://{WAREHOUSE_PATH}",
    },
)
identifier = "benchmark.value_tbl"
schema = Schema(NestedField(1, "value", DoubleType(), required=False))
catalog.create_namespace("benchmark")
table = catalog.create_table(identifier, schema=schema, location=str(ICEBERG_TABLE_PATH))
abs_parquet = PARQUET_PATH.resolve().as_posix()
table.add_files([abs_parquet])


# -- Run benchmark
duckdb.sql("load iceberg;SET unsafe_enable_version_guessing = true;")

parquet_q = f"SELECT SUM(value) FROM read_parquet('{PARQUET_PATH}')"
iceberg_q = f"SELECT SUM(value) FROM iceberg_scan('{ICEBERG_TABLE_PATH}')"

def time_query(query: str, threads: int) -> float:
    duckdb.sql(f"SET threads = {threads};")
    start = time.perf_counter()
    duckdb.sql(query).fetchall()
    return time.perf_counter() - start

runs = 4
stats: dict[tuple[str, int], list[float]] = {}

for thr in (1, 2, 4, 6):
    for _ in range(runs):
        stats.setdefault(("parquet", thr), []).append(time_query(parquet_q, thr))
        stats.setdefault(("iceberg", thr), []).append(time_query(iceberg_q, thr))

# -- Build summary rows
rows: list[tuple[str, int, float, float]] = []
for (iface, thr), times in stats.items():
    rows.append(
        (
            iface,
            thr,
            statistics.mean(times),
            statistics.stdev(times) if len(times) > 1 else 0.0,
        )
    )

name_width = max(len(name) for name, *_ in rows)
print(f"\nBenchmark results over {runs} runs (lower mean is better)")
print(f"{'interface':{name_width}} | threads |   mean s |  stddev")
print("-" * (name_width + 35))
for name, thr, mean_s, sd in sorted(rows):
    print(f"{name:{name_width}} | {thr:>7} | {mean_s:>8.2f} | {sd:>7.2f}")

For me it prints:

$ python parquet_vs_iceberg.py
Generated 125,000,000 rows of test data...

Benchmark results over 4 runs (lower mean is better)
interface | threads |   mean s |  stddev
------------------------------------------
iceberg |       1 |     1.56 |    0.15
iceberg |       2 |     1.59 |    0.30
iceberg |       4 |     1.84 |    0.32
iceberg |       6 |     1.58 |    0.31
parquet |       1 |     1.55 |    0.12
parquet |       2 |     0.77 |    0.09
parquet |       4 |     0.42 |    0.02
parquet |       6 |     0.36 |    0.10

[szarnyasg]

@alippai Thanks, I could reproduce the results on macOS too.