[Bug]: Regression in transaction log parsing performance

[Tom-Newton]

What happened?

Median transaction log parsing performance with python-v1.6.2 takes about 50% longer than on python-v1.1.4

Expected behavior

New versions would be the same performance or better. Reporting as a bug seemed more appropriate than a feature request but is not ideal.

Operating System

Linux

Binding

Python

Bindings Version

python-v1.6.2

Steps to reproduce

Ultimately its this python code that I care about

deltalake.DeltaTable(
    "abfss://<container>@<storage account>.dfs.core.windows.net/path/to/table/with/long/transaction_log",
    storage_options={"use_azure_cli": "true"},
)

I'm testing this on one of our real tables, that I can't really share. It has JSON stats collected on 425 and the most recent checkpoint is about 800MiB. The obvious ideas to improve are:

1. Use parquet stats instead of JSON, but when I tested this recently it was significantly slower
2. Reduce the number of columns we collect stats for. This is a fair point, but there are downsides so I can't just unilaterally do that.

But I've also been testing directly on the rust code at times to make profiling a bit easier and its definitely the rust code that affects the performance.

Attaching profiles captured.
delta_1.6.2_profile.json.gz --profile profiling
delta_1.1.4_profile.json.gz --profile release-with-debug

Relevant logs