What happened?
Given a partitioned dataset with small files problem (millions or tens of millions of files), scanning for files when trying to execute vacuum, takes lots of time because of storage pagination mechanism, which is sequential.
Expected behavior
Use more parallelism based on defined partitioning columns for better performance.
Operating System
None
Binding
None
Bindings Version
No response
Steps to reproduce
Given a partitioned dataset with millions of files (ie: 50 million files), the file scanning takes lots of time because of storage pagination which is sequential. In the case of these partitioned datasets, for the ones partitioned by more than one column we have some options. On AWS the max number of URIs returned per page is 1000 thus leading to tens of thousands of pages listed one after another.
In the case of a partitioned dataset, where the number of partitioning columns is n > 1, we can use prefixes and delimiters to achieve a greater parallelism when scanning the files in the dataset.
Given a 50_000_000 files dataset partitioned like /a=[1..5_000]/b=[1..10_000] we could parallelise by first partitioning column. If we choose a pool of 256 threads, this will lead to about 200x better performance when retrieving pages in parallel.
Rea Life Example
I have multiple production dataset with tens of millions of files spread in two Hive partition columns. I'll give the one of my findings on such a dataset: first partition has a cardinality of approx 5_000 and the second partition columns has a cardinality of approx 10_000. Current approach does listing sequentially the 5_000 * 10_000 object, page by page with 1_000 objects per page and is very slow.
Relevant logs
What happened?
Given a partitioned dataset with small files problem (millions or tens of millions of files), scanning for files when trying to execute vacuum, takes lots of time because of storage pagination mechanism, which is sequential.
Expected behavior
Use more parallelism based on defined partitioning columns for better performance.
Operating System
None
Binding
None
Bindings Version
No response
Steps to reproduce
Given a partitioned dataset with millions of files (ie: 50 million files), the file scanning takes lots of time because of storage pagination which is sequential. In the case of these partitioned datasets, for the ones partitioned by more than one column we have some options. On AWS the max number of URIs returned per page is 1000 thus leading to tens of thousands of pages listed one after another.
In the case of a partitioned dataset, where the number of partitioning columns is
n > 1, we can use prefixes and delimiters to achieve a greater parallelism when scanning the files in the dataset.Given a
50_000_000files dataset partitioned like/a=[1..5_000]/b=[1..10_000]we could parallelise by first partitioning column. If we choose a pool of256threads, this will lead to about 200x better performance when retrieving pages in parallel.Rea Life Example
I have multiple production dataset with tens of millions of files spread in two Hive partition columns. I'll give the one of my findings on such a dataset: first partition has a cardinality of approx
5_000and the second partition columns has a cardinality of approx10_000. Current approach does listing sequentially the5_000 * 10_000object, page by page with1_000objects per page and is very slow.Relevant logs