Avoid downloading all parts of a parquet dataset if to needed (-test workloads) #49
Labels
enhancement
New feature or request
Milestone
Comments
daverigby
modified the milestones:
Phase 1 - YCSB equivalence ,
Phase 2: More workloads, more databases
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
*-testworkload variants consist of ~1% of the complete dataset; however if the dataset is multi-file (e.g. nq768, ...) then we still download all the files even though we may only need the first couple.We should limit this to just the files we need - probably by adding an early exit to the download loop if we have downloaded enough records.
The text was updated successfully, but these errors were encountered: