Skip to content

parquet-rewrite: supports compression level and write batch size #8639

@mapleFU

Description

@mapleFU

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

parquet-rewrite works well currently, however:

  1. The batch-size is the default batch-size, assume we want to test page-size == 1000, but it would always uses 1024
  2. The compression level is always 1. zstd's default level is 3, but here, it uses 1.

Describe the solution you'd like

Allow set the compression level and batch size.

Describe alternatives you've considered

No

Additional context

No

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions