[Draft RFC] Update guidance on versioning to use parquet-format version #588
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # PARX Parquet Format Specification | ||
|
There was a problem hiding this comment. I think we should separate discussion of
Contributor
Author
There was a problem hiding this comment. The conversation that spurred this thread and a bunch of the discussion involved reader/writer coordination. In particular the original doc posed these questions:
The PARX format is one means of solving this coordination, which I think satisfies requirements from the community, so I'm not sure it makes sense to decouple it completely? There was a problem hiding this comment. I think if we're going to design a new magic number it should be done independently and then we use that representation to effect a change like this. I don't think they should be bundled together.
Contributor
Author
There was a problem hiding this comment. If we are talking specifically about the change to path_in_schema, I agree, we can decouple these once we get alignment on the overall direction. The reason for coupling them in this PR is to illustrate how it would be used in practice. |
||
|
|
||
| This specification details a new magic number and associated fixed length footer metadata changes | ||
| that accompany the footer. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Most parts of the parquet specification lend themselves naturally to comptability checks | ||
|
emkornfield marked this conversation as resolved.
Outdated
|
||
| when a new feature is added (e.g. encodings and compression values have an enum value added) | ||
| and fail appropriately. | ||
| However, some semantic changes or footer changes are impossible to communicate appropriately | ||
| within existing structures (e.g. changing the serialization of the footer). The motivation | ||
| for the new magic number and layout is to accomodate the latter set of changes by introducing | ||
| a new extensible mechanism for readers to detect these changes and fail accordingly. | ||
|
|
||
| ## Design Motivations | ||
|
|
||
| * Provide a mechanism to only introduce a single new magic number for parquet that can | ||
| last at least a decade. | ||
| * Provide integrity checks for the footer. | ||
| * Provide the ability for readers to have a granular understanding of structural and semantic | ||
| backward incompatible features that are required to read a particular file. | ||
|
|
||
| ## File Layout | ||
|
|
||
| A PARX file has the same overall structure as a standard Parquet file, with two differences: | ||
| the leading and trailing magic bytes are `PARX` instead of `PAR1/PARE`, and the trailing footer is | ||
| 16 bytes instead of 8. | ||
|
|
||
| The file layout is as follows: | ||
|
|
||
| ``` | ||
| +-----------+-------------------+--------------------+-------------+ | ||
| | 'PARX' | File Data | Footer Metadata | Footer tail | | ||
| | (4 bytes) | (variable length) | (variable length) | (16 bytes) | | ||
| +-----------+----------+--------+--------------------+-------------+ | ||
| ``` | ||
|
|
||
| All multi-byte integer fields are **little-endian**. | ||
|
|
||
|
|
||
| ### PARX Footer Tail — 16 bytes | ||
|
|
||
| ``` | ||
| +------------------+-----------+----------+--------+ | ||
| | metadata_len | flags | crc32 | 'PARX' | | ||
| +------------------+-----------+----------+--------+ | ||
| offset 0 offset 4 offset 8 offset 12 | ||
| ``` | ||
|
|
||
| | Field | Type | Offset | Description | | ||
| |----------------|---------|--------|-------------------------------------------------------------------| | ||
| | `metadata_len` | i32 LE | 0 | Byte length of the Thrift-encoded `FileMetaData` block | | ||
| | `flags` | u32 LE | 4 | Feature flags (see [Feature Flags](#feature-flags)) | | ||
|
Contributor
There was a problem hiding this comment. That's a permanent restriction to 32 flags; using a u64 would add minimal cost at this point and avoid complications if 33 flags were ever needed many years in the future
Contributor
There was a problem hiding this comment. I had the same thought, but I can also understand keeping this simple for now, as I think the intent is to use this bitfield sparingly. Most changes will be communicated with a major version bump, so it's only structural changes like a new footer format that will require a new bit position. I think any new footer we design should itself be extensible without requiring the use of additional bits here. As to clean-ups of the existing thrift structure, perhaps we should roll up the If we are concerned about running out of bits, then we could instead say the footer isn't fixed in size, and use continuation bits to make the bitfield arbitrary in size. Or keep it fixed, and say bit 32 signifies an additional bitfield located in a TBD location in the footer.
Contributor
Author
There was a problem hiding this comment. I don't feel too strongly. I think even the ones listed only gets us less then 10 bits. And if we end up going with a new footer hopefully that would take 1 bit and hopefully keep this pretty low. I do think it is a good idea to update the wording that this might not be fixed size (could grow in the future, but that would be detectable via inspecting the bitmap). Another option, given we expect this field to not evolve too quickly past an initial set of changes is to have bit 31, effectively be a version bit, which indicates all structural changes as of a certain date are always true.
Contributor
Author
There was a problem hiding this comment. One nice thing about 32 bits is it makes the current implementation a little easier since a valid parquet file today must always be at least 16 bytes (2* |
||
| | `crc32` | u32 LE | 8 | CRC32 checksum (see [Integrity Check](#integrity-check)) | | ||
| | `magic` | [u8; 4] | 12 | Always the bytes `P A R X` (0x50 0x41 0x52 0x58) | | ||
|
|
||
| ## Feature Flags | ||
|
|
||
| The `flags` field is a 32-bit bitfield. A reader **must** reject any file whose `flags` field | ||
| contains bits that are not recognized or not supported, because unknown flags may imply structural changes | ||
| to the metadata or semantic changes to the file layout that the reader cannot properly interpret. | ||
|
|
||
| The PARX format is independent of the `version` field in `FileMetaData`; a file may use the PARX | ||
| magic number regardless of which specification version its metadata declares. | ||
|
|
||
| | Bit Index| Name | Description | | ||
| |----------|-------------------------|-------------------------------------------------------------------------------------------------------------| | ||
| | 0 | `ENCRYPTED_FOOTER` | The `FileMetaData` block is encrypted (equivalent to the `PARE` format). | | ||
|
Contributor
There was a problem hiding this comment. Should we add these to parquet.thrift as constants?
Contributor
Author
There was a problem hiding this comment. Seems reasonable, I'm not clear on the support for constants in thrift, I'll do some research if we can align with this approach. |
||
| | 1 | `PATH_IN_SCHEMA_OMITTED` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| | ||
|
|
||
|
|
||
| The zero index is least signficant bit in the field. | ||
| All other bits are reserved and must be zero. | ||
|
|
||
|
|
||
| ## Integrity Check | ||
|
|
||
| The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) | ||
|
Contributor
There was a problem hiding this comment. crc32s is where integrity checksums should be going, because x86 and ARM parts can do in a single instruction. java has a native method for this now.
Contributor
Author
There was a problem hiding this comment. I went back and forth on this. I don't feel too strongly but having the same CRC used across the file ultimately seemed better, then having a one off difference. |
||
| computed over the following byte sequence, in order: | ||
|
|
||
| 1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 16-byte footer tail) | ||
| 2. The first 8 bytes of the footer tail (metadata_len and flags bitmap) | ||
|
|
||
| When `ENCRYPTED_FOOTER` (bit 0) is set, the CRC is computed over the footer bytes **as they appear in the | ||
| file** (i.e. the encrypted bytes). The CRC itself is always stored unencrypted in the footer tail. | ||
| A reader should verify the checksum **before** decrypting the footer, and **after** validating the feature | ||
| flags (in the rare case that a feature flag indicates a change in the fixed size tail of the file). | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"read under an older version of the format" is a bit confusing here. Presumably it means "by a library/app only written to work with earlier versions"