diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d6049a887..dafd4fb12 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,7 +91,7 @@ demonstrate that the feature is mergeable to its implementation. 1. To the greatest extent possible changes should have an option for forward compatibility (old readers can still read files). The [compatibility and - feature enablement](#compatibility-and-feature-enablement) section below + feature enablement](#compatibility-and-feature-enablement) section below provides more details on expectations for changes that break compatibility. 2. New encodings should be fully specified in this repository and not @@ -110,10 +110,19 @@ demonstrate that the feature is mergeable to its implementation. The Parquet PMC aims to do releases of the format package only as needed when new features are introduced. If multiple new features are being proposed simultaneously some features might be consolidated into the same release. -Guidance is provided below on when implementations should enable features added -to the specification. Due to confusion in the past over Parquet versioning it -is not expected that there will be a 3.x release of the specification in the -foreseeable future. +Guidance is provided below on when implementations should enable features introduced +by new specification versions. If there is contributor bandwidth forwards-compatible +features may be backported to previous major versions of the specification and released +as a new minor version. + +Every forward incompatible change added to the specification requires +a major version bump (e.g. new encodings, new compression algorithms, new page layouts, +structural changes to the footer, etc). Forward compatible changes (e.g. new +logical types and sort orders) are as released as minor version changes. + +Versions 2.x.x did not follow this versioning, both forward incompatible changes and +forward compatible changes where released with minor version bumps. + ### Compatibility and Feature Enablement @@ -131,45 +140,51 @@ For the purposes of this discussion we classify features into the following buck 3. Forward incompatible. A file written under a newer version of the format with the feature enabled cannot be read under an older version of the format (e.g. - adding and using a new compression algorithm). It is expected any feature in + adding and using a new compression algorithm). Any feature in this category will provide a signal to older readers, so they can - unambiguously determine that they cannot properly read the file (e.g. via - adding a new value to an existing enum). + unambiguously determine that they cannot properly read the file. There are + two mechanisms for providing this functionality: + 1. Adding values to existing enums/unions (e.g. Sort Order, + encodings, compression) is covered by this mechanism. + 2. Reserving a new bit in the [PARX](ParxMagicNumber.md) footer feature bitmap covers other + structural changes that an older reader would not be able to accept. New features are intended to be widely beneficial to users of Parquet, and therefore it is hoped third-party implementations will adopt them quickly after -they are introduced. It is assumed that writing new parts of the format, and -especially forward incompatible features, will be configured with a feature flag -defaulted to "off", and at some future point the feature is turned on by default -(reading of the new feature will typically be enabled without configuration or -defaulted to on). Some amount of lead time is desirable to ensure a critical -mass of Parquet implementations support a feature to avoid compatibility issues +they are introduced. It is expected that implementations will provide a configuration +mechanism for users to enable features. It is recommended that implementations provide +at least a way to enable all relevant features given a specification version +(e.g. major and minor version). In addition, implementations might choose to +enable features at a finer-grained level, with feature flags initially defaulted to "off". + +Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a given specification version across the ecosystem. Therefore, the Parquet PMC gives the following -recommendations for managing features: +recommendations for managing the default specification version used for writing: 1. Backward compatibility is the concern of implementations but given the ubiquity of Parquet and the length of time it has been used, libraries should support reading older versions of the format to the greatest extent possible. -2. Forward compatible features/changes may be enabled and used by default in +2. Minor format versions may be enabled and used by default in implementations once the parquet-format containing those changes has been - formally released. For features that may pose a significant performance - regression to older format readers, libaries should consider delaying default - enablement until 1 year after the release of the parquet-java implementation - that contains the feature implementation. - -3. Forward incompatible features/changes should not be turned on by default - until 2 years after the parquet-java implementation containing the feature is - released. It is recommended that changing the default value for a forward - incompatible feature flag should be clearly advertised to consumers (e.g. via - a major version release if using Semantic Versioning, or highlighed in + formally released. For releases that may pose a significant performance + regression to older format readers, libraries should consider delaying default + enablement until 1 year after the parquet-java implementation for that format + version is released. + +3. Major version upgrades should not be enabled by default + until 2 years after the parquet-java implementation for the specification has been + released. It is recommended that changing the default format version for a major version bump + be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighted in release notes). For forward compatible changes which have a high chance of performance regression for older readers and forward incompatible changes, implementations should clearly document the compatibility issues. Additionally, while it is up to maintainers of individual open-source implementations to make the best decision to serve -their ecosystem, they are encouraged to start enabling features by default along +their ecosystem, they are encouraged to start enabling specific format versions by default along the same timelines as `parquet-java`. Parquet-java will wait to enable features by default until the most conservative timelines outlined above have been exceeded. This timeline is an attempt to balance ensuring @@ -179,14 +194,13 @@ encourage earlier adoption of new features when an organization using Parquet can guarantee that all readers of the parquet files they produce can read a new feature. -After turning a feature on by default implementations -are encouraged to keep a configuration to turn off the feature. +After changing defaults, implementations are encouraged to keep a configuration +mechanism to specify a prior format version or turn off specific features. A recommendation for full deprecation will be made in a future iteration of this document. -For features released prior to October 2024, target dates for each of these -categories will be updated as part of the `parquet-java 2.0` release process -based on a collected feature compatibility matrix. +As of June 2026, the current recommended default specification release +version to use is 2.10.0. For each release of `parquet-java` or `parquet-format` that influences this guidance it is expected exact dates will be added to parquet-format to provide @@ -198,5 +212,4 @@ implementation date/release version information when updating the feature matrix. End users of software are generally encouraged to consult the feature matrix -and vendor documentation before enabling features that are not yet widely -adopted. +and vendor documentation before enabling a specific format version. diff --git a/ParxMagicNumber.md b/ParxMagicNumber.md new file mode 100644 index 000000000..243092929 --- /dev/null +++ b/ParxMagicNumber.md @@ -0,0 +1,89 @@ +# PARX Parquet Format Specification + +This specification details a new magic number and associated fixed length footer metadata changes +that accompany the footer. + +## Motivation + +Most parts of the parquet specification lend themselves naturally to compatibility checks +when a new feature is added (e.g. encodings and compression values have an enum value added) +and fail appropriately. +However, some semantic changes or footer changes are impossible to communicate appropriately +within existing structures (e.g. changing the serialization of the footer). The motivation +for the new magic number and layout is to accomodate the latter set of changes by introducing +a new extensible mechanism for readers to detect these changes and fail accordingly. + +## Design Motivations + +* Provide a mechanism to only introduce a single new magic number for parquet that can + last at least a decade. +* Provide integrity checks for the footer. +* Provide the ability for readers to have a granular understanding of structural and semantic + backward incompatible features that are required to read a particular file. + +## File Layout + +A PARX file has the same overall structure as a standard Parquet file, with two differences: +the leading and trailing magic bytes are `PARX` instead of `PAR1/PARE`, and the trailing footer is +16 bytes instead of 8. + +The file layout is as follows: + +``` ++-----------+-------------------+--------------------+-------------+ +| 'PARX' | File Data | Footer Metadata | Footer tail | +| (4 bytes) | (variable length) | (variable length) | (16 bytes) | ++-----------+----------+--------+--------------------+-------------+ +``` + +All multi-byte integer fields are **little-endian**. + + +### PARX Footer Tail — 16 bytes + +``` ++------------------+-----------+----------+--------+ +| metadata_len | flags | crc32 | 'PARX' | ++------------------+-----------+----------+--------+ + offset 0 offset 4 offset 8 offset 12 +``` + +| Field | Type | Offset | Description | +|----------------|---------|--------|-------------------------------------------------------------------| +| `metadata_len` | i32 LE | 0 | Byte length of the Thrift-encoded `FileMetaData` block | +| `flags` | u32 LE | 4 | Feature flags (see [Feature Flags](#feature-flags)) | +| `crc32` | u32 LE | 8 | CRC32 checksum (see [Integrity Check](#integrity-check)) | +| `magic` | [u8; 4] | 12 | Always the bytes `P A R X` (0x50 0x41 0x52 0x58) | + +## Feature Flags + +The `flags` field is a 32-bit bitfield. A reader **must** reject any file whose `flags` field +contains bits that are not recognized or not supported, because unknown flags may imply structural changes +to the metadata or semantic changes to the file layout that the reader cannot properly interpret. + +The PARX format is independent of the `version` field in `FileMetaData`; a file may use the PARX +magic number regardless of which specification version its metadata declares. + +| Bit Index| Name | Description | +|----------|-------------------------|-------------------------------------------------------------------------------------------------------------| +| 0 | `ENCRYPTED_FOOTER` | The `FileMetaData` block is encrypted (equivalent to the `PARE` format). | +| 1 | `PATH_IN_SCHEMA_OMITTED` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| + + +The zero index is least signficant bit in the field. +All other bits are reserved and must be zero. + + +## Integrity Check + +The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) +computed over the following byte sequence, in order: + +1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 16-byte footer tail) +2. The first 8 bytes of the footer tail (metadata_len and flags bitmap) + +When `ENCRYPTED_FOOTER` (bit 0) is set, the CRC is computed over the footer bytes **as they appear in the +file** (i.e. the encrypted bytes). The CRC itself is always stored unencrypted in the footer tail. +A reader should verify the checksum **before** decrypting the footer, and **after** validating the feature +flags (in the rare case that a feature flag indicates a change in the fixed size tail of the file). + diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index e99c46195..c05ad5110 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -880,8 +880,13 @@ struct ColumnMetaData { * whether we can decode those pages. **/ 2: required list encodings - /** Path in schema **/ - 3: required list path_in_schema + /** Path in schema + * + * Made optional in parquet-format 3.0. If not written + * PARX magic number must be used (PATH_IN_SCHEMA_OMITTED, bit 1 in the + * feature flag bitmap, must be set). + **/ + 3: optional list path_in_schema /** Compression codec **/ 4: required CompressionCodec codec