Make format version pages data driven by alamb · Pull Request #189 · apache/parquet-site

alamb · 2026-06-24T16:06:23Z

Follow on to

Document Parquet Features by Version #186

Rendered Preview:

Rationale

Implements @emkornfield 's request in #186 (comment)

could we somehow integrate this as YAML data + rendering like we do for feature compatibility?

Among other things, rendering the page using data files forces it to remain consistent with the implementation status page.

Changes

Updates the Parquet format version and implementation status pages to generate their feature tables from structured data under data/implementations.

Planned follow on

Making this change already shows value -- some features only on the versions page but not the implementation status page

This PR simply ignores rows that have implementation_status: false on the implementation status page

As a follow on PR, I will propose adding the new features to the implementation status page, but this PR is already quite large

Examples:

logical-type-union
logical-nanosecond-time-timestamp
logical-variant-shredding
format-data-page-v1
format-geospatial-statistics
format-binary-protocol-extensions
format-ieee-754-total-order-and-nan-counts

alamb · 2026-06-24T16:14:54Z

 * [polars](https://github.com/pola-rs/polars) (Rust)

-<!-- Status source in data/implementations -->
+<!-- Data driven table, see /layouts/shortcodes -->


I added a readme and new link in layouts/shortcodes

alamb · 2026-06-24T16:16:24Z


 Forward incompatible features and the format version each became available in:

-| Feature | Released in | Source | Notes |


The point of the PR is to automatically derive this table from the .yml files

alamb · 2026-06-24T16:17:20Z

  - id: compression-uncompressed
    display_name: UNCOMPRESSED
-
+    compatibility: forward_incompatible


I added the various fields needed for the version page to the existing data files. I could probably make this less verbose by automatically generating the urls, but I think this is clearer and easier to understand

alamb · 2026-06-24T16:17:56Z

  - id: encoding-byte-stream-split-extended
    display_name: BYTE_STREAM_SPLIT (Additional Types)
    spec_url: https://github.com/apache/parquet-format/commit/e517ac4dbe08d518eb5c2e58576d4c711973db94
-    format_version:


some features already had a "format_version" which was redundant with the information from the versions page, so I consolidated them into the same fields

From

format_version: version: "2.11.0" date: "2024-03-21"

To

released_in: 2.11.0 release_date: '2024-03-21'

my one concern is 2.11.0, interpretted properly as a string without quote marks?

Good idea -- I quoted them all in commit 9f8d58a

alamb · 2026-06-24T16:20:03Z

@@ -0,0 +1,15 @@
+# Hugo shortcodes


I found the whole hugo rendering thing new, so I also left some small documentation

This reverts commit 1fe107f.

etseidl · 2026-06-24T20:53:16Z

Thanks @alamb, this is neat! One thing occurred to me while verifying external links...it would be nice for the features to link to their specific PR. The current "Source" column links to all changes between versions, so there's a lot of housekeeping changes to wade through.

Not for now, but as a follow-on.

emkornfield · 2026-06-24T22:06:11Z

    display_name: 8, 16, 32, 64 bit signed and unsigned INT
-
+    version_display_name: Signed and unsigned integer logical types (INT32, INT64)
+    compatibility: forward_compatible


I think this is incompatible?

negative physical values are interpetted incorrectly?

I think we've agreed that new logical types are forward compatible.

New logical types are considered forward compatible despite the loss of semantic meaning.

https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement

Yeah, this is broader discussion, I think we should clarify the wording here. we can leave it as is, but I worry about the difference between different semantics:

Losing information that a int64 represents a timestamp is fine in my mind. It still looks like an integer, and following arithmetic properties.

Losing information that an int64 is actually a uint64 feels like a data corruption case.

But I'd have to clarify my thinking here a bit more. We don't need to make the change here.

emkornfield · 2026-06-24T22:07:13Z

  - id: logical-decimal-int64
    display_name: DECIMAL (INT64)
-
+    compatibility: forward_compatible


technically incompatible. numbers cannot be understood properly without scale?

Same comment as #189 (comment) applies, not a blocker.

emkornfield · 2026-06-24T22:09:50Z

+        {{- $featureUrl = $feature.spec_url -}}
+      {{- end -}}
+      {{- $releasedIn := printf "%v" $feature.released_in -}}
+      {{- $legacyReleaseTags := slice "1.0.0" "2.0.0" "2.1.0" -}}


what does the slice do here?

Apparently in hugo makes an array -- this is part of some overly complicated logic to translate release names like 1.0.0 into tag names (whose format changed over time)

I simplified the whole thing by changing this to a lookup table -- which simplifies this logic and I think makes it clearer what is going on

emkornfield · 2026-06-24T22:10:16Z

      <tbody>
        {{- range $features -}}
+          {{- /* Include rows by default; skip rows explicitly marked implementation_status: false. */ -}}
+          {{- if ne .implementation_status false -}}


Thank you for checking this.

emkornfield

Overall, looks good to me, I think some of the logical changes are actually forward incompatible but that might be a broader conversation on what exactly is compatible/incompatible with them.

Thank you for doing this.

alamb

Thank you @etseidl and @emkornfield

I reworked the rendering to

use a table for each parquet format release to avoid fancy rendering logic
restore a Notes column
quoted the version numbers

I have also re-generated the rendered preview https://alamb.github.io/parquet-site/docs/file-format/versions/

I think there are two follow ons here:

Figure out what to do with the features that are in parquet-format but don't have entry on implementation status: https://github.com/apache/parquet-site/pull/189/changes#r3470611445
@etseidl 's suggestion to research the approving PRs #189 (comment) (I'll do this as a follow on PR)

Stuff I am not really sure how to proceed:

@emkornfield 's comment in https://github.com/apache/parquet-site/pull/189/changes#r3470700287 about what actually constitutes "forward compatible" -- I think this is a larger consensus building exercise so I would like to not address it in this PR

alamb · 2026-06-26T17:27:06Z

+        {{- $featureUrl = $feature.spec_url -}}
+      {{- end -}}
+      {{- $releasedIn := printf "%v" $feature.released_in -}}
+      {{- $legacyReleaseTags := slice "1.0.0" "2.0.0" "2.1.0" -}}


Apparently in hugo makes an array -- this is part of some overly complicated logic to translate release names like 1.0.0 into tag names (whose format changed over time)

I simplified the whole thing by changing this to a lookup table -- which simplifies this logic and I think makes it clearer what is going on

alamb · 2026-06-26T17:38:45Z

  - id: encoding-byte-stream-split-extended
    display_name: BYTE_STREAM_SPLIT (Additional Types)
    spec_url: https://github.com/apache/parquet-format/commit/e517ac4dbe08d518eb5c2e58576d4c711973db94
-    format_version:


Good idea -- I quoted them all in commit 9f8d58a

emkornfield · 2026-06-26T18:16:19Z

Figure out what to do with the features that are in parquet-format but don't have entry on implementation status: https://github.com/apache/parquet-site/pull/189/changes#r3470611445

Honestly I think most of these are so old, that it doesn't necessary add value, other then maybe providing a note on that page? Is there anything worth surfacing?

Looking more closely, these seem to be the modern ones:

logical-variant-shredding - should be covered under variant type support? But maybe called out separately
format-geospatial-statistics - should be covered under geospatial support? But maybe called out separately
format-binary-protocol-extensions - Not clear if this is actually something on its own without a defined footer we want to use?
format-ieee-754-total-order-and-nan-counts - This should be added (and we would ideally add a 2026 feature column for read support).

@emkornfield 's comment in https://github.com/apache/parquet-site/pull/189/changes#r3470700287 about what actually constitutes "forward compatible" -- I think this is a larger consensus building exercise so I would like to not address it in this PR

Yeah, I have a feeling this might come up in the context of versioning conversations no need to block.

alamb · 2026-06-26T18:36:31Z

Looking more closely, these seem to be the modern ones:

I will try and address it in a follow on PR

etseidl · 2026-06-26T19:31:30Z

2. @etseidl 's suggestion to research the approving PRs #189 (comment) (I'll do this as a follow on PR)

I certainly didn't mean to task you 😉. I'd be happy to compile the data...I'd just need rendering help.

alamb · 2026-06-26T19:35:04Z

@etseidl 's suggestion to research the approving PRs #189 (comment) (I'll do this as a follow on PR)

I certainly didn't mean to task you 😉. I'd be happy to compile the data...I'd just need rendering help.

Oh no problem -- I'll send an agent to do it and then verify. Your help will be most useful in helping to review it as well

alamb · 2026-07-01T16:35:52Z

I think I have addressed all feedback in this PR -- so I will plan to merge it in tomorrow unless people would like more time or have other comments. I'll also file tickets to track follow on work

alamb · 2026-07-07T13:20:40Z

@etseidl 's suggestion to research the approving PRs #189 (comment) (I'll do this as a follow on PR)

I certainly didn't mean to task you 😉. I'd be happy to compile the data...I'd just need rendering help.

Oh no problem -- I'll send an agent to do it and then verify. Your help will be most useful in helping to review it as well

filed a ticket Add specific approval PRs to the features page #191

alamb · 2026-07-07T13:22:47Z

Looking more closely, these seem to be the modern ones:

I will try and address it in a follow on PR

Tracking in

Rationalize features that are in parquet-format but don't have entry on implementation status #192

alamb · 2026-07-07T17:45:22Z

@etseidl here is a PR to add the PR research

Add PRs column linking features to their spec pull requests #193

alamb added 2 commits June 24, 2026 12:05

Make format versions data driven

86274fa

Use reference links for footer note

708f012

alamb commented Jun 24, 2026

View reviewed changes

alamb added 2 commits June 24, 2026 12:28

Restore format version explanatory note

1fe107f

Revert "Restore format version explanatory note"

5875d4f

This reverts commit 1fe107f.

alamb marked this pull request as ready for review June 24, 2026 16:33

alamb mentioned this pull request Jun 24, 2026

Document Parquet Features by Version #186

Merged

emkornfield reviewed Jun 24, 2026

View reviewed changes

emkornfield approved these changes Jun 24, 2026

View reviewed changes

alamb added 7 commits June 26, 2026 13:08

Add comments about what shortcode is doing

fbd7406

Make approval / dates expicit and simplfy short code

e7dc3b0

simplify short code again

7ef051a

simplify comments

75652eb

simplify comments

9b1c6b5

restore notes

c0258a8

Quote release versions

9f8d58a

alamb commented Jun 26, 2026

View reviewed changes

alamb merged commit 09e2b29 into apache:production Jul 7, 2026
1 check passed

alamb mentioned this pull request Jul 7, 2026

Add specific approval PRs to the features page #191

Open

alamb mentioned this pull request Jul 7, 2026

Rationalize features that are in parquet-format but don't have entry on implementation status #192

Open

alamb mentioned this pull request Jul 7, 2026

Add PRs column linking features to their spec pull requests #193

Open


		Forward incompatible features and the format version each became available in:

		\| Feature \| Released in \| Source \| Notes \|

Uh oh!

Conversation

alamb commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale

Changes

Planned follow on

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jun 26, 2026

Uh oh!

etseidl commented Jun 26, 2026

Uh oh!

alamb commented Jun 26, 2026

Uh oh!

alamb commented Jul 1, 2026

Uh oh!

Uh oh!

alamb commented Jul 7, 2026

Uh oh!

alamb commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alamb commented Jun 24, 2026 •

edited

Loading

emkornfield commented Jun 26, 2026 •

edited

Loading

alamb commented Jul 7, 2026 •

edited

Loading