Introduces a new LogicalType: FILE by brkyvz · Pull Request #585 · apache/parquet-format

brkyvz · 2026-06-09T22:04:45Z

Rationale for this change

Introduces a new type called File as a typed FileReference. The design document is here.

The motivation is as follows:

Unstructured data ingestion is getting extremely popular with the advances in Generative AI. 
Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, 
or point to files that exist in some object store with a string. These solutions fail to address these use 
cases, because of scalability, usability, and governance issues.

We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that 
contains a path reference to a file with additional metadata. This reference may be to a file that exists 
(or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields 
that would allow a client to correctly read the referenced data. Any additional metadata can be optionally 
stored by engines and table formats as necessary adjacent to this type.

What changes are included in this PR?

Introduces the specification for FileType.

Do these changes have PoC implementations?

Yes:

emkornfield

Looks reasonable to me a few minor comments for clarification.

etseidl

Just a few questions I have after reviewing the Rust implementation.

alkis · 2026-07-06T07:28:27Z

+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.


Let's make this clearer.

Suggested change

The referenced bytes are compressed with the same `CompressionCodec` as the one

specified for the `inline` column.

The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.

Should we point out CompressionCodec can differ per page?

@alkis thoughts? Is there a reason that the data needs to be compressed additionally? Most formats may be optimal anyway.

The reason is because we are making the assumption that the type of data stored in the file column is homogeneous. I argue that's a good assumption. If this column contains text, there is little reason to assume it will be a string when small and an image when large. Ergo if the writer chose to compress the column it should compress the self-referenced packed blobs too.

I wanted to make all packed references compressed as well but there was some pushback for that with the argument that external may be referenced by different parquet files and it may choose its own compression scheme (or none at all). I can see arguments both ways - it is a tradeoff.

Making all packed references inherit the CompressionCodec makes the spec more consistent.
Making only self-references inherit the CompressionCodec is friendlier to shared external references.

We pick one of the two, it is a tradeoff.

Should we point out CompressionCodec can differ per page?

brkyvz · 2026-07-08T01:11:38Z

@etseidl @sfc-gh-sgrafberger @alkis @rdblue addressed your comments, can you please take another look?

etseidl

Thanks @brkyvz, this looks good to me now.

sfc-gh-sgrafberger

Thank you @brkyvz! Looks good to me.

…a-io#585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR delta-io#585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac

brkyvz · 2026-07-10T15:53:47Z

Addressed your comments, @rok @pitrou please take a look. There's only one unaddressed comment on the CompressionCodec, which I will address after talking to @alkis

rdblue · 2026-07-13T20:37:21Z

+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*


Nit: we should be clear that implementations are not expected to treat empty strings as null.

rdblue · 2026-07-13T20:42:22Z

+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).


Minor: would be nice to have a reference like the URI reference. I thought that we also wanted to have an assumed MIME type, application/octet-stream? I'm fine if it isn't there, but I remember it coming up.

rdblue · 2026-07-13T20:44:21Z

+| `SHA-256` | e.g. S3 additional checksums                                    |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.


An example or a few examples would help here because of the use of base64 in the first paragraph.

Are we really going to use base64 for formats that are normally hexadecimal like etags?

rdblue · 2026-07-13T20:46:42Z

+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by


I don't think that we should specify where bytes must be written. Other restrictions apply, but it would be reasonable to place these just before the indexes and footer as well.

rdblue · 2026-07-13T20:48:17Z

+
+A self-reference points within the same Parquet file using `offset` and `size` (both
+required); the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to


The phrase "never an absolute path back to the current file" sounds like you're saying "path must not be an absolute reference to the current file and must instead be unset". If that's true, I'd use a more clear phrasing to instruct people to replace self-references with null.

rdblue · 2026-07-13T20:49:10Z

+
+* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or
+  `offset` is set; if none of them are set, the value does not resolve and is invalid, even
+  if `size` is set. Use column nullability to represent a null value.


Use column nullability to represent a null value

But empty string is considered unset?

rdblue · 2026-07-13T20:50:02Z

+* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor
+  `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+  is invalid. Because a self-reference must set `offset`, it must also set `size`.


How should implementations behave if offset is set but size is not? Are they required to fail?

rdblue · 2026-07-13T20:50:40Z

+* `size` must be set whenever `offset` is set. A value that sets `offset` without `size`
+  is invalid. Because a self-reference must set `offset`, it must also set `size`.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+  locator fields as mutually exclusive.


Readers may not because locator fields may be persisted to record provenance?

rdblue

I left a few comments for clarification, but I think that this is ready either way.

Add FILE type definitions

7d966f0

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield approved these changes Jun 10, 2026

View reviewed changes

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Address comments

b6ae0de

RussellSpitzer approved these changes Jun 10, 2026

View reviewed changes

danielcweeks approved these changes Jun 10, 2026

View reviewed changes

etseidl reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

wgtmac reviewed Jun 11, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

divjotarora reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jun 12, 2026

View reviewed changes

Comment thread LogicalTypes.md

alkis reviewed Jun 17, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

CTTY reviewed Jun 17, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

pitrou reviewed Jun 30, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jul 2, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift

brkyvz added 2 commits July 2, 2026 16:30

update spec based on Alkis' proposal

0f05340

minor changes

bcbabcc

etseidl reviewed Jul 2, 2026

View reviewed changes

alkis reviewed Jul 6, 2026

View reviewed changes

This was referenced Jul 6, 2026

[PROTOCOL RFC] File Data Type delta-io/delta#7147

Open

[Protocol RFC] File data type delta-io/delta#7148

Open

brkyvz added 2 commits July 7, 2026 22:05

address comments

117b3d1

save changes

4ff444e

address more

278c9c0

Save

7186bca

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

alkis reviewed Jul 8, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

etseidl approved these changes Jul 8, 2026

View reviewed changes

sfc-gh-sgrafberger approved these changes Jul 8, 2026

View reviewed changes

brkyvz added 2 commits July 8, 2026 18:03

address comments

a60d312

address final comments

f03610e

rok reviewed Jul 9, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rok reviewed Jul 9, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

address comments, update path spec to URI

8a2ae51

danielcweeks approved these changes Jul 13, 2026

View reviewed changes

rdblue reviewed Jul 13, 2026

View reviewed changes

rdblue approved these changes Jul 13, 2026

View reviewed changes

		The referenced bytes are compressed with the same `CompressionCodec` as the one
		specified for the `inline` column.

	The referenced bytes are compressed with the same `CompressionCodec` as the one
	specified for the `inline` column.
	The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.


		##### content_type

		The media type (MIME type) of the resolved bytes (e.g., `image/png`).

Uh oh!

Conversation

brkyvz commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Uh oh!

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brkyvz commented Jul 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

brkyvz commented Jun 9, 2026 •

edited

Loading