Introduces a new LogicalType: FILE by brkyvz · Pull Request #585 · apache/parquet-format

brkyvz · 2026-06-09T22:04:45Z

Rationale for this change

Introduces a new type called File as a typed FileReference. The design document is here.

The motivation is as follows:

Unstructured data ingestion is getting extremely popular with the advances in Generative AI. 
Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, 
or point to files that exist in some object store with a string. These solutions fail to address these use 
cases, because of scalability, usability, and governance issues.

We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that 
contains a path reference to a file with additional metadata. This reference may be to a file that exists 
(or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields 
that would allow a client to correctly read the referenced data. Any additional metadata can be optionally 
stored by engines and table formats as necessary adjacent to this type.

What changes are included in this PR?

Introduces the specification for FileType.

Do these changes have PoC implementations?

Yes:

emkornfield

Looks reasonable to me a few minor comments for clarification.

etseidl

Just a few questions I have after reviewing the Rust implementation.

wgtmac · 2026-06-11T03:28:27Z

+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:


Suggested change

The annotated group must contain the following fields, identified by name. Field IDs

may also be used for projection:

The annotated group must contain the following fields, identified by name (not by order).

Field IDs (if exist) may also be used for projection:

wgtmac · 2026-06-11T03:31:05Z

+##### size
+
+The length of the content in bytes. Must be zero or a positive integer if provided.
+A value of 0 indicates an empty file. If not provided, the length of the referenced


Do we want to advise the reader to ignore negative value here?

IMO we should specify that readers error on invalid values, not silently ignore

We can't require readers to fail, we can only say what it means when the length is -1. Failure is a responsibility of whatever uses this data, and it's even hard to require that component to fail because users don't want their query to fail for one bad row. They expect reasonable fallback behavior.

Also, does 0 indicate an empty file or does it indicate 0-length content? With offset, I think this indicates that the content length is 0, not that the underlying file is empty.

divjotarora · 2026-06-11T06:53:42Z

+A value of 0 indicates an empty file. If not provided, the length of the referenced
+content is unknown and the entirety of the content can be read.
+
+##### offset


Should we specify that it must be non-negative?

rdblue · 2026-06-11T20:47:34Z

+The annotated group must contain the following fields, identified by name. Field IDs
+may also be used for projection:
+
+| Field    | Type   | Required |


I think it would be better to use repetition instead of required, and either required or optional in the table.

I would also state that the struct must contain exactly these fields, rather than saying they are all required since "required" has a special meaning here (the Parquet type's repetition).

CTTY · 2026-06-17T19:22:26Z

+| `path`   | STRING | Yes      |
+| `size`   | INT64  | No       |
+| `offset` | INT64  | No       |
+| `etag`   | STRING | No       |


I'm neutral on whether the Parquet spec should stay minimal or include convenient metadata (content type, metadata map etc.) for common use cases

But if we are leaning toward keeping the spec simple, I don't think etag should be part of the spec either. etag feels more like storage specific metadata than a portable field with consistent semantics across storage systems

Would be happy to hear more thoughts on this!

I actually brought this up today in the Parquet sync, it's the odd one out of this pack of properties.

I concur. The type should be minimal. Also we should make all fields optional and define the semantics of them missing:

path: the file path if present, otherwise this is the path of the current file

offset: the start of the binary data, otherwise 0

size: the size of the binary data, otherwise the length of the file. size, given or derived, is capped such that offset + size <= file_size

pitrou · 2026-06-30T15:49:28Z

+ *
+ * Annotates a group that represents a reference to an external file.
+ * The group must contain the following fields identified by name:
+ *   - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)


If it's meant to be a URI, can we mandate it and call the field "uri" instead?

it's not meant to be a URI. it can be anything, but everyone asked for an example, so I provided a URI example. It's up to the clients on what they store and how they read/consume it

Disagreed. The standard should say how the field is to be interpreted. If there is no standard interpretation then this is not a proper spec.

We've gotten pretty far with it just defined as "path" within the Iceberg context. So I would probably be fine with that being the same here although I"m not sure that's actually been useful for Iceberg. I assume most consumers will attempt to parse a URI here anyway and relative paths are also allowed within the URI.

There is no unambiguous way to distinguish between URIs and "paths", so this needs to be specified explicitly.

The example shows an URI, so the spec should mandate an URI.

I think Rfc for URI covers all cases mentioned.
We could specify something like

A URI-reference as defined by RFC 3986, encoded as a Parquet STRING. If absolute, it identifies an external resource. If relative, readers must resolve it against the URI of the Parquet file containing this value using RFC 3986 section 5. If uri is not set, the value refers to the current Parquet file.

Readers may support an implementation-defined set of URI schemes. If the scheme is unsupported or the base URI is unavailable for a relative reference, the value is not resolvable.

I think it should be left to engines. e.g iceberg could choose to resolve relative paths against table's base location. Other engines might do it differently.

URIs can be relative.

To answer the "it can be left to engines" argument: this is fine for closed environments where the Parquet file never leaves some company cloud of sorts (and even there, several Parquet implementations may be involved).

But Parquet files are a widespread way of publishing and sharing columnar data. If FILE columns cannot be interpreted unambiguously, then ingesting a Parquet file needs additional hands-holding from the user.

I don't understand the push against uri. A path is a subset of URI, ARN is a URI, database path is an URI. URI comes with the contract that it is a a string identifying a resource. If the idea is to add a free form metadata field then path is not the correct name either.

alkis · 2026-07-06T07:26:21Z

+| –        | set    | set      | –      | external `path`, `[offset, EOF)`             |
+| –        | set    | –        | set    | external `path`, `[0, size)`                 |
+| –        | set    | set      | set    | external `path`, `[offset, offset + size)`   |
+| –        | –      | set      | –      | this file, `[offset, EOF)` (self-reference)  |


This would be invalid with the new requirements for size.

alkis · 2026-07-06T07:26:39Z

+|----------|--------|----------|--------|----------------------------------------------|
+| set      | –      | –        | –      | the inline bytes                             |
+| –        | set    | –        | –      | whole external file at `path`                |
+| –        | set    | set      | –      | external `path`, `[offset, EOF)`             |


This would be invalid with the new requirements for size.

alkis · 2026-07-06T07:27:04Z

+| `inline` | `path` | `offset` | `size` | Resolves to                                  |
+|----------|--------|----------|--------|----------------------------------------------|
+| set      | –      | –        | –      | the inline bytes                             |
+| –        | set    | –        | –      | whole external file at `path`                |


This would be invalid with the new requirements for size.

alkis · 2026-07-06T07:28:27Z

+The referenced bytes are compressed with the same `CompressionCodec` as the one
+specified for the `inline` column.


Let's make this clearer.

Suggested change

The referenced bytes are compressed with the same `CompressionCodec` as the one

specified for the `inline` column.

The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.

Should we point out CompressionCodec can differ per page?

brkyvz · 2026-07-08T01:11:38Z

@etseidl @sfc-gh-sgrafberger @alkis @rdblue addressed your comments, can you please take another look?

alkis · 2026-07-08T15:45:05Z

+`size` must be set whenever `offset` is set or `path` is not set, so a self-reference
+and any offset-based read always carry an explicit `size`. It may be omitted only for a
+whole-file external reference, where the range runs to the end of the referenced file.


Update this to match.

etseidl

Thanks @brkyvz, this looks good to me now.

sfc-gh-sgrafberger

Thank you @brkyvz! Looks good to me.

rok · 2026-07-09T10:09:55Z

+|----------|--------|----------|--------|-------------------------------------------------------|
+| set      | –      | –        | –      | the inline bytes                                      |
+| –        | set    | –        | –      | whole external file at `path`                         |
+| –        | set    | set      | -      | external `path`, `[offset, EOF)`                      |


As per text below ("size must be set whenever offset is set, so any offset-based read always carries an explicit size") this should be invalid?

Suggested change

| – | set | set | - | external `path`, `[offset, EOF)` |

| – | set | set | - | external `path`, invalid |

rok · 2026-07-09T10:20:12Z

+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+`offset` must be set for a self-reference (`path` not set); it is optional for an
+external reference (`path` set).


Perhaps worth specifying?

Suggested change

external reference (`path` set).

external reference (`path` set). `offset must not be < 0.

…a-io#585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR delta-io#585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac

Add FILE type definitions

7d966f0

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield approved these changes Jun 10, 2026

View reviewed changes

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

danielcweeks reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Address comments

b6ae0de

RussellSpitzer approved these changes Jun 10, 2026

View reviewed changes

danielcweeks approved these changes Jun 10, 2026

View reviewed changes

etseidl reviewed Jun 10, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

Comment thread LogicalTypes.md Outdated

wgtmac reviewed Jun 11, 2026

View reviewed changes

divjotarora reviewed Jun 11, 2026

View reviewed changes

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

rdblue reviewed Jun 11, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

sfc-gh-sgrafberger reviewed Jun 12, 2026

View reviewed changes

Comment thread LogicalTypes.md

alkis reviewed Jun 17, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

CTTY reviewed Jun 17, 2026

View reviewed changes

pitrou reviewed Jun 30, 2026

View reviewed changes

sfc-gh-sgrafberger reviewed Jul 2, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift

brkyvz added 2 commits July 2, 2026 16:30

update spec based on Alkis' proposal

0f05340

minor changes

bcbabcc

etseidl reviewed Jul 2, 2026

View reviewed changes

alkis reviewed Jul 6, 2026

View reviewed changes

This was referenced Jul 6, 2026

[PROTOCOL RFC] File Data Type delta-io/delta#7147

Open

[Protocol RFC] File data type delta-io/delta#7148

Open

brkyvz added 2 commits July 7, 2026 22:05

address comments

117b3d1

save changes

4ff444e

address more

278c9c0

Save

7186bca

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread src/main/thrift/parquet.thrift Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

emkornfield reviewed Jul 8, 2026

View reviewed changes

Comment thread LogicalTypes.md Outdated

alkis reviewed Jul 8, 2026

View reviewed changes

etseidl approved these changes Jul 8, 2026

View reviewed changes

sfc-gh-sgrafberger approved these changes Jul 8, 2026

View reviewed changes

brkyvz added 2 commits July 8, 2026 18:03

address comments

a60d312

address final comments

f03610e

rok reviewed Jul 9, 2026

View reviewed changes

		The annotated group must contain the following fields, identified by name. Field IDs
		may also be used for projection:

		The referenced bytes are compressed with the same `CompressionCodec` as the one
		specified for the `inline` column.

	The referenced bytes are compressed with the same `CompressionCodec` as the one
	specified for the `inline` column.
	The bytes referenced by a self-reference are compressed with the same `CompressionCodec` as the one specified for the `inline` column.

	\| – \| set \| set \| - \| external `path`, `[offset, EOF)` \|
	\| – \| set \| set \| - \| external `path`, invalid \|

	external reference (`path` set).
	external reference (`path` set). `offset must not be < 0.

Uh oh!

Conversation

brkyvz commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Uh oh!

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brkyvz commented Jun 9, 2026 •

edited

Loading

rok Jul 9, 2026 •

edited

Loading