Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 66 additions & 1 deletion LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -635,7 +635,72 @@ The type has two type parameters:

The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
are found during reading, they must be ignored.
are found during reading, they must be ignored.

### FILE

`FILE` annotates a group that represents a reference to an external file, along with
the minimum metadata required to read it. It is intended for use cases such as
storing file inventories, manifests, and unstructured data references (e.g., images
or audio files stored in object storage).

The annotated group must contain the following fields, identified by name. Field IDs
Comment thread
brkyvz marked this conversation as resolved.
Outdated
may also be used for projection:
Comment thread
etseidl marked this conversation as resolved.
Outdated
Comment thread
brkyvz marked this conversation as resolved.
Outdated

| Field | Type | Required |
Comment thread
brkyvz marked this conversation as resolved.
Outdated
|----------|--------|----------|
| `path` | STRING | Yes |
| `size` | INT64 | No |
| `offset` | INT64 | No |
| `etag` | STRING | No |
Comment thread
brkyvz marked this conversation as resolved.
Outdated

Comment thread
brkyvz marked this conversation as resolved.
#### path

An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special
Comment thread
brkyvz marked this conversation as resolved.
Outdated
encoding (e.g., URI encoding) is applied. This is the only required field.
Comment thread
brkyvz marked this conversation as resolved.
Outdated

Comment thread
etseidl marked this conversation as resolved.
#### size

The length of the content in bytes. Must be zero or a positive integer if provided.
A value of 0 indicates an empty file.
Comment thread
brkyvz marked this conversation as resolved.
Outdated

#### offset

A byte offset for range reads. If not provided, readers must treat the value as 0.
Comment thread
brkyvz marked this conversation as resolved.
Outdated
If provided and non-zero, readers must seek to this offset and read `size` bytes.
Comment thread
brkyvz marked this conversation as resolved.
Outdated
If `offset` is provided, `size` must also be provided.

#### etag

An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage).
Can be used to detect whether the referenced file has been updated. If the reference
points to a byte range within a file, the eTag applies to the entire file.
Comment thread
brkyvz marked this conversation as resolved.
Outdated

Validation rules for readers and writers:
Comment thread
brkyvz marked this conversation as resolved.
Outdated

* The `path` field is required and must be present. Readers must reject a `FILE`-annotated
group that does not contain `path`.
Comment thread
brkyvz marked this conversation as resolved.
Outdated
* If `offset` is present and non-zero, `size` must also be provided.
* Additional metadata about the file (e.g., content type, modification timestamp) should
Comment thread
brkyvz marked this conversation as resolved.
Outdated
be stored adjacent to this struct by engines or table formats, not inside it.

Statistics may be collected for the individual fields of a `FILE`-annotated group
according to the sort order of each field's logical type.

This is an example `FILE`-annotated group in Parquet:

```
optional group my_file (FILE) {
required binary path (STRING);
optional int64 size;
optional int64 offset;
optional binary etag (STRING);
}
```

*Compatibility*
Comment thread
brkyvz marked this conversation as resolved.
Outdated

`FILE` has no corresponding `ConvertedType`.

## Nested Types

Expand Down
16 changes: 16 additions & 0 deletions src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -468,6 +468,21 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}

/**
* File logical type annotation
*
* Annotates a group that represents a reference to an external file.
* The group must contain the following fields identified by name:
Comment thread
brkyvz marked this conversation as resolved.
Outdated
* - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg)
Comment thread
brkyvz marked this conversation as resolved.
Outdated
Comment thread
brkyvz marked this conversation as resolved.
Outdated
* - size (INT64, optional): the length of the content in bytes; must be zero or positive
* - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided
Comment thread
brkyvz marked this conversation as resolved.
Outdated
* - etag (STRING, optional): eTag from the storage system for staleness detection
Comment thread
brkyvz marked this conversation as resolved.
Outdated
*
* See LogicalTypes.md for details.
*/
struct FileType {
}

/**
* LogicalType annotations to replace ConvertedType.
*
Expand Down Expand Up @@ -501,6 +516,7 @@ union LogicalType {
16: VariantType VARIANT // no compatible ConvertedType
17: GeometryType GEOMETRY // no compatible ConvertedType
18: GeographyType GEOGRAPHY // no compatible ConvertedType
19: FileType FILE // no compatible ConvertedType
Comment thread
brkyvz marked this conversation as resolved.
}

/**
Expand Down