Introduces a new LogicalType: FILE

LogicalTypes.md

@@ -635,7 +635,164 @@ The type has two type parameters:
 
 The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
 statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored. 
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes, which may
+be stored inline in the value, elsewhere within the current file, or in an external file. It
+is intended for use cases such as storing file inventories, manifests, and unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name. Field IDs
+may also be used for projection. Every field is optional both in the schema and in the
+data: a writer may omit any field from the group definition, and any field that is
+present has a field repetition type of `OPTIONAL`. A group need only define the fields
+it uses (for example, an inline-only group may define just `inline`, and an external
+reference may define just `path`).
+
+| Field          | Type       |
+|----------------|------------|
+| `path`         | STRING     |
+| `offset`       | INT64      |
+| `size`         | INT64      |
+| `content_type` | STRING     |
+| `checksum`     | STRING     |
+| `inline`       | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `path` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE` group
+and its value is non-null (and, for string fields, non-empty). A field is *not set*
+when it is absent from the group, or is present but null or empty.
+
+##### path
+
+An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special
+encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is
+not set, the value refers to the current file (a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the referenced data.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if set; a
+value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set
+or `path` is not set. It may be omitted only for a whole-file external reference (`path`
+set, `offset` not set), in which case the range runs to the end of the referenced file.
+
+##### content_type
+
+The media type (MIME type) of the resolved bytes (e.g., `image/png`).
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:base64(<digest bytes>)`. It generalizes the storage-system eTag. The
+recognized algorithms are:
+
+| Algorithm | Notes                                                          |
+|-----------|----------------------------------------------------------------|
+| `ETAG`    | the object-store eTag — equality-only, not recomputable        |
+| `MD5`     | the usual S3/HTTP eTag and Content-MD5                         |
+| `CRC32`   | Parquet's page-checksum algorithm (gzip/zlib)                 |
+| `CRC32C`  | common in object stores, hardware-accelerated                  |
+| `SHA-256` | e.g. S3 additional checksums                                  |
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `path`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it supplies the
+bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are
+set:
+
+| `inline` | `path` | `offset` | `size` | Resolves to                                           |
+|----------|--------|----------|--------|-------------------------------------------------------|
+| set      | –      | –        | –      | the inline bytes                                      |
+| –        | set    | –        | –      | whole external file at `path`                         |
+| –        | set    | –        | set    | external `path`, `[0, size)`                          |
+| –        | set    | set      | set    | external `path`, `[offset, offset + size)`            |
+| –        | –      | –        | set    | current file, `[0, size)` (self-reference)            |
+| –        | –      | set      | set    | current file, `[offset, offset + size)` (self-reference) |
+| –        | –      | –        | –      | nothing — invalid                                     |
+
+`size` must be set whenever `offset` is set or `path` is not set, so a self-reference
+and any offset-based read always carry an explicit `size`. It may be omitted only for a
+whole-file external reference, where the range runs to the end of the referenced file.
+
+A self-reference typically points within the same Parquet file using `offset` and
+`size`; the bytes are written between column chunks and are not otherwise referenced by
+the footer. A self-reference is when `path` is not set, never an absolute path back to
+the current file, so a file containing self-references is renamed or relocated as a
+single unit.
+
+The bytes referenced by a self-reference are compressed with the same `CompressionCodec`
+as the one specified for the `inline` column.
+
+#### Validation
+
+* A value must resolve to some referenced data. If none of `inline`, `path`, `offset`, or
+  `size` is set, the value does not resolve and is invalid; use column nullability to
+  represent a null value.
+* `size` must be set whenever `offset` is set or `path` is not set. A value that sets
+  `offset` without `size`, or is a self-reference (no `path`) without `size`, is invalid.
+* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the
+  locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+  be stored adjacent to this group by engines or table formats, not inside it.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated group
+according to the sort order of each field's logical type.
+
+This is an example of a `FILE`-annotated group that defines all fields:
+
+```
+optional group my_file (FILE) {
+  optional binary path (STRING);
+  optional int64 offset;
+  optional int64 size;
+  optional binary content_type (STRING);
+  optional binary checksum (STRING);
+  optional binary inline;
+}
+```
+
+Because every field is optional, a group need only define the fields it uses. A group
+whose values are always stored inline may define just `inline`:
+
+```
+optional group inline_file (FILE) {
+  optional binary inline;
+  optional binary content_type (STRING);
+}
+```
+
+A group whose values are always whole external files may define just `path` and optionally
+`content_type` and `checksum` for validation.:
+
+```
+optional group external_file (FILE) {
+  optional binary path (STRING);
+  optional binary content_type (STRING);
+  optional binary checksum (STRING);
+}
+```
+
 
 ## Nested Types
 

src/main/thrift/parquet.thrift

@@ -468,6 +468,35 @@ struct GeographyType {
   2: optional EdgeInterpolationAlgorithm algorithm;
 }
 
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file. All fields are optional and are identified by name:
+ *   - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg);
+ *     if absent, the value refers to this file (self-reference)
+ *   - offset (INT64): start of the byte range; if absent, treated as 0
+ *   - size (INT64): byte length of the referenced data; required when `offset` is set
+ *     or `path` is absent. May be omitted only for a whole-file external reference
+ *     (`path` set, `offset` absent), in which case the range runs to the end of the
+ *     referenced file
+ *   - content_type (STRING): media type (MIME) of the resolved bytes
+ *   - checksum (STRING): an algorithm-tagged integrity token for the resolved
+ *     bytes, of the form "<algorithm>:base64(<digest>)"
+ *   - inline (BYTE_ARRAY): the referenced bytes stored inline in the value
+ *
+ * A value resolves to bytes determined by inline / path / offset / size; if
+ * inline is set it supplies the bytes and any locator fields are provenance
+ * only. A value with none of inline, path, offset, or size set does not
+ * resolve to any referenced data and is invalid (use column nullability for nulls).
+ * content_type and checksum are metadata describing whichever bytes resolve.
+ *
+ * See LogicalTypes.md for details.
+ */
+struct FileType {
+}
+
 /**
  * LogicalType annotations to replace ConvertedType.
  *
@@ -501,6 +530,7 @@ union LogicalType {
   16: VariantType VARIANT     // no compatible ConvertedType
   17: GeometryType GEOMETRY   // no compatible ConvertedType
   18: GeographyType GEOGRAPHY // no compatible ConvertedType
+  19: FileType FILE           // no compatible ConvertedType
 }
 
 /**