Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 84 additions & 14 deletions docs/preview/core_extensions/iceberg/iceberg_rest_catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,24 +50,94 @@ SHOW ALL TABLES;
A REST Catalog with OAuth2 authorization can also be attached with just an `ATTACH` statement. See the complete list of `ATTACH` options for a REST catalog below.


| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization |
| Parameter | Type | Default | Description |
|-------------------------------|-----------|----------|-------------------------------------------------------------------------------------------------------|
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization, `none` for catalogs that don't need auth |
| `SUPPORT_NESTED_NAMESPACES` | `BOOLEAN` | `true` | Option for catalogs that support nested namespaces. |
| `SUPPORT_STAGE_CREATE` | `BOOLEAN` | `false` | Option for catalogs that do not support stage create. |



The following options can only be passed to a `CREATE SECRET` statement, and they require `AUTHORIZATION_TYPE` to be `OAUTH2`

| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |
| Parameter | Type | Default | Description |
|---------------------|-----------|---------|-----------------------------------------------------|
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |


### Supported Operations

The DuckDB Iceberg extensions supports the following operations when used with a REST catalog attached:

- `CREATE/DROP SCHEMA`
- `CREATE/DROP TABLE`
- `INSERT INTO`
- `SELECT`

Since these operations are supported, the following would also work:

```sql
COPY FROM DATABASE duckdb_db TO iceberg_datalake;

-- Or
COPY FROM DATABASE iceberg_datalake to duckdb_db;
```

This functionality enables deep copies between Iceberg and DuckDB storage.

### Metadata Operations

The functions `iceberg_metadata` and `iceberg_snapshots` are also available to use with an Iceberg REST catalog using a fully qualified path, e.g.

```sql
SELECT * FROM iceberg_metadata(my_datalake.default.t)

-- Or
SELECT * FROM iceberg_snapshots(my_datalake.default.t)
```

This functionality enables the user to grab a `snapshot_from_id` to do **time-traveling**.

```sql
SELECT * FROM my_datalake.default.t AT (VERSION => ⟨SNAPSHOT_ID⟩)

-- Or using a timestamp
SELECT * FROM my_datalake.default.t AT (TIMESTAMP => TIMESTAMP '2025-09-22 12:32:43.217')
```

### Interoperability with DuckLake

The DuckDB Iceberg extensions exposes a function to do metadata only copies of the Iceberg metadata to DuckLake, which enables users to query Iceberg tables as if they where DuckLake tables.

```sql
-- Given that we have an Iceberg catalog attached aliased to iceberg_datalake
ATTACH `ducklake:my_ducklake.ducklake` AS my_ducklake;

CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake');
```

It is also possible to skip a set of tables provided the `skip_tables` parameter.

```sql
CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake', skip_tables := ['table_to_skip']);
```

### Unsupported Operations

The following operations are not supported by the Iceberg DuckDB extension:

- `UPDATE`
- `DELETE`
- `MERGE INTO`
- `ALTER TABLE`

## Specific Catalog Examples

Expand Down
12 changes: 9 additions & 3 deletions docs/preview/core_extensions/iceberg/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ layout: docu
title: Iceberg Extension
---

The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
In this page we will go over the basic usage of the extension without the need to attach to an Iceberg catalog. For full support —including write support— see [how to attach Iceberg REST catalogs]({% link docs/preview/core_extensions/iceberg/iceberg_rest_catalogs.md %}).

## Installing and Loading

Expand Down Expand Up @@ -71,7 +72,7 @@ FROM iceberg_scan('lineitem_iceberg/metadata/v1.metadata.json');
|-------------:|
| 60175 |

The `iceberg` works together with the [`httpfs` extension]({% link docs/preview/core_extensions/httpfs/overview.md %}) or the [`azure` extension]({% link docs/preview/core_extensions/azure.md %}) to access Iceberg tables in object stores such as S3 or Azure Blob Storage.
The `iceberg` works together with the [`httpfs` extension]({% link docs/stable/core_extensions/httpfs/overview.md %}) or the [`azure` extension]({% link docs/stable/core_extensions/azure.md %}) to access Iceberg tables in object stores such as S3 or Azure Blob Storage.

```sql
SELECT count(*)
Expand Down Expand Up @@ -182,4 +183,9 @@ FROM iceberg_scan(

## Limitations

Writing (i.e., exporting to) Iceberg files is currently not supported.
- Updates and deletes.
- Inserts into v3 Iceberg specification tables.
- Reads from v3 tables with v2 data types.
- Geometry data type

For a set of unsupported operations when attaching to an iceberg catalog, [see]({% link docs/preview/core_extensions/iceberg/iceberg_rest_catalogs.md %}#unsupported-operations).
98 changes: 84 additions & 14 deletions docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,24 +50,94 @@ SHOW ALL TABLES;
A REST Catalog with OAuth2 authorization can also be attached with just an `ATTACH` statement. See the complete list of `ATTACH` options for a REST catalog below.


| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization |
| Parameter | Type | Default | Description |
|-------------------------------|-----------|----------|-------------------------------------------------------------------------------------------------------|
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization, `none` for catalogs that don't need auth |
| `SUPPORT_NESTED_NAMESPACES` | `BOOLEAN` | `true` | Option for catalogs that support nested namespaces. |
| `SUPPORT_STAGE_CREATE` | `BOOLEAN` | `false` | Option for catalogs that do not support stage create. |



The following options can only be passed to a `CREATE SECRET` statement, and they require `AUTHORIZATION_TYPE` to be `OAUTH2`

| Parameter | Type | Default | Description |
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |
| Parameter | Type | Default | Description |
|---------------------|-----------|---------|-----------------------------------------------------|
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |


### Supported Operations

The DuckDB Iceberg extensions supports the following operations when used with a REST catalog attached:

- `CREATE/DROP SCHEMA`
- `CREATE/DROP TABLE`
- `INSERT INTO`
- `SELECT`

Since these operations are supported, the following would also work:

```sql
COPY FROM DATABASE duckdb_db TO iceberg_datalake;

-- Or
COPY FROM DATABASE iceberg_datalake to duckdb_db;
```

This functionality enables deep copies between Iceberg and DuckDB storage.

### Metadata Operations

The functions `iceberg_metadata` and `iceberg_snapshots` are also available to use with an Iceberg REST catalog using a fully qualified path, e.g.

```sql
SELECT * FROM iceberg_metadata(my_datalake.default.t)

-- Or
SELECT * FROM iceberg_snapshots(my_datalake.default.t)
```

This functionality enables the user to grab a `snapshot_from_id` to do **time-traveling**.

```sql
SELECT * FROM my_datalake.default.t AT (VERSION => ⟨SNAPSHOT_ID⟩)

-- Or using a timestamp
SELECT * FROM my_datalake.default.t AT (TIMESTAMP => TIMESTAMP '2025-09-22 12:32:43.217')
```

### Interoperability with DuckLake

The DuckDB Iceberg extensions exposes a function to do metadata only copies of the Iceberg metadata to DuckLake, which enables users to query Iceberg tables as if they where DuckLake tables.

```sql
-- Given that we have an Iceberg catalog attached aliased to iceberg_datalake
ATTACH `ducklake:my_ducklake.ducklake` AS my_ducklake;

CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake');
```

It is also possible to skip a set of tables provided the `skip_tables` parameter.

```sql
CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake', skip_tables := ['table_to_skip']);
```

### Unsupported Operations

The following operations are not supported by the Iceberg DuckDB extension:

- `UPDATE`
- `DELETE`
- `MERGE INTO`
- `ALTER TABLE`

## Specific Catalog Examples

Expand Down
10 changes: 8 additions & 2 deletions docs/stable/core_extensions/iceberg/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ redirect_from:
title: Iceberg Extension
---

The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
In this page we will go over the basic usage of the extension without the need to attach to an Iceberg catalog. For full support —including write support— see [how to attach Iceberg REST catalogs]({% link docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md %}).

## Installing and Loading

Expand Down Expand Up @@ -189,4 +190,9 @@ FROM iceberg_scan(

## Limitations

Writing (i.e., exporting to) Iceberg files is currently not supported.
- Updates and deletes.
- Inserts into v3 Iceberg specification tables.
- Reads from v3 tables with v2 data types.
- Geometry data type

For a set of unsupported operations when attaching to an iceberg catalog, [see]({% link docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md %}#unsupported-operations).