Skip to content

Commit 82221d9

Browse files
committed
add supported operations for attached catalogs plus interoperability with DuckLake
1 parent 923cdd7 commit 82221d9

File tree

3 files changed

+66
-19
lines changed

3 files changed

+66
-19
lines changed

docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md

Lines changed: 64 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -50,24 +50,74 @@ SHOW ALL TABLES;
5050
A REST Catalog with OAuth2 authorization can also be attached with just an `ATTACH` statement. See the complete list of `ATTACH` options for a REST catalog below.
5151

5252

53-
| Parameter | Type | Default | Description |
54-
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
55-
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
56-
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
57-
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
58-
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
59-
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
60-
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
61-
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
62-
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization |
53+
| Parameter | Type | Default | Description |
54+
|-------------------------------|-----------|----------|-------------------------------------------------------------------------------------------------------|
55+
| `ENDPOINT_TYPE` | `VARCHAR` | `NULL` | Used for attaching S3Tables or Glue catalogs. Allowed values are 'GLUE' and 'S3_TABLES' |
56+
| `ENDPOINT` | `VARCHAR` | `NULL` | URL endpoint to communicate with the REST Catalog. Cannot be used in conjunction with `ENDPOINT_TYPE` |
57+
| `SECRET` | `VARCHAR` | `NULL` | Name of secret used to communicate with the REST Catalog |
58+
| `CLIENT_ID` | `VARCHAR` | `NULL` | CLIENT_ID used for Secret |
59+
| `CLIENT_SECRET` | `VARCHAR` | `NULL` | CLIENT_SECRET needed for Secret |
60+
| `DEFAULT_REGION` | `VARCHAR` | `NULL` | A Default region to use when communicating with the storage layer |
61+
| `OAUTH2_SERVER_URI` | `VARCHAR` | `NULL` | OAuth2 server url for getting a Bearer Token |
62+
| `AUTHORIZATION_TYPE` | `VARCHAR` | `OAUTH2` | Pass `SigV4` for Catalogs the require SigV4 authorization, `none` for catalogs that don't need auth |
63+
| `SUPPORTED_NESTED_NAMESPACES` | `BOOLEAN` | `true` | Option for catalogs that support nested namespaces. |
64+
| `SUPPORT_STAGE_CREATE` | `BOOLEAN` | `false` | Option for catalogs that do not support stage create. |
65+
6366

6467

6568
The following options can only be passed to a `CREATE SECRET` statement, and they require `AUTHORIZATION_TYPE` to be `OAUTH2`
6669

67-
| Parameter | Type | Default | Description |
68-
| ---------------------------- | ---------- | -------- | ---------------------------------------------------------- |
69-
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
70-
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |
70+
| Parameter | Type | Default | Description |
71+
|---------------------|-----------|---------|-----------------------------------------------------|
72+
| `OAUTH2_GRANT_TYPE` | `VARCHAR` | `NULL` | Grant Type when requesting an OAuth Token |
73+
| `OAUTH2_SCOPE` | `VARCHAR` | `NULL` | Requested scope for the returned OAuth Access Token |
74+
75+
76+
### Supported Operations
77+
78+
The DuckDB Iceberg extensions supports the following operations when used with a REST catalog attached:
79+
80+
- `CREATE/DROP SCHEMA`
81+
- `CREATE/DROP TABLE`
82+
- `INSERT INTO`
83+
- `SELECT`
84+
- `ALTER TABLE`
85+
86+
Since these operations are supported, the following would also work:
87+
88+
```sql
89+
COPY FROM DATABASE duckdb_db TO iceberg_datalake;
90+
91+
-- Or
92+
COPY FROM DATABASE iceberg_datalake to duckdb_db;
93+
```
94+
95+
This functionality enables deep copies between Iceberg and DuckDB storage.
96+
97+
### Interoperability with DuckLake
98+
99+
The DuckDB Iceberg extensions exposes a function to do metadata only copies of the Iceberg metadata to DuckLake, which enables users to query Iceberg tables as if they where DuckLake tables.
100+
101+
```sql
102+
-- Given that we have an Iceberg catalog attached aliased to iceberg_datalake
103+
ATTACH `ducklake:my_ducklake.ducklake` AS my_ducklake;
104+
105+
CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake');
106+
```
107+
108+
It is also possible to skip a set of tables provided the `skip_tables` parameter.
109+
110+
```sql
111+
CALL iceberg_to_ducklake('iceberg_datalake', 'my_ducklake', skip_tables := ['table_to_skip']);
112+
```
113+
114+
### Unsupported Operations
115+
116+
The following operations are not supported by the Iceberg DuckDB extension:
117+
118+
- `UPDATE`
119+
- `DELETE`
120+
- `MERGE INTO`
71121

72122
## Specific Catalog Examples
73123

docs/stable/core_extensions/iceberg/overview.md

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ redirect_from:
1111
title: Iceberg Extension
1212
---
1313

14-
The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
14+
The `iceberg` extension implements support for the [Apache Iceberg open table format](https://iceberg.apache.org/).
15+
In this page we will go over the basic usage of the extension without the need to attach to an Iceberg catalog. For full support —including write support— see [how to attach Iceberg REST catalogs]({% link docs/stable/core_extensions/iceberg/iceberg_rest_catalogs.md %}).
1516

1617
## Installing and Loading
1718

@@ -186,7 +187,3 @@ FROM iceberg_scan(
186187
allow_moved_paths = true
187188
);
188189
```
189-
190-
## Limitations
191-
192-
Writing (i.e., exporting to) Iceberg files is currently not supported.

tmp_ducklake.db

2.01 MB
Binary file not shown.

0 commit comments

Comments
 (0)