inlined tables not being cleaned up

[tkerwin]

What happens?

I'm using postgres as the catalog database and a simple networked filesystem as the data path.

When I run checkpoint; I would have expected the ducklake_inlined_data_*_* files to be removed from the postgres database, but they are not. This seems to be related to #1145 but it doesn't seem to be working for my usage.

I've tested this with DuckDB v1.5.3 and doing force install ducklake from core_nightly;, however, my ducklake was originally created using a slightly earlier version.

To Reproduce

Insert some data into the ducklake.

Call checkpoint;

Check to see if inline tables still exist.

OS:

Windows

DuckDB Version:

1.5.3

DuckLake Version:

core_nightly

DuckDB Client:

DuckDB cli

Hardware:

No response

Full Name:

Thomas Kerwin

Affiliation:

OSU

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[bhvngt]

I am facing similar issue. Here's a small script to reproduce this issue.

I am using locally hosted supabase storage for s3 storage and supabase postgres for ducklake catalog database.

I replacing the same table to mimick separate duckdb_inlined_data_*_* creation.

I have called the maintenance task that I believe should have dropped the duckdb_inlined_data_*_* tables except may be the one pointing to last living snapshot, since all of the snapshots are expired.

CREATE OR REPLACE SECRET supa_catalog (TYPE postgres, HOST localhost, PORT 54322, DATABASE 'postgres', USER 'postgres', PASSWORD 'postgres');
ATTACH IF NOT EXISTS '' AS supa (TYPE postgres, SECRET supa_catalog);

CREATE OR REPLACE SECRET supa_storage (TYPE s3, KEY_ID '625729a08b95bf1b7ff351a663f3a23c', SECRET '850181e4652dd023b7a98c58ae0d2d34bd487ee0cc3254aed6eda37307425907', REGION 'local', ENDPOINT '127.0.0.1:54321/storage/v1/s3', URL_STYLE 'path', USE_SSL FALSE, SCOPE 's3://test/');

DROP SCHEMA IF EXISTS supa.test_lake CASCADE;
CREATE SCHEMA supa.test_lake;

CREATE OR REPLACE SECRET lake_secret (TYPE ducklake, METADATA_PATH '', DATA_PATH 's3://test/', METADATA_PARAMETERS MAP {'TYPE': 'postgres', 'SECRET': 'supa_catalog', 'SCHEMA': 'test_lake'});
ATTACH 'ducklake:lake_secret' AS lake;

CREATE OR REPLACE TABLE lake.test_table (id INT);
CREATE OR REPLACE TABLE lake.test_table (id INT);
CREATE OR REPLACE TABLE lake.test_table (id INT);

SELECT COUNT(*) FROM supa.test_lake.ducklake_inlined_data_tables; 
-- prints 3

CALL ducklake_flush_inlined_data('lake');
CALL ducklake_expire_snapshots('lake', older_than => now());
CALL ducklake_cleanup_old_files('lake', cleanup_all => true);

SELECT COUNT(*) FROM supa.test_lake.ducklake_inlined_data_tables; 
-- prints 3
-- Expected 1 referring to snapshot pointing to current schema