Skip to content

feat: Allow ducklake_merge_adjacent_files to optionally run across schema version#1049

Open
zhming0 wants to merge 1 commit into
duckdb:mainfrom
zhming0:ming/feat
Open

feat: Allow ducklake_merge_adjacent_files to optionally run across schema version#1049
zhming0 wants to merge 1 commit into
duckdb:mainfrom
zhming0:ming/feat

Conversation

@zhming0
Copy link
Copy Markdown

@zhming0 zhming0 commented Apr 21, 2026

This solves #1048.
Fixes #536.

Summary

Adds an opt-in allow_cross_schema boolean parameter to ducklake_merge_adjacent_files (default false). When set, files written under different schema_versions are merged together into the current schema.
Fixes the limitation where ADD COLUMN / RENAME / etc. leave small files stranded on each side of a schema seam forever.

Safety considration

  • Default behavior is unchanged, current test compaction_schema_version_per_table.test still passes.
  • I suppose not all schema changes are compatible to each other, but I propose that we leave this decision to user to decide.

Usage

CALL ducklake_merge_adjacent_files('ducklake', 't', allow_cross_schema => true);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

merge_adjacent_files not compacting a table with lots of single row files

1 participant