Implement RemoveFiles for S3FileSystem

[utay]

Following this discussion and duckdb/duckdb#20333 which introduces RemoveFiles on FileSystem, this PR implements RemoveFiles for S3FileSystem.

It uses the S3 DeleteObjects API to delete objects in batches of 1000.

I couldn't find how to test this, as the other methods (like RemoveFile) don't seem to be unit tested, and the RemoveFiles method is not exposed as a SQL command. Please let me know if there's a way to do so! I was still able to test it end-to-end with a SQL test in the ducklake repo (PR incoming).

[pdet]

Hi @utay, can you retarget this PR to main? I'm also unaware of a way to test the deletion function directly. Maybe @samansmink or @carlopi has an idea?

[utay]

Hi @utay, can you retarget this PR to main?

Done!

I'm also unaware of a way to test the deletion function directly. Maybe @samansmink or @carlopi has an idea?

Yep, let me know, happy to help write a test

[carlopi]

I need to bump duckdb further, since currently it doesn't include duckdb/duckdb#20333 (see failure in MinIO tests). I'll ping once this can be rebased.

[carlopi]

@utay: can you merge with / rebase to the main branch? Should be ready after that CI side.

I think the change set might slightly change behaviour, and would indeed need to be more properly tested.

I am not sure of what strategy to recommend, probably combination of populating files + globs + deletion + globs in SQL to be execute as integration test would be handy.

[utay]

@carlopi Thanks, I've rebased the branch on main.

I’ll fix the compilation issues and try to add an integration test as you suggested when I have some time!

[utay]

@carlopi I've fixed compilation, CI should pass now.

Regarding the integration test, I think there are 2 options:

• Rely on the COPY overwrite mode, but this would require removing this check
  in DuckDB. Also, this doesn’t explicitly delete the existing files; it overwrites them implicitly
• Introduce a SQL function like s3_remove_files (or similar) to explicitly clean up the target path. There don’t seem to be any existing S3 helper functions for this today, but adding one might be acceptable

I couldn't find another way to test this with the existing test infrastructure, let me know what you think!

[carlopi]

@utay: I've opened duckdb/duckdb#20487, I think removing the check is the path forward, but need to check with the team.