Skip to content

Implement bound truncation in parquet writer #29164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ballard26 wants to merge 10 commits into
base: dev
Choose a base branch
from
Draft

Implement bound truncation in parquet writer #29164

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6791d83
bytes: allow for pointers to all byte-sized types in io_placeholder
ballard26 Jan 7, 2026
81b6f4d
bytes: expose reverse fragment iterator in iobuf
ballard26 Jan 7, 2026
3abadda
bytes: implement reverse byte iterator for iobuf
ballard26 Jan 7, 2026
6262bca
bytes: fix case where the last rhs fragment is skipped during comparison
ballard26 Jan 7, 2026
b004a33
strings: add additional utils for utf-encoded byte sequences
ballard26 Jan 7, 2026
af2da8f
serde/parquet: use zero-copy for bound encoding
ballard26 Jan 7, 2026
6cccc78
test_utils: make bound checks more permissive in parquet_verifier
ballard26 Jan 7, 2026
96e9f3e
serde/parquet: add min/max value truncators for byte array types
ballard26 Jan 7, 2026
9ce1d1a
serde/parquet: integrate bound truncation into parquet writer
ballard26 Jan 7, 2026
c3837e1
serde/parquet/tests: increase size of generated byte array fields
ballard26 Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion src/v/bytes/details/io_placeholder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,10 @@ class io_placeholder {
, _byte_index(initial_index)
, _remaining_size(max_size_to_write) {}

[[gnu::always_inline]] void write(const char* src, size_t len) {
template<typename T>
[[gnu::always_inline]] void write(const T* src, size_t len)
requires(sizeof(T) == 1)
Copy link
Member

@travisdowns travisdowns Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function needs a doc now that it's not entirely obvious what it is doing.

It will do an element-wise copy from the src array to this placeholder, if T can be assigned to a char, right?

{
details::check_out_of_range(len, _remaining_size);
std::copy_n(src, len, mutable_index());
_remaining_size -= len;
Expand Down