Skip to content

Commit

Permalink
refactor: Remove unnecessary whitespace
Browse files Browse the repository at this point in the history
Remove redundant spaces to improve code readability and consistency

Co-authored-by: Daijiro Fukuda <[email protected]>
Signed-off-by: ddukbg <[email protected]>

refactor: Simplify data compression logic

refactor: Simplify data compression logic

Remove duplicate file reading and streamline compression process

Co-authored-by: Daijiro Fukuda <[email protected]>
Signed-off-by: ddukbg <[email protected]>
  • Loading branch information
ddukbg and daipom committed Nov 2, 2024
1 parent 0ad8cdf commit 7e28a32
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 10 deletions.
6 changes: 1 addition & 5 deletions lib/fluent/plugin/out_s3.rb
Original file line number Diff line number Diff line change
Expand Up @@ -641,11 +641,7 @@ def content_type
end

def compress(chunk, tmp)
uncompressed_data = ''
chunk.open do |io|
uncompressed_data = io.read
end
compressed_data = Zstd.compress(uncompressed_data, level: @level)
compressed_data = Zstd.compress(chunk.read, level: @level)
tmp.write(compressed_data)
rescue => e
log.warn "zstd compression failed: #{e.message}"
Expand Down
10 changes: 5 additions & 5 deletions test/test_out_s3.rb
Original file line number Diff line number Diff line change
Expand Up @@ -467,20 +467,20 @@ def test_write_with_custom_s3_object_key_format_containing_hex_random_placeholde
def test_write_with_zstd
setup_mocks(true)
s3_local_file_path = "/tmp/s3-test.zst"

expected_s3path = "log/events/ts=20110102-13/events_0-#{Socket.gethostname}.zst"

setup_s3_object_mocks(s3_local_file_path: s3_local_file_path, s3path: expected_s3path)

config = CONFIG_TIME_SLICE + "\nstore_as zstd\n"
d = create_time_sliced_driver(config)

time = event_time("2011-01-02 13:14:15 UTC")
d.run(default_tag: "test") do
d.feed(time, { "a" => 1 })
d.feed(time, { "a" => 2 })
end

File.open(s3_local_file_path, 'rb') do |file|
compressed_data = file.read
uncompressed_data = Zstd.decompress(compressed_data)
Expand Down

0 comments on commit 7e28a32

Please sign in to comment.