Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1029631: Optimize streaming PUT memory usage #1266

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

sfc-gh-ext-simba-jl
Copy link
Collaborator

@sfc-gh-ext-simba-jl sfc-gh-ext-simba-jl commented Dec 5, 2024

Description

Optimize streaming PUT memory usage.
Compared the memory usage for uploading 1GB file using runtime.ReadMemStats
Before change: 11GB
After change: 1.5GB

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README / documentation, if necessary

@sfc-gh-ext-simba-jl sfc-gh-ext-simba-jl force-pushed the SNOW-1029631-optimize-put-mem branch from 92386e2 to 9b01fd1 Compare December 5, 2024 08:37
@sfc-gh-ext-simba-jl sfc-gh-ext-simba-jl changed the title SNOW-1029631: Optimize PUT memory usage SNOW-1029631: Optimize streaming PUT memory usage Dec 5, 2024
@sfc-gh-ext-simba-jl sfc-gh-ext-simba-jl force-pushed the SNOW-1029631-optimize-put-mem branch from 9b01fd1 to 673e693 Compare December 5, 2024 18:58
@sfc-gh-ext-simba-jl sfc-gh-ext-simba-jl force-pushed the SNOW-1029631-optimize-put-mem branch from 1ecb657 to cd542f1 Compare December 11, 2024 14:44
Copy link

codecov bot commented Dec 11, 2024

Codecov Report

Attention: Patch coverage is 71.95122% with 23 lines in your changes missing coverage. Please review.

Project coverage is 82.13%. Comparing base (3364567) to head (cd542f1).

Files with missing lines Patch % Lines
file_transfer_agent.go 59.25% 11 Missing ⚠️
file_util.go 54.54% 10 Missing ⚠️
connection_util.go 93.54% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1266      +/-   ##
==========================================
- Coverage   82.19%   82.13%   -0.07%     
==========================================
  Files          55       55              
  Lines       13484    13539      +55     
==========================================
+ Hits        11083    11120      +37     
- Misses       2401     2419      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants