Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize FileBlob deletion in cleanup #68556

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

caseyduquettesc
Copy link

Fixes #67545

cleanup_unused_files can take over an hour to run regardless of the concurrency setting. This creates new processes for deleting unused files in parallel. In my own install, I've observed this process now taking 35 minutes instead of 60 with concurrency set to 4, making this more scalable for us.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

@caseyduquettesc caseyduquettesc marked this pull request as ready for review April 9, 2024 19:27
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Apr 9, 2024
Copy link

codecov bot commented Apr 9, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.32%. Comparing base (6bdbae6) to head (1f64dca).
Report is 1060 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #68556      +/-   ##
==========================================
+ Coverage   71.87%   79.32%   +7.45%     
==========================================
  Files        6402     6395       -7     
  Lines      283333   283455     +122     
  Branches    48812    48824      +12     
==========================================
+ Hits       203653   224860   +21207     
+ Misses      79319    58239   -21080     
+ Partials      361      356       -5     

see 1144 files with indirect coverage changes

@caseyduquettesc caseyduquettesc force-pushed the parallelize-fileblob-cleanup branch 3 times, most recently from 728dd0d to 594fe90 Compare April 9, 2024 20:58
@caseyduquettesc caseyduquettesc force-pushed the parallelize-fileblob-cleanup branch from 594fe90 to 1f64dca Compare April 9, 2024 21:01
@getsantry
Copy link
Contributor

getsantry bot commented May 1, 2024

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry bot added the Stale label May 1, 2024
@caseyduquettesc
Copy link
Author

bump

@getsantry getsantry bot closed this May 9, 2024
@github-actions github-actions bot locked and limited conversation to collaborators May 24, 2024
@JoshFerge JoshFerge reopened this Dec 9, 2024
@JoshFerge JoshFerge self-assigned this Dec 9, 2024
@getsantry getsantry bot removed the Stale label Dec 10, 2024
@getsantry getsantry bot added Stale and removed Stale labels Jan 1, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize FileBlob cleanup
2 participants