Skip to content
This repository has been archived by the owner on Nov 4, 2022. It is now read-only.

multiple threads writing to same disk causes issues with disk cleanup #214

Open
emillynge opened this issue Nov 1, 2019 · 10 comments
Open

Comments

@emillynge
Copy link
Contributor

Hi

I realize that having stenographer use multiple threads witing to different directories on the same underlying disk may not be a supported use case.

But I really do have a viable use for this.

In such a setup, the automatic disk cleanup does not work as intended since cleanup is done in serial per thread. The first thread to do cleanup will then take on the burden of cleaning files until the desired percentage of free space is achieved. Only if the first thread has no more files left to delete will other threads using the same disk begin deleting their files.

I think there are 2 viable solutions:

  1. quick and dirty - randomize the order in which threads calls SyncFiles(). This will work fairly well as long as the threads write equal amounts of data.

  2. more complex - move deletion to Env and delete files alternatingly.

I already have a fork with solution 1) that I could turn into a PR if you would like it.

@gconnell
Copy link
Contributor

gconnell commented Nov 1, 2019

You're right that this is not an expected use-case, but you're also right that it's one we should probably support, since there's nothing saying you CAN'T do it. Given that, if you've got a PR you'd like to send out I'm happy to take a look.

@emillynge
Copy link
Contributor Author

I can very quickly have a PR on solution 1) :)

Speaking of PRs, would you mind taking a look at #213 ? It has been open for about a month now and fixes a bug I found.

@gconnell
Copy link
Contributor

gconnell commented Nov 4, 2019

Done, thanks, and sorry for the late reply!

@rskjetlein
Copy link

Was this fixed?

@rskjetlein
Copy link

I made a working solution for 1. mentioned. Can assemble a PR for it if any interest

@jtgreen-cse
Copy link

jtgreen-cse commented Mar 3, 2022

I'd be interested in seeing this merged. I've come across this issue and have this use case.

Edit: @rskjetlein I found your fork, here's the commit that fixes it -->rskjetlein@6f6d2cb

@rskjetlein
Copy link

Will have a look. Its been dormant for some time...

@rskjetlein
Copy link

I'd be interested in seeing this merged. I've come across this issue and have this use case.

Edit: @rskjetlein I found your fork, here's the commit that fixes it -->rskjetlein@6f6d2cb

For the most part its working fine, during very high traffic rate (+10G) a unbalanced deletion can be observed.

@jtgreen-cse
Copy link

I've been testing the random thread picking strategy with 16 threads .. that makes it too random. Threads delete files very unevenly. Like suggested I think moving the logic in env and deciding which thread to pick for delete based on which has the most files may solve this.

@rskjetlein
Copy link

I've been testing the random thread picking strategy with 16 threads .. that makes it too random. Threads delete files very unevenly. Like suggested I think moving the logic in env and deciding which thread to pick for delete based on which has the most files may solve this.

I extended the API with a object delete function and had an external process that used the same API to control the disk fill percentage. The external process took into account other things that used the same disk, might be a usable approach for you.

Please reach out if you want to code snippets. [email protected]

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants