Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up unused checkouts on dev-desktops #389

Merged
merged 3 commits into from
Mar 12, 2024

Conversation

jdno
Copy link
Member

@jdno jdno commented Feb 16, 2024

User might checkout repositories on the dev-desktops, work on them for a while, and then move on to other projects. The unused checkouts, in particular their build or target directories, can use up a lot of space that we want to reclaim periodically. A script has been created that finds all cache directories, checks recursively if any file in their root directory has been modified within a given time frame, and if not deletes the cache directory.

User might checkout repositories on the dev-desktops, work on them for a
while, and then move on to other projects. The unused checkouts, in
particular their `build` or `target` directories, can use up a lot of
space that we want to reclaim periodically. A script has been created
that finds all cache directories, checks recursively if any file in
their root directory has been modified within a given time frame, and if
not deletes the cache directory.
A new cronjob has been created that looks for unused build artifacts and
deletes them. The job runs every day at midnight.
@jdno jdno marked this pull request as ready for review February 26, 2024 13:05
# recursively look for directories that either have a file named `x.py` and a
# directory named `build`, or a file named `Cargo.toml` and a directory named
# `target`.
all_cache_directories=$(find /home -type d \( -name build -execdir test -f "x.py" \; -o -name target -execdir test -f "Cargo.toml" \; \) -print | sort | uniq)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may also want to sweep build-rust-analyzer too, which is a secondary build dir recommended for r-a workflows: https://rustc-dev-guide.rust-lang.org/building/suggested.html#configuring-rust-analyzer-for-rustc

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make a note of this in #390, which tracks rewriting the script in Rust. I fear adding more conditions to the find command will get unmanageable quickly. 🙈

@spastorino
Copy link
Member

Great @jdno. I was wondering if on top of removing old builds, we shouldn't have certain amount of free space always ensured just in case some of us are creating new build and may run out of space.
Maybe the script can check for certain amount of free disk space threshold and if we don't have enough consider removing from the oldest to the newest until we can free up to that threshold.

@jdno
Copy link
Member Author

jdno commented Mar 11, 2024

Great @jdno. I was wondering if on top of removing old builds, we shouldn't have certain amount of free space always ensured just in case some of us are creating new build and may run out of space.

There are two separate issues that we want to handle differently, I think. First, we don't want to run out of disk space on the root partition / to avoid locking up the machine. We could do this by extending this script, for example, but I think ultimately we want to move the / and /home partitions to different disks. Second, we want people to always be able to create a new checkout (within their quota). Cleaning up "unused" projects more aggressively might help. 🤔

@jdno jdno merged commit 2232706 into rust-lang:master Mar 12, 2024
3 checks passed
@jdno jdno deleted the gc-dev-desktops branch March 12, 2024 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants