Add time-based cleanup for Maven snapshot versions #33420
Conversation
GiteaBot
added
the
lgtm/need 2
pull-request-size
bot
added
the
size/L
github-actions
bot
added
modifies/go
Considered Alternative ApproachI considered another way to handle Maven snapshot cleanup that uses the This method would need a database migration to work, so it's a bit more complicated. I have the code ready for this alternative if we think it's worth the extra steps later on. |
delvh
changed the title
|
Note to any reviewer: Review this PR carefully. |
|
A remark: Yes, it's true that I used ChatGPT for help since I'm not an experienced Go developer, but I handled the core logic and testing myself. I'd appreciate any comments or critiques to help us improve this, as we really need the feature :)
|
modules/packages/maven/metadata.go
Outdated
| @@ -109,3 +131,20 @@ func ParsePackageMetaData(r io.Reader) (*Metadata, error) { | |||
| Dependencies: dependencies, | |||
| }, nil | |||
| } | |||
|
|
|||
| // ParseMavenMetadata parses the Maven metadata XML to extract the build number. | |||
| func ParseMavenMetaData(r io.Reader) (string, error) { | |||
There was a problem hiding this comment.
I am thoroughly confused by this method signature and docs: Nothing indicates to me that this method parses the entire XML, and throws the result except for the build number away.
There was a problem hiding this comment.
I've changed the signature and added a new snapshotMetadata type which can be extended later, does it make it more readable?
models/packages/package_file.go
Outdated
| } | ||
| } | ||
|
|
||
| log.Info("Filtered %d files out of %d total files for version ID %d with maxBuildNumber %d", len(filteredFiles), len(files), versionID, maxBuildNumber) |
There was a problem hiding this comment.
Hmm… I do not like seeing log statements inside models/.
That should be the job of the services/ layer.
Thus, what about adding an extra return value (skippedFiles) that contains all skipped files?
There was a problem hiding this comment.
yes, thanks, will do!
models/packages/package_file.go
Outdated
| } | ||
| } | ||
|
|
||
| log.Info("Filtered %d files out of %d total files for version ID %d with maxBuildNumber %d", len(filteredFiles), len(files), versionID, maxBuildNumber) |
There was a problem hiding this comment.
Info logs are rare within Gitea.
At most, gather statistics inside a success message after successfully running the cron job, or downgrade it to debug.
services/packages/maven/cleanup.go
Outdated
|
|
||
| thresholdBuildNumber := maxBuildNumber - retainBuilds | ||
| if thresholdBuildNumber <= 0 { | ||
| log.Info("No files to clean up, as the threshold build number is less than or equal to zero for versionID %d", versionID) |
There was a problem hiding this comment.
Regarding the error messages and logs in general, I recommend explicitly stating during what sort of action they occurred.
When you actually see these logs, it will be hard to guess that they belong to the Maven snapshot cleanup otherwise, especially when a request happens in the meantime.
services/packages/maven/cleanup.go
Outdated
| return nil | ||
| } | ||
|
|
||
| func extractMaxBuildNumberFromMetadata(ctx context.Context, metadataFile *packages.PackageFile) (int, error) { |
There was a problem hiding this comment.
Didn't you define this method in models already?
GiteaBot
added
lgtm/blocked
…ich will list files for deletion
Pull Request: Implement Cleanup Function for Maven Snapshot Versions
Overview
This pull request introduces a cleanup function for Maven snapshot versions in Gitea, enabling more efficient management of package storage. The new feature allows users to specify how many of the most recent Maven snapshot builds to retain, optimizing storage by automatically removing older files.
Features
RETAIN_MAVEN_SNAPSHOT_BUILDSallows setting the number of Maven snapshot builds to keep.-1, which keeps all builds.Implementation
The feature extends the existing cleanup job to include files from Maven snapshot versions based on the specified retention policy set in
app.ini. It checks against the highest build number from the Maven metadata file to determine which files to retain.Impact
This enhancement helps manage disk space more effectively by providing control over how many builds of Maven snapshots are retained, potentially reducing storage requirements for projects using Maven within Gitea.