Add ability to evaluate versioned buckets #51
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR adds the ability for users to list and evaluate all versioned objects in the bucket. The existing behavior was using ListObjectsV2 API which only listed current/latest objects on a versioned bucket. As a result all the metrics did not account for older versions of objects which were present on the bucket and taking up space. For my use case this was unacceptable, as I need to know when I'm approaching the storage limit of my bucket.
In order to address this a few changes were introduced:
--s3.with-versions
which changes the behavior of the API calls and is purely opt in (default is false) which maintains backwards compatibility.ItemAggregator
struct which keeps track of statistics and define a separate parallel method on how to evaluate all objects in a bucket using ListObjectVersions API.Further things of note:
Tested on my account end to end across 3 different buckets (versioned, non-versioned, empty non-versioned).
Happy to consider any changes you want me to make or any other suggestions you might have.