Migration aws sdk v2 to v3 : cleanupBuckets #348

SylvainSenechal · 2025-09-01T12:46:45Z

bert-e · 2025-09-01T12:46:49Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

CRR/ReplicationStatusUpdater.js

francoisferrand

same comments as https://github.com/scality/s3utils/pull/349/files#r2333101487, looks so very similar.... Maybe we should create a ticket to clean this up and deduplicate the redundancy (maybe some sort of common processBucket function, which calls it's parameter to "act" on matching object)

SylvainSenechal · 2025-09-22T14:58:00Z

cleanupBuckets.js

+                const bodyContent = Buffer.from(args.request.body);
+                const md5Hash = crypto.createHash('md5').update(bodyContent).digest('base64');
+                // eslint-disable-next-line no-param-reassign
+                args.request.headers['Content-MD5'] = md5Hash;


"name":"BadDigest" error without this stuff

is this not the "known issue" about content-md5, on which @leif-scality worked and which @benzekrimaha further worked when bumping SDK in cloudserver?

do we have a better/simpler/... fix there?

DeleteObjects, PutBucketCors and all the other /lib/api routes don't require an MD5 checksum since b10a2d6eeb20118af0a8975ce471b8d438214641.

Note: some backbeat routes may require an MD5 sum today.

Yep, doubled checked with Leif and Maha and tried on cloudserver local, no badDigest after I pulled the latest branch, I'll remove this

bert-e · 2025-09-22T15:11:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

williamlardier · 2025-09-22T12:50:29Z

CRR/ReplicationStatusUpdater.js

+                ))
+                    .then(res => next(null, res.ReplicationConfiguration))


Suggested change

))

.then(res => next(null, res.ReplicationConfiguration))

)).then(res => next(null, res.ReplicationConfiguration))

williamlardier · 2025-09-29T13:22:49Z

cleanupBuckets.js

+    try {
+        await Promise.all([
+            cleanupVersions(bucket),
+            // abortAllMultipartUploads(bucket),


can be fully removed, or please add a comment to explain why it's commented

My bad, it's needed, I commented for a test this afternoon and forgot to remove it..

williamlardier · 2025-09-29T13:24:41Z

cleanupBuckets.js

+const { http, https } = require('httpagent');
 const async = require('async');
-const AWS = require('aws-sdk');
+const crypto = require('crypto');


The module doesn't seem to be used, we can remove it

I really messed up this afternoon, removed something that became useless and forgot to cleanup some stuffs -_-

williamlardier · 2025-09-29T13:25:22Z

cleanupBuckets.js

-function _deleteVersions(bucket, objectsToDelete, cb) {
-    // multi object delete can delete max 1000 objects
+async function _deleteVersions(bucket, objectsToDelete) {
+    if (!objectsToDelete || objectsToDelete.length === 0) {


This is a bit redundant with the check already in cleanupVersions? I think we should be able to merge them

Issue : S3UTILS-203

SylvainSenechal · 2025-09-30T08:15:02Z

/approve

bert-e · 2025-09-30T08:15:12Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.16

The following branches have NOT changed:

development/1.13
development/1.14
development/1.15
development/1.4

Please check the status of the associated issue S3UTILS-203.

Goodbye sylvainsenechal.

The following options are set: approve

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from 5c10149 to 167833f Compare September 1, 2025 14:17

francoisferrand reviewed Sep 9, 2025

View reviewed changes

CRR/ReplicationStatusUpdater.js Outdated Show resolved Hide resolved

francoisferrand reviewed Sep 9, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from 167833f to b06747d Compare September 22, 2025 14:49

SylvainSenechal commented Sep 22, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from b06747d to e9dc1b2 Compare September 22, 2025 15:04

scality deleted a comment from bert-e Sep 22, 2025

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from e9dc1b2 to da3f21f Compare September 22, 2025 15:14

francoisferrand approved these changes Sep 29, 2025

View reviewed changes

francoisferrand mentioned this pull request Sep 29, 2025

Started migration for buckerVersionStats #351

Open

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from da3f21f to 0947a1b Compare September 29, 2025 13:05

williamlardier reviewed Sep 29, 2025

View reviewed changes

Migrated cleanupBuckets to sdk v3

eeb784a

Issue : S3UTILS-203

SylvainSenechal force-pushed the improvement/S3UTILS-203 branch from 0947a1b to eeb784a Compare September 29, 2025 16:12

williamlardier approved these changes Sep 30, 2025

View reviewed changes

bert-e merged commit eeb784a into development/1.16 Sep 30, 2025
10 checks passed

bert-e deleted the improvement/S3UTILS-203 branch September 30, 2025 08:15

	))
	.then(res => next(null, res.ReplicationConfiguration))
	)).then(res => next(null, res.ReplicationConfiguration))

Migration aws sdk v2 to v3 : cleanupBuckets #348

Migration aws sdk v2 to v3 : cleanupBuckets #348

Uh oh!

Conversation

SylvainSenechal commented Sep 1, 2025

Uh oh!

bert-e commented Sep 1, 2025

Hello sylvainsenechal,

Uh oh!

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Sep 22, 2025

Waiting for approval

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented Sep 30, 2025

Uh oh!

bert-e commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

francoisferrand Sep 29, 2025 •

edited

Loading