copy location task reads the source data from a remote site - #2843
Merged
Merged
Conversation
maeldonn
approved these changes
Sep 10, 2026
delthas
approved these changes
Sep 11, 2026
francoisferrand
force-pushed
the
improvement/BB-813
branch
from
September 11, 2026 14:14
777d6f9 to
f9f4df4
Compare
francoisferrand
force-pushed
the
improvement/BB-812
branch
from
September 11, 2026 14:14
054fae4 to
ded3b69
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 5 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.6 #2843 +/- ##
===================================================
+ Coverage 76.70% 76.82% +0.12%
===================================================
Files 205 205
Lines 14225 14269 +44
===================================================
+ Hits 10911 10962 +51
+ Misses 3304 3297 -7
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Contributor
Hello francoisferrand,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
A DR mirrors buckets from a production site it has no data path to: the objects' metadata is replicated, but their data still lives on production, in a CRR location Cloudserver has no data client for. So the copy cannot go through the usual multiple-backend route, and reads the data over S3 on the production site itself. The location describes how to get there -- the servers to reach it, and the STS to assume roles on -- and the object's location part says where the data landed there and which role to assume to read it back, one role per production account owning objects. Clients come from the replication ClientManager, which already does this for replication: it refreshes the assumed-role credentials, forgets the accounts that went quiet, and supports an STS key read from a file. A production site is Scality too, so both reads are the same GetObject and only differ in where they point and whether a location constraint means anything; they carry the same request uids, so a copy can be followed across the two sites. Failing to read the data is reported as an error rather than skipped, so the lifecycle side gets to decide what it means for the transition and the object is not left flagged in transition for good. A location that is not reachable, or that holds more than the single part it is meant to, fails the same way instead of quietly reading the wrong bytes. Issue: BB-812 Co-authored-by: sylvain senechal <senechalsylvain.dev@gmail.com>
francoisferrand
force-pushed
the
improvement/BB-812
branch
from
September 11, 2026 15:02
ded3b69 to
322f768
Compare
Contributor
Author
|
/approve |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -307,6 +445,7 @@ class CopyLocationTask extends BackbeatTask { | |||
| method: 'CopyLocationTask._getAndPutObjectOnce', | |||
| peer: this.sourceConfig.s3, | |||
There was a problem hiding this comment.
peer: this.sourceConfig.s3 identifies the local Cloudserver, but _sendGetObject can now route through a remote site's client. When the remote client.send() fails (network error, auth failure at the S3 level), these catch handlers log the local Cloudserver as the peer, pointing the operator at the wrong server. Same applies to the stream error handler above (lines 397, 407).
The new methods already log the correct endpoint when they detect an error, but transport-level errors from the remote send() would appear under the wrong peer.
Contributor
|
I have successfully merged the changeset of this pull request
The following branches have NOT changed:
This pull request did not target the following hotfix branch(es) so they
Please check the status of the associated issue BB-812. Goodbye francoisferrand. The following options are set: approve |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DR mirrors buckets from a production site it has no data path to: the
objects' metadata is replicated, but their data still lives on
production, in a CRR location Cloudserver has no data client for. So the
copy cannot go through the usual multiple-backend route, and reads the
data over S3 on the production site itself.
The location describes how to get there -- the servers to reach it, and
the STS to assume roles on -- and the object's location part says where
the data landed there and which role to assume to read it back, one role
per production account owning objects. Clients come from the replication
ClientManager, which already does this for replication: it refreshes the
assumed-role credentials, forgets the accounts that went quiet, and
supports an STS key read from a file.
A production site is Scality too, so both reads are the same GetObject
and only differ in where they point and whether a location constraint
means anything; they carry the same request uids, so a copy can be
followed across the two sites.
Failing to read the data is reported as an error rather than skipped, so
the lifecycle side gets to decide what it means for the transition and
the object is not left flagged in transition for good. A location that
is not reachable, or that holds more than the single part it is meant
to, fails the same way instead of quietly reading the wrong bytes.
Issue: BB-812
Co-authored-by: sylvain senechal senechalsylvain.dev@gmail.com