Add a D/R mode to the mongo-processor by delthas · Pull Request #2833 · scality/backbeat

delthas · 2026-08-27T12:41:18Z

The mongo-processor was written as the out-of-band ingestion consumer, and the D/R metadata sink now reuses it. The two disagree about most of what it does to an object: ingestion rewrites identity and placement to local values, because the source system's accounts and locations do not exist here, while D/R replicates both and so applies what the source-side pipeline sends.

Put those decisions behind a mode, following the shape the notification extension uses for its destinations — an abstract ProcessorMode, an implementation per mode, and an index mapping the configured name to the class. IngestionMode carries today's behaviour verbatim and mode defaults to ingestion, so nothing changes for an existing deployment.

What the mongo-processor does

flowchart LR
    K[("Kafka topic")] --> E["read entry"]
    E --> B["look up the bucket"]
    B --> Z["read the stored object metadata"]
    Z --> A["apply the entry"]
    A --> M[("sink MongoDB")]

Where the two modes diverge

Every decision the two modes disagree about is a hook on ProcessorMode, so the surrounding flow stays single-sourced.

flowchart TD
    E["entry"] --> T{"type"}

    T -->|del| G0["shouldProcessDelete"]
    G0 -->|ingestion| I0["only while the stored placement<br/>still matches the bucket"]
    G0 -->|D/R| D0["always"]
    I0 --> DEL[("delete in MongoDB")]
    D0 --> DEL

    T -->|put| G1["needsExistingMetadata"]
    G1 -->|ingestion| I1["only with a scal header or<br/>an enabled replication rule"]
    G1 -->|D/R| D1["always"]

    I1 --> G2["resolveVersionId"]
    D1 --> G2
    G2 -->|ingestion| I2["the scal version id when present"]
    G2 -->|D/R| D2["the entry's own"]

    I2 --> G3["getChangedContent"]
    D2 --> G3
    G3 -->|ingestion| I3["tags"]
    G3 -->|D/R| D3["tags, object lock, and placement<br/>while the data is still remote"]

    I3 --> S{"stored document"}
    D3 --> S

    S -->|absent| G4["applyNewObjectMetadata"]
    S -->|present| G6["replacesExistingMetadata"]
    G6 -->|ingestion| G5["mergeExistingMetadata"]
    G6 -->|"D/R, versioned"| G5
    G6 -->|"D/R, no version id<br/>or a null version"| G4

    G4 -->|ingestion| I4["local owner, location and data part,<br/>ACLs reset"]
    G4 -->|D/R| D4["ACLs reset"]
    G5 -->|ingestion| I5["keep the stored document,<br/>take the tags"]
    G5 -->|D/R| D5["keep the stored document, take tags<br/>and object lock, placement by locality"]

    I4 --> W[("write to MongoDB")]
    D4 --> W
    I5 --> W
    D5 --> W

The hooks

Hook	Ingestion	D/R	Beyond the ticket
`needsExistingMetadata`	Reads the stored document only when the entry carries a scal header or the bucket has an enabled replication rule	Always reads it	yes
`resolveVersionId`	The `x-amz-meta-scal-version-id` when present, otherwise the entry's	The entry's own, always †	yes
`getChangedContent`	Diffs tags only	Diffs tags, object-lock state, and placement while the data is still remote; compares the whole document when the entry replaces it, so a replay is still skipped	yes
`applyNewObjectMetadata`	Rewrites owner, `dataStoreName` and the data part to local values, resets ACLs	Resets ACLs, nothing else	no
`replacesExistingMetadata`	Never: a stored document is always merged into	True for a write carrying no version id, or a null version — the same key is being rewritten, so the entry replaces the document rather than merging into it	no
`mergeExistingMetadata`	Keeps the stored document, takes the entry's tags	Keeps the stored document, takes tags and object-lock state; keeps the stored placement for a localized version, takes the entry's for one still on the remote site	no
`shouldProcessDelete`	Skips the delete unless the stored placement still matches the bucket	Applies it always	yes

The three marked beyond the ticket are not in the ticket text but follow from the design:

The stored document is always read. Without it the merge is unreachable — the fetch is skipped when there is no scal header and no enabled replication rule, which is the normal D/R case, so every entry took the create path and an update overwrote the placement. The design requires the merge and asserts replay idempotence but never mentions the read; making it unconditional is our choice, at one getObject per entry.
Change detection covers the mutable set. getContentType only diffs tags, so a retention or legal-hold change with unchanged tags was dropped as a duplicate before the merge could run. The requirement is the matrix's "Merged on update"; the mechanism is ours.
The delete guard is skipped. Its five-way location test ignores every D/R deletion, since a replicated object's location legitimately differs — cold, on the source, or localized. The design endorses applying deletions directly: "the mongo-processor applies replicated operations -including version deletions- at the metadata level, below the S3 API enforcement".

† The one decision the design does not cover. x-amz-meta-scal-version-id is ignored in D/R mode. Both call sites used it mode-independently to choose which version to read and to delete. A production object that was itself OOB-ingested or cold-restored carries that header, naming a version of the system production ingested from — so honouring it here reads and deletes the wrong version. Version ids are the source's own and identical on both sides, so the entry's own id is authoritative. The design never considers a production site that itself ingests, so this is a guess, and the easiest thing in the PR to reverse.

What happens to each field

Field	Ingestion	D/R
`owner-id`, `owner-display-name`	Replaced with the bucket owner	Kept as sent
`dataStoreName`	Replaced with the Zenko location	Kept as sent on a first write; on an update kept from the stored document for a localized version, taken from the entry for one still remote
`location` data parts	Rewritten to a single part pointing at the Zenko location	Kept as sent
`acl`	Reset to the defaults	Reset to the defaults
`tags`	Taken from the entry	Taken from the entry
`retentionMode`, `retentionDate`, `legalHold`	Not merged: only tags are	Taken from the entry, cleared values included
`versionId`	The scal version id when present	The entry's own
`x-amz-meta-scal-version-id`	Honoured	Ignored
`replicationInfo`	Reset, shared between the modes	Reset, shared between the modes
`content-length`, `content-md5`, and the rest	As sent	As sent

Configuration

The mongo-processor now reads two settings from its own extension config instead of from blocks it does not own: the mongo client config, so a D/R sink no longer carries a queuePopulator block it never runs, and the service account, so it no longer carries a partial extensions.ingestion block. The second was not optional — backbeat validates every extension listed under extensions against that extension's own schema, and the ingestion schema also requires a topic, a zookeeper path and a source list, so naming the account there made the whole configuration invalid. Both fall back to the old location for a deployment that enables both extensions, and auth is required in D/R mode, which enables no other extension.

Also stops extending the health-check address whitelist when no server section is configured. The section is optional, but it was dereferenced unconditionally, so a process serving no API exited before starting.

Follow-ups and exclusions

Topic	Ticket	Notes
GC `deleteData` when a localized version is deleted	BB-813 / BB-815	Worth being accurate: the design does not defer this. It states it as part of deletion handling and it appears in none of its lists of later increments
Circuit-breaker rules	BB-822	The plumbing already exists; the design names the gating signal as local oplog and copy-queue lag
`__metastore` and Vault entities	OS-1110
`transitionInProgress` suppression	BB-819	Depends on this merge. Keeping the stored document and overlaying a short list is what leaves it open; the non-localized branch restores placement specifically rather than wholesale, so BB-819 adds to a list that already exists
The operator's `locationConfig.json` has to stop being `{}`	ZKOP-566	`isCRRLocation` reads it. Harmless for metadata-only D/R, where every lookup is false and every object is cold, which is the correct answer there
Duplicated `lib/util/locations.js`	#2828	The first commit copies it byte-for-byte rather than stacking on a draft. Whichever of the two lands second has that commit dropped as already applied — verified by rebasing onto #2828 and watching git drop it
`replicationInfo`	none	Stays shared and unchanged: the source pipeline resets it and bucket workflows are stripped, so a correct sink bucket has no replication config and this reduces to a reset
Metadata RPO metric	none found	The observability section asks for the timestamp of the last metadata write replicated, and alerts on it. `MongoProcessorMetrics` has no such gauge. Every metric here is also `ingestion*`-prefixed, which will read oddly on a D/R dashboard

Unblocks ZKOP-562, whose generated configuration sets mode, mongodb and auth, and which cannot start a mongo-processor without this.

Issue: BB-811

bert-e · 2026-08-27T12:41:21Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request.
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

codecov · 2026-08-27T12:41:58Z

Codecov Report

❌ Patch coverage is 98.09524% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.37%. Comparing base (7242d8a) to head (e1dd485).

Files with missing lines	Patch %	Lines
extensions/mongoProcessor/mongoProcessorTask.js	0.00%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...ns/mongoProcessor/MongoProcessorConfigValidator.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/MongoQueueProcessor.js	`72.68% <100.00%> (-2.11%)`	⬇️
extensions/mongoProcessor/modes/DRMode.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/modes/IngestionMode.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/modes/ProcessorMode.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/modes/index.js	`100.00% <100.00%> (ø)`
lib/Config.js	`81.96% <100.00%> (+0.14%)`	⬆️
lib/util/locations.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/mongoProcessorTask.js	`0.00% <0.00%> (ø)`

... and 3 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`80.25% <ø> (ø)`
Core Library	`82.41% <100.00%> (-0.58%)`	⬇️
Ingestion	`75.91% <97.97%> (+1.41%)`	⬆️
Lifecycle	`80.68% <ø> (ø)`
Oplog Populator	`85.80% <ø> (ø)`
Replication	`62.24% <ø> (ø)`
Bucket Scanner	`85.76% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/9.6    #2833      +/-   ##
===================================================
- Coverage            76.50%   76.37%   -0.13%     
===================================================
  Files                  204      209       +5     
  Lines                14120    14191      +71     
===================================================
+ Hits                 10802    10839      +37     
- Misses                3308     3342      +34     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.53% <23.80%> (+0.11%)`	⬆️
api:routes	`9.29% <23.80%> (+0.11%)`	⬆️
bucket-scanner	`85.76% <ø> (ø)`
ft_test:queuepopulator	`9.76% <23.80%> (-1.66%)`	⬇️
ingestion	`12.93% <91.42%> (+0.39%)`	⬆️
lib	`9.29% <23.80%> (+0.11%)`	⬆️
lifecycle	`19.60% <23.80%> (+0.05%)`	⬆️
notification	`1.00% <0.00%> (-0.01%)`	⬇️
oplogPopulator	`0.13% <0.00%> (-0.01%)`	⬇️
replication	`19.25% <23.80%> (+0.05%)`	⬆️
unit	`56.87% <33.33%> (-0.09%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-08-27T12:42:02Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

bert-e · 2026-08-27T13:37:56Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Data on an `isCRR` location belongs to a remote site: it may be read, but never deleted, and a version whose data still lives there has not been localized yet. Several places need to ask that question. Lifted verbatim from BB-813 (#2828), which introduces the same helper and carries its unit test and the location fixture. Kept byte-identical so that whichever branch lands second has this commit dropped as already applied, rather than conflicting. Issue: BB-811

mongoProcessorTask takes the mongo client config from config.queuePopulator.mongo, which works when a queue populator is deployed beside the processor. The D/R metadata sink deploys none, so its configuration would have to carry a queuePopulator block it never runs. Accept extensions.mongoProcessor.mongodb, validated with the shared mongoJoi, and fall back to the queue populator's config when absent. The fallback is guarded: a config omitting queuePopulator entirely would otherwise throw before validation could report anything useful. Issue: BB-811

The `server` section is optional, but the whitelist of addresses allowed to reach the health checks was extended unconditionally, so a process serving no API exited before starting: a D/R sink runs the mongo-processor alone, and configuring a section it never reads to get past this is no answer. Issue: BB-811

The mongo-processor was written as the out-of-band ingestion consumer, and the D/R metadata sink now reuses it. The two disagree about most of what it does to an object, so put those decisions behind a mode: an abstract ProcessorMode whose methods assert, an implementation per mode, and an index mapping the configured name to the class, as the notification extension does for its destinations. IngestionMode carries today's behaviour verbatim, so this commit changes nothing. The default lives beside the mode map, so a processor built programmatically gets the same mode as one built from a config file. Issue: BB-811

The D/R metadata sink replicates production's objects, accounts included, so it applies what the source-side pipeline sends rather than rewriting it into something local: a new object is written as it arrives, and an update takes the entry's tags, object-lock state and ACLs while keeping the placement already stored. Keeping the stored placement is the point. The copy engine rewrites location and dataStoreName to a local location after the first write, and applying the entry's would send reads back to the source and leak a local copy that is never garbage-collected. Cleared values are applied like any other: removing a legal hold is an update. Three things follow from that and were unreachable before it: - the stored document is always read, because it is what distinguishes a first write from an update, and the entry cannot -- an insert is redelivered on replay and overlaps the bootstrap dump, so it is no promise that the object is absent here; - object-lock and ACL changes count as changes, where the ingestion diff looks only at tags and dropped them as duplicates; - a delete always applies, where the ingestion guard skips one whose object has moved location, which for a replicated object it always has. Issue: BB-811

Backbeat validates every extension listed under `extensions` against that extension's own schema, so naming the service account in a partial `extensions.ingestion` block makes the whole configuration invalid: the ingestion schema also requires a topic, a zookeeper path and a source list. A D/R sink runs no extension but this one, so it now carries the account in its own configuration, falling back to the ingestion extension for the deployments that enable both. Issue: BB-811

Change detection compared the metadata an update is allowed to carry, which left out the content itself: an object overwritten in place kept its version id, so the entry diffed to nothing and was dropped as a duplicate, and the sink went on describing the object that had been replaced. An object with no version of its own is rewritten in place, and so is the master a versioning suspended bucket marks null. Such an entry does not update the stored document, it replaces it: content, headers and user metadata all move, and the merge kept none of them. The Kafka Connect sink this replaces wrote the whole document -- the source pipeline turned a whole value write into a replace -- so the merge was a regression against it. Decide in the processor whether an entry replaces the stored object or updates it, and apply a replacing one as a first write. A version is immutable, so an entry for one still merges: clean room replicates versioned buckets only and keeps the semantics it had. Nothing of the stored document survives a replacement. Placement is the one field this site could own, and it cannot here: a cold object holds what the source pipeline derives from its storage class, and a restore this site performed describes bytes that are gone -- its expiration is sorbet's own to reconcile. Duplicate detection compares the document a write would leave with the one stored, so a replay is still a no-op while a rewrite that moved only a header is not. The content digest alone cannot tell the two apart: a PUT replaces the whole metadata whatever the bytes do. Issue: BB-811

delthas · 2026-09-11T15:07:18Z

Requested @maeldonn in place of Sylvain Senechal, who is currently on PTO.

delthas · 2026-09-14T00:32:29Z

Requested @SylvainSenechal in place of Mael Donnart, who is currently on PTO.

scality deleted a comment from bert-e Aug 27, 2026

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch 2 times, most recently from 1a74a03 to 8675518 Compare August 27, 2026 13:37

claude Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/mongoProcessorTask.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from e77010b to 5112a3a Compare August 31, 2026 08:31

scality deleted a comment from bert-e Aug 31, 2026

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 5112a3a to 0eb82b1 Compare August 31, 2026 09:24

claude Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/MongoProcessorConfigValidator.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch 2 times, most recently from 2956946 to d161e55 Compare September 4, 2026 15:18

claude Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/modes/IngestionMode.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from d161e55 to 486c8e2 Compare September 7, 2026 15:44

delthas mentioned this pull request Sep 7, 2026

Cover D/R deletion and in-place overwrite in the PRA test scality/Zenko#2529

Closed

delthas marked this pull request as ready for review September 7, 2026 15:55

delthas added 3 commits September 10, 2026 14:38

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 486c8e2 to 8b693e6 Compare September 10, 2026 12:52

delthas added 3 commits September 10, 2026 16:18

delthas requested review from a team, SylvainSenechal and francoisferrand September 11, 2026 07:50

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 8b693e6 to e1dd485 Compare September 11, 2026 09:19

delthas requested review from maeldonn and removed request for SylvainSenechal September 11, 2026 15:07

delthas requested review from SylvainSenechal and removed request for maeldonn September 14, 2026 00:32

delthas mentioned this pull request Sep 14, 2026

Extend PRA coverage to deletions, overwrites and non-versioned buckets scality/Zenko#2532

Open

Navigation Menu

Uh oh!

Add a D/R mode to the mongo-processor - #2833

Add a D/R mode to the mongo-processor#2833
delthas wants to merge 7 commits into
development/9.6from
improvement/BB-811/dr-mode-mongo-processor

delthas commented Aug 27, 2026 •

edited

Loading

Uh oh!

bert-e commented Aug 27, 2026

Uh oh!

codecov Bot commented Aug 27, 2026 •

edited

Loading

Uh oh!

bert-e commented Aug 27, 2026

Uh oh!

bert-e commented Aug 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

delthas commented Sep 11, 2026

Uh oh!

delthas commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

delthas commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What the mongo-processor does

Where the two modes diverge

The hooks

What happens to each field

Configuration

Follow-ups and exclusions

Uh oh!

bert-e commented Aug 27, 2026

Hello delthas,

Uh oh!

codecov Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bert-e commented Aug 27, 2026

Waiting for approval

Uh oh!

bert-e commented Aug 27, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

delthas commented Sep 11, 2026

Uh oh!

delthas commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

delthas commented Aug 27, 2026 •

edited

Loading

codecov Bot commented Aug 27, 2026 •

edited

Loading