Conversation
The examples table carried `Non versioned` alone until 137e69d swapped it for `Versioned`, in the same commit that added the cross-site restore steps. Nothing records why: the commit message says only "Complete PRA logic", the pull request body names the tests it was missing, its review discussed the zenkoversion, the locations and the scripts, and the ticket has no description at all. Nor does the code suggest a reason -- the restore helper that commit extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state, which is where a restore writes `x-amz-restore` onto a master with nothing beneath it. Both rows rather than the swap back: the version delete this scenario covers needs a bucket with versions to delete. Issue: ZENKO-5355
Three paths the PRA scenario never touched: it deletes nothing, and its only bucket is versioned, so an object is never overwritten in place and never deleted without a version to remove. None of the deletions D/R has never replicated, nor the metadata a non-versioned overwrite has to carry, is visible to CI today. Removing a version and overwriting in place are steps of the nominal scenario, which already holds the state they need. Removing an unversioned object gets a scenario of its own: a failing step abandons the rest of its scenario, and this is the one whose outcome on a released deployment is not yet known. @pra is assigned at most one pickle at a time, so scenarios cannot fight over the two clusters. Delete markers stay uncovered, as ZENKO-4886 records: a marker carries no placement, so the location the source selects on cannot match it. A suspended bucket gets a scenario of its own too. A put there writes a null version, which the delete addresses as a version even though the master may be the only document holding it, so what removes the object is the teardown of the placeholder left on that master -- a write the source makes in no other case. Issue: ZENKO-5355
Not for merge. Exercises ZKOP-562, BB-811 and DRCTL-83 end to end: the @pra CTST run replicates cold objects through the mongo-processor rather than the Kafka Connect mongo sink. - backbeat 5e8a9c0b (BB-811, the D/R processor mode, with an object rewritten in place taken whole, and BB-853 on top: the sink omits the metrics block, and the producer is a hard startup dependency without it) - zenko-operator cf7e60c4 (ZKOP-562, the sink deployment and the source pipeline that shapes the entries it reads, rebased on the 1.9 that carries ZKOP-588 -- so drctl no longer needs DRCTL-81's merge patch to get past the analytics section marshalling as a null) - drctl v1.1.0-preview.1 (carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sink; its vendor tree carries an operator new enough that the analytics section marshals as absent rather than as a null the apiserver rejects) The retag prefix is 9.5.3 because that branch really is 9.5.2, and every operator gate on the backbeat tag sits either below 9.5.1 or at 9.6.0-preview.1, so it reads exactly as 9.5.1 did. drctl is pinned to a released tag rather than to the per-commit image its own CI pushes. backlogMetricsIntervalSeconds moves to configurationOverrides, which is what replaced it: ZKOP-569 added the field, and ZKOP-561 reverted it right after v1.8.16 in favour of the generic override, so the operator this pins rejects the field outright. The interval reaches the same config key through the env var backbeat derives from its schema, on the consumers whose lag the lifecycle conductor waits on. Without it the conductor idles 60-120s per batch, and a Days=0 transition on a versioned bucket takes three batches -- 319.6s measured, past the 300s the CTST transition steps allow.
Contributor
Author
|
@DarkIsDude Moved #2529 here (to get the right ticket ref in the branch) |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # yq eval 'sortKeys(.)' -i deps.yaml | ||
| backbeat: | ||
| sourceRegistry: ghcr.io/scality | ||
| sourceRegistry: ghcr.io/scality/playground/delthas |
There was a problem hiding this comment.
backbeat pulls from ghcr.io/scality/playground/delthas — a personal playground registry. The review criteria require Scality-internal images to resolve from ghcr.io/scality with a concrete released tag. The PR description says this will be replaced before merge; flagging so it isn't overlooked.
| dashboard: zenko-operator/zenko-operator-dashboards | ||
| image: zenko-operator | ||
| tag: v1.8.16 | ||
| tag: cf7e60c45ce51a598ef5c39ea3a58057c86627ac |
There was a problem hiding this comment.
zenko-operator is pinned to a bare commit SHA (cf7e60c...), not a released tag. The PR description notes that zenko-operator#631 is still open. This needs a released tag before merge.
Comment on lines
+28
to
+36
| configurationOverrides: | ||
| backbeat-lifecycle-producer: | ||
| KAFKA_BACKLOG_METRICS_INTERVALS: "5" | ||
| backbeat-lifecycle-bucket-processor: | ||
| KAFKA_BACKLOG_METRICS_INTERVALS: "5" | ||
| backbeat-lifecycle-object-processor: | ||
| KAFKA_BACKLOG_METRICS_INTERVALS: "5" | ||
| backbeat-replication-data-processor: | ||
| KAFKA_BACKLOG_METRICS_INTERVALS: "5" |
Contributor
There was a problem hiding this comment.
is it not the same? or do we have a bug?
Contributor
Author
There was a problem hiding this comment.
I added the previous value in https://github.com/scality/zenko-operator/pull/626 and it was removed as part of your PR in https://github.com/scality/zenko-operator/pull/633 moving to generic overrides. So I'm updating the CR. (But is this what you're asking?)
| image: vault2 | ||
| tag: 8.11.9 | ||
| envsubst: VAULT_TAG | ||
| zenko-operator: |
Contributor
There was a problem hiding this comment.
we can't bump ZKOP to 1.9 at this point, analytics is still in flux...
Contributor
Author
There was a problem hiding this comment.
Keeping this thread open until we decide what to do
DarkIsDude
removed their request for review
September 15, 2026 08:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PRA scenario deleted nothing, and its only bucket was versioned — so an object was never overwritten in place, and never deleted without a version to remove. This adds those paths and points the solution at the builds that make them pass.
Scenarios
PRA (nominal case)—VersionedPRA (nominal case)—Non versionedPRA (suspended null version deletion)PRA (unversioned deletion)The
Non versionedrow was in the table until137e69dd91swapped it forVersioned, in the same commit that added the cross-site restore steps. Nothing records why — not the commit message, not the pull request, not the ticket — and the restore helper it extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state. Both rows run again.The two deletions get scenarios of their own rather than more steps in the nominal one: a failing step abandons the rest of its scenario, and
@PRAis assigned at most one pickle at a time, so scenarios cannot fight over the two clusters.What each of those two covers. A suspended bucket's delete creates a null delete marker, and the marker itself is not replicated — it carries the bucket's location constraint, which can never be a cold one. What removes the object on the sink is the other write the delete makes: cloudserver removes the null version the marker replaces, and that removal carries the cold object's own metadata. An unversioned bucket holds no version to remove at all, so its delete takes away the object's only document — a different metadata event from the version removal the nominal scenario covers.
Deleting an unversioned object already passes on
development/2.16, but for the wrong reason — the delete flag lands on the master, the document a version-lessHeadObjectreads, so the object reads as absent while its document stays behind forever. That row is a regression guard, not a demonstration.What the bumps point at
v1.1.0-preview.1— carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sinkThe two backbeat PRs stay pinned as one build, since the sink needs both and #2833 has not merged. The bump commit is replaced by released versions before this merges, so expect it to be amended until then.
A known race, not fixed here
Bucket definitions reach the sink through the Kafka Connect entity sync; object entries reach it through the mongo-processor. Nothing orders the two. An object entry that arrives before its bucket gets
NoSuchBucket, and the offset is committed regardless — so that object's metadata is dropped for good, and the scenario fails on an assertion no amount of waiting can satisfy.We have not tried to fix it here. The entity path this races is expected to be retired.
Issue: ZENKO-5355