Databus Throttling Changes #847

nabajyotiDash-hub · 2024-11-04T15:00:23Z

Github Issue

What Are We Doing Here?

Here is where you should describe the problem you are solving as it relates to the Github Issue referred above, adding any fine
details on the solution that might otherwise not be recognizable for someone
unfamiliar with the changes. Add some pictures if it helps.

How to Test and Verify

Check out this PR
Run Command X, Click Button Y
Profit

Risk

Level

Low, Medium, or High. Give an indication of what you think is the level of change introduced by this PR. High means a massive change to a core functionality.
Low means a really minor change that shouldn't have any regression effect.

Required Testing

Smoke, Regression, or Manual. (All changes except documentation need smoke
testing at a minimum).

Risk Summary

Add one or a few complete sentences about the possible risks or concerns for
this change.

Code Review Checklist

Tests are included. If not, make sure you leave us a line or two for the reason.
Pulled down the PR and performed verification of at least being able to
build and run.
Well documented, including updates to any necessary markdown files. When
we inevitably come back to this code it will only take hours to figure out, not
days.
Consistent/Clear/Thoughtful? We are better with this code. We also aren't
a victim of rampaging consistency, and should be using this course of action.
We don't have coding standards out yet for this project, so please make sure to address any feedback regarding STYLE so the codebase remains consistent.
PR has a valid summary, and a good description.

1. /UpdateParameterStore 2. /QueueExecutionAttributes/<queue_type>/<queue_name>

sor/pom.xml

sor/src/main/java/com/bazaarvoice/emodb/sor/core/DefaultDataStore.java

* PD-257044-changes-for-databus * duplicated kafka and ssm util packages in emo sor module * removed redundant dependencies

* PD-257044-changes-for-databus * duplicated kafka and ssm util packages in emo sor module * removed redundant dependencies * additional changes * additional changes * additional changes * removed unneccessary dependencies

This reverts commit 62dc31e.

quality/integration/src/test/java/test/integration/auth/TableRoleManagerDAOTest.java

queue-api/src/main/java/com/bazaarvoice/emodb/queue/api/DedupQueueService.java

queue-api/src/main/java/com/bazaarvoice/emodb/queue/api/QueueService.java

queue/src/main/java/com/bazaarvoice/emodb/queue/core/stepfn/StepFunctionService.java

queue/src/test/java/com/bazaarvoice/emodb/queue/core/stepfn/StepFunctionServiceTest.java

sor-api/src/main/java/com/bazaarvoice/emodb/sor/api/Names.java

web/src/main/java/com/bazaarvoice/emodb/web/resources/queue/QueueResource1.java

web/src/main/java/com/bazaarvoice/emodb/web/resources/sor/DataStoreResource1.java

sor/src/main/java/com/bazaarvoice/emodb/sor/core/DefaultDataStore.java

bharadwaj-k

LGTM

IgorKruiper

one potential blocking issue

IgorKruiper · 2024-11-06T11:00:51Z

databus/src/main/java/com/bazaarvoice/emodb/databus/core/OwnerAwareDatabus.java

@@ -41,6 +41,9 @@ Subscription getSubscription(String ownerId, String subscription)
    long getEventCount(String ownerId, String subscription)
        throws UnauthorizedSubscriptionException;

+    public long getMasterEventCountUncached(String ownerId)
+            throws UnauthorizedSubscriptionException;;


Suggested change

throws UnauthorizedSubscriptionException;;

throws UnauthorizedSubscriptionException;

IgorKruiper · 2024-11-06T11:01:46Z

queue/src/main/java/com/bazaarvoice/emodb/queue/core/AbstractQueueService.java

            // Check if queueType is "dedupq" and prepend "D" to execution name if true
-            String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;
-
+            String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;;


Suggested change

String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;;

String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;

Removed. :)

IgorKruiper · 2024-11-06T11:08:45Z

sor/src/main/java/com/bazaarvoice/emodb/sor/core/DefaultDataStore.java

+                    if(getDataThrottlerValue())
+                        _kafkaProducerService.sendMessages(MASTER_FANOUT_TOPIC, updateRefs, "update");
+                    else
+                        _eventWriterRegistry.getDatabusWriter().writeEvents(updateRefs);


issue(blocker): this writeEvents function is called twice (line 760 and line 764) - I don't think that's desirable.

This line got duplicated as part of rebasing. Thanks for noticing this.

IgorKruiper

Thanks

rishuyadavbv and others added 5 commits November 4, 2024 17:53

feat: execution name sanitization and unit test for stepfn

cce3d28

master_size api changes

49cfe91

Admin API changes (#845)

f082d99

1. /UpdateParameterStore 2. /QueueExecutionAttributes/<queue_type>/<queue_name>

PD-257044-changes-for-databus (#846)

05fb5ed

added datadog metrics changes

62dc31e

nabajyotiDash-hub requested a review from bharadwaj-k November 4, 2024 15:00

nabajyotiDash-hub requested a review from a team as a code owner November 4, 2024 15:00

nabajyotiDash-hub changed the title ~~Master2~~ Databus Throttling Changes Nov 4, 2024

bharadwaj-k reviewed Nov 5, 2024

View reviewed changes

sor/pom.xml Outdated Show resolved Hide resolved

sor/src/main/java/com/bazaarvoice/emodb/sor/core/DefaultDataStore.java Show resolved Hide resolved

anurag0510 and others added 6 commits November 5, 2024 16:03

[PD-257044] databus changes rebase (#848)

72ed737

* PD-257044-changes-for-databus * duplicated kafka and ssm util packages in emo sor module * removed redundant dependencies

admin api and queue name split

0dad904

Pd 257044 databus changes rebase (#849)

63d05f0

* PD-257044-changes-for-databus * duplicated kafka and ssm util packages in emo sor module * removed redundant dependencies * additional changes * additional changes * additional changes * removed unneccessary dependencies

Revert "added datadog metrics changes"

be3df07

This reverts commit 62dc31e.

build fix

6b0b672

endpoint change and databus changes

6fab83e

bharadwaj-k reviewed Nov 6, 2024

View reviewed changes

anurag0510 and others added 2 commits November 6, 2024 05:41

updated updateRefInDatabus method

ff905b0

address review commnets

02c3a35

bharadwaj-k previously approved these changes Nov 6, 2024

View reviewed changes

branch admin -prepare release emodb-6.5.205

7d71b48

bv-jenkins dismissed bharadwaj-k’s stale review via 7d71b48 November 6, 2024 06:43

branch admin -prepare for next development iteration

6193999

bharadwaj-k previously approved these changes Nov 6, 2024

View reviewed changes

nabajyotiDash-hub enabled auto-merge (squash) November 6, 2024 07:12

nabajyotiDash-hub disabled auto-merge November 6, 2024 07:16

nabajyotiDash-hub enabled auto-merge (squash) November 6, 2024 07:16

nabajyotiDash-hub requested a review from anandujayan November 6, 2024 07:24

anandujayan previously approved these changes Nov 6, 2024

View reviewed changes

edit doc

05cfcbf

nabajyotiDash-hub dismissed anandujayan’s stale review via 05cfcbf November 6, 2024 07:34

nabajyotiDash-hub dismissed bharadwaj-k’s stale review via 05cfcbf November 6, 2024 07:34

bharadwaj-k previously approved these changes Nov 6, 2024

View reviewed changes

fix tests

1861f8b

nabajyotiDash-hub dismissed bharadwaj-k’s stale review via 1861f8b November 6, 2024 07:58

anandujayan previously approved these changes Nov 6, 2024

View reviewed changes

fix tests

da60cdb

nabajyotiDash-hub dismissed anandujayan’s stale review via da60cdb November 6, 2024 08:18

bharadwaj-k previously approved these changes Nov 6, 2024

View reviewed changes

IgorKruiper suggested changes Nov 6, 2024

View reviewed changes

fix review comments

a40a48c

nabajyotiDash-hub dismissed bharadwaj-k’s stale review via a40a48c November 6, 2024 11:34

removed ;

ab2c2d1

IgorKruiper approved these changes Nov 6, 2024

View reviewed changes

jenkins and others added 2 commits November 6, 2024 13:28

branch admin -prepare release emodb-6.5.206

6e1cf00

upgrade pom version

3373932

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Databus Throttling Changes #847

Databus Throttling Changes #847

nabajyotiDash-hub commented Nov 4, 2024

bharadwaj-k left a comment

IgorKruiper left a comment

IgorKruiper Nov 6, 2024

nabajyotiDash-hub Nov 6, 2024

IgorKruiper Nov 6, 2024

nabajyotiDash-hub Nov 6, 2024

IgorKruiper Nov 6, 2024

nabajyotiDash-hub Nov 6, 2024

IgorKruiper left a comment

	throws UnauthorizedSubscriptionException;;
	throws UnauthorizedSubscriptionException;

	String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;;
	String executionName = (queueType.equalsIgnoreCase("dedupq") ? "D_" : "") + queueName + "_" + timestamp;

Databus Throttling Changes #847

Are you sure you want to change the base?

Databus Throttling Changes #847

Conversation

nabajyotiDash-hub commented Nov 4, 2024

Github Issue

What Are We Doing Here?

How to Test and Verify

Risk

Level

Required Testing

Risk Summary

Code Review Checklist

bharadwaj-k left a comment

Choose a reason for hiding this comment

IgorKruiper left a comment

Choose a reason for hiding this comment

IgorKruiper Nov 6, 2024

Choose a reason for hiding this comment

nabajyotiDash-hub Nov 6, 2024

Choose a reason for hiding this comment

IgorKruiper Nov 6, 2024

Choose a reason for hiding this comment

nabajyotiDash-hub Nov 6, 2024

Choose a reason for hiding this comment

IgorKruiper Nov 6, 2024

Choose a reason for hiding this comment

nabajyotiDash-hub Nov 6, 2024

Choose a reason for hiding this comment

IgorKruiper left a comment

Choose a reason for hiding this comment