Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Databus Throttling Changes #847

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
+1,492 −159
Open

Databus Throttling Changes #847

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
cce3d28
feat: execution name sanitization and unit test for stepfn
rishuyadavbv Oct 28, 2024
49cfe91
master_size api changes
nabajyotiDash-hub Nov 4, 2024
f082d99
Admin API changes (#845)
shan-1094 Nov 4, 2024
05fb5ed
PD-257044-changes-for-databus (#846)
anurag0510 Nov 4, 2024
62dc31e
added datadog metrics changes
anurag0510 Oct 26, 2024
72ed737
[PD-257044] databus changes rebase (#848)
anurag0510 Nov 5, 2024
0dad904
admin api and queue name split
anandujayan Nov 5, 2024
63d05f0
Pd 257044 databus changes rebase (#849)
anurag0510 Nov 5, 2024
be3df07
Revert "added datadog metrics changes"
nabajyotiDash-hub Nov 5, 2024
6b0b672
build fix
nabajyotiDash-hub Nov 5, 2024
6fab83e
endpoint change and databus changes
anandujayan Nov 5, 2024
ff905b0
updated updateRefInDatabus method
anurag0510 Nov 6, 2024
02c3a35
address review commnets
nabajyotiDash-hub Nov 6, 2024
7d71b48
branch admin -prepare release emodb-6.5.205
Nov 6, 2024
6193999
branch admin -prepare for next development iteration
Nov 6, 2024
05cfcbf
edit doc
nabajyotiDash-hub Nov 6, 2024
1861f8b
fix tests
nabajyotiDash-hub Nov 6, 2024
da60cdb
fix tests
nabajyotiDash-hub Nov 6, 2024
a40a48c
fix review comments
nabajyotiDash-hub Nov 6, 2024
ab2c2d1
removed ;
nabajyotiDash-hub Nov 6, 2024
6e1cf00
branch admin -prepare release emodb-6.5.206
Nov 6, 2024
3373932
upgrade pom version
nabajyotiDash-hub Nov 6, 2024
2941007
branch admin -prepare release emodb-6.5.208
Nov 6, 2024
54122d9
branch admin -prepare for next development iteration
Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion auth/auth-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-store/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion auth/auth-util/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>emodb</artifactId>
<groupId>com.bazaarvoice.emodb</groupId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion blob-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob-clients/blob-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion blob/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion cachemgr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/astyanax/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jax-rs-2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/dropwizard/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/jersey-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/json/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/stash/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/uuid/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion common/zookeeper/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
4 changes: 4 additions & 0 deletions databus-api/src/main/java/com/bazaarvoice/emodb/databus/api/Databus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ Subscription getSubscription(String subscription)
*/
long getEventCountUpTo(String subscription, long limit);

default long getMasterCount(){
return 0;
};

/** Counts the number of events with outstanding claims that cause the events to be skipped by {@link #poll}. */
long getClaimCount(String subscription);

Expand Down
2 changes: 1 addition & 1 deletion databus-client-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client-jersey2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion databus/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
5 changes: 5 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DatabusFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ public long getEventCountUpTo(String subscription, long limit) {
return _ownerAwareDatabus.getEventCountUpTo(ownerId, subscription, limit);
}

@Override
public long getMasterCount() {
return _ownerAwareDatabus.getMasterEventCountUncached(ownerId);
}

@Override
public long getClaimCount(String subscription) {
return _ownerAwareDatabus.getClaimCount(ownerId, subscription);
Expand Down
12 changes: 12 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/DefaultDatabus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,18 @@ public long getEventCount(String ownerId, String subscription) {
return getEventCountUpTo(ownerId, subscription, Long.MAX_VALUE);
}

@Override
public long getMasterEventCountUncached(String ownerId) {
_log.info("Inside getMasterEventCountUncached with _masterFanoutChannels length {}", _masterFanoutChannels.size());
long size = 0;
for(String channel : _masterFanoutChannels) {
size += _eventStore.getSizeEstimate(channel, Long.MAX_VALUE);
_log.info("From channel size {} {}:", channel, size);
}
return size;
}


@Override
public long getEventCountUpTo(String ownerId, String subscription, long limit) {
checkSubscriptionOwner(ownerId, subscription);
Expand Down
3 changes: 3 additions & 0 deletions databus/src/main/java/com/bazaarvoice/emodb/databus/core/OwnerAwareDatabus.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ Subscription getSubscription(String ownerId, String subscription)
long getEventCount(String ownerId, String subscription)
throws UnauthorizedSubscriptionException;

public long getMasterEventCountUncached(String ownerId)
throws UnauthorizedSubscriptionException;;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
throws UnauthorizedSubscriptionException;;
throws UnauthorizedSubscriptionException;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed.


long getEventCountUpTo(String ownerId, String subscription, long limit)
throws UnauthorizedSubscriptionException;

Expand Down
2 changes: 1 addition & 1 deletion datacenter/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion event/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion job/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion kafka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion megabus/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion parent/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<packaging>pom</packaging>

<name>EmoDB Parent</name>
Expand Down
2 changes: 1 addition & 1 deletion plugins/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>parent/pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion quality/integration/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>com.bazaarvoice.emodb</groupId>
<artifactId>emodb-parent</artifactId>
<version>6.5.205-SNAPSHOT</version>
<version>6.5.206-SNAPSHOT</version>
<relativePath>../../parent/pom.xml</relativePath>
</parent>

Expand Down
Loading