Skip to content

[feat]PIP-410: Add a metric to track publish buffer usage.#23751

Open
dao-jun wants to merge 3 commits intoapache:masterfrom
dao-jun:feature/publish_buffer_usage_metric
Open

[feat]PIP-410: Add a metric to track publish buffer usage.#23751
dao-jun wants to merge 3 commits intoapache:masterfrom
dao-jun:feature/publish_buffer_usage_metric

Conversation

@dao-jun
Copy link
Copy Markdown
Member

@dao-jun dao-jun commented Dec 18, 2024 •
edited
Loading

PIP: #24028

Motivation

Add a metric to track broker side publish buffer usage.

Modifications

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

@github-actions github-actions Bot added the doc-required Your PR changes impact docs and you will update later. label Dec 18, 2024
lhotari
lhotari requested changes Dec 20, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since Pulsar is slowly migrating to OTel for metrics, please also handle OTel for this added metric.

For metric changes, it would be useful to at least have a discussion on the dev mailing list since our PIP policy states that a PIP is required for changes in metrics.

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java Outdated
@dao-jun
Copy link
Copy Markdown
Member Author

dao-jun commented Jan 10, 2025

@lhotari

Since Pulsar is slowly migrating to OTel for metrics, please also handle OTel for this added metric.

For metric changes, it would be useful to at least have a discussion on the dev mailing list since our PIP policy states that a PIP is required for changes in metrics.

Thanks for your comment. You are right, this a WIP PR, and I will do it later when I am available

@dao-jun
Merge branch 'master' into feature/publish_buffer_usage_metric
b4dd337 
# Conflicts:
#	pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@dao-jun
impl publish buffer usage metrics
3e9241b
@dao-jun dao-jun self-assigned this Feb 26, 2025
@dao-jun dao-jun closed this Feb 26, 2025
@dao-jun dao-jun reopened this Feb 26, 2025
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Feb 26, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.19%. Comparing base (bbc6224) to head (3e9241b).
⚠️ Report is 1771 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #23751      +/-   ##
============================================
+ Coverage     73.57%   74.19%   +0.62%     
+ Complexity    32624    32385     -239     
============================================
  Files          1877     1861      -16     
  Lines        139502   144189    +4687     
  Branches      15299    16420    +1121     
============================================
+ Hits         102638   106987    +4349     
+ Misses        28908    28752     -156     
- Partials       7956     8450     +494
Flag Coverage Δ
inttests 26.72% <100.00%> (+2.14%) ⬆️
systests 23.14% <100.00%> (-1.19%) ⬇️
unittests 73.69% <100.00%> (+0.85%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...rg/apache/pulsar/broker/service/BrokerService.java 84.57% <100.00%> (+3.78%) ⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java 72.00% <100.00%> (-0.15%) ⬇️

... and 1054 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dao-jun dao-jun changed the title [feature][monitoring] Add a metric to track publish buffer usage. [monitoring]PIP-410: Add a metric to track publish buffer usage. Feb 26, 2025
@dao-jun dao-jun changed the title [monitoring]PIP-410: Add a metric to track publish buffer usage. [feat]PIP-410: Add a metric to track publish buffer usage. Feb 26, 2025
crossoverJie
crossoverJie reviewed Feb 28, 2025
View reviewed changes
Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
OpenTelemetryAssertions.assertThat(metricData).anySatisfy(metric ->
OpenTelemetryAssertions.assertThat(metric).hasName("pulsar.broker.publish.buffer.usage")
.hasLongGaugeSatisfying(gauge ->
gauge.hasPointsSatisfying(point -> point.hasValue(0))));
Copy link
Copy Markdown
Member

@crossoverJie crossoverJie Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initial value and result are both 0, it cannot be determined whether it has taken effect?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@crossoverJie crossoverJie crossoverJie left review comments

@lhotari lhotari lhotari requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@dao-jun dao-jun

Labels

area/broker area/metrics doc-required Your PR changes impact docs and you will update later. ready-to-test type/PIP

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dao-jun @codecov-commenter @lhotari @crossoverJie