Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support Incremental Mode for metrics streaming #12290

Closed
2 of 3 tasks
wu-sheng opened this issue Jun 3, 2024 · 3 comments
Closed
2 of 3 tasks

[Feature] Support Incremental Mode for metrics streaming #12290

wu-sheng opened this issue Jun 3, 2024 · 3 comments
Assignees
Labels
backend OAP backend related. core feature Core and important feature. Sometimes, break backwards compatibility. feature New feature rejected The issue or PR can't be accepted by upstream.
Milestone

Comments

@wu-sheng
Copy link
Member

wu-sheng commented Jun 3, 2024

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

#8720 is being planned as a part of the OSPP and BanyanDB milestone. OAP need a kernel-level change to adopt this server-side feature.

Use case

Currently, OAP is using this workflow, Receiver -> L1 aggregation -> OAP in-cluster hashing calling -> L2 aggregation -> cache + DB merging + persistent.

By leveraging #8720, once the server-side merging functions are supported (even partially), the workflow could be evolved to Receiver -> L1 aggregation -> no cache and pending persistent.

All these supported metrics would not need to do

  1. Serialization and Deserialization for in-cluster aggregation
  2. No RPCs among OAP nodes
  3. No L2 aggregation
  4. No memory cost(no cache mode) before persistent after the L2 aggregation

L1 aggregation would deliver incremental metrics for persistence directly.


As OAP could run in the incremental mode, we will face the impacts as there was no latest data.

  1. Alerting context will be missed
  2. Total mode exporting will not be supported

We could use periodical reading to retrieve metrics for active metrics entities detected by the workflows. <2> will not be supported.


Besides the above-mentioned things, we should consider adding configurations to control the OAP mode

  1. Run db-side-merging for metrics supported by BanyanDB. DB metadata query should support to indicate the supported function lists.
  2. Run original workflow for not supported metrics, even the db-side-merging is supported.
  3. Run duplicate writing for every metrics. If the metric functions are supported in <1>,

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

  • Yes I am willing to submit a pull request on my own!

Code of Conduct

@wu-sheng wu-sheng added core feature Core and important feature. Sometimes, break backwards compatibility. backend OAP backend related. feature New feature labels Jun 3, 2024
@wu-sheng wu-sheng added this to the 10.1.0 milestone Jun 3, 2024
@wu-sheng
Copy link
Member Author

wu-sheng commented Jun 3, 2024

@hanahmily We could implement most of the logic before the DB side is ready. The only dependencies

  1. APIs about whether the metric function is supported
  2. APIs about creating db-side-merging metric

@wu-sheng
Copy link
Member Author

Metrics should consider having @DBMergingCapability(function="avg") annotation to indicate this is a metric could support DB server side merging. But the OAP would rely on the server API to check whether the current function is provided at the DB side.

@wu-sheng wu-sheng modified the milestones: 10.1.0, 10.2.0 Aug 30, 2024
@wu-sheng
Copy link
Member Author

@wankai123 @hanahmily I am moving this to 10.2. Although I am not sure whether we will be ready for this, this is so expected for BanyanDB for sure.

@wu-sheng wu-sheng modified the milestone: 10.2.0 Sep 27, 2024
@wu-sheng wu-sheng added the rejected The issue or PR can't be accepted by upstream. label Sep 27, 2024
@wu-sheng wu-sheng closed this as not planned Won't fix, can't repro, duplicate, stale Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend OAP backend related. core feature Core and important feature. Sometimes, break backwards compatibility. feature New feature rejected The issue or PR can't be accepted by upstream.
Projects
None yet
Development

No branches or pull requests

3 participants