SearchBackPressure Service Node/Cluster RCA

[CoderJeffrey]

Is your feature request related to a problem? Please provide an existing Issue # , or describe.
After Reader Component of RCA digest Search Back Pressure service related stats and populate the stats in sqlitedb, we need Cluster/Node level RCA to analyze the stats and send node/cluster level Resource Flow Units to deciders. Then, deciders can use the resource flow units to suggest actions for downstream classes.

Describe the solution you are proposing
Create Cluster/Node Level RCA to generate Resource FlowUnit by comparing stats of SearchBackPressure Service and threshold, and added corresponding UTs.

Search Back Pressure RCA analyze the stats of Searchbp_Stats Table in sqlitedb from previous PR of Reader Component for RCA (#427). Example stats (e.g heap_cancellation_count with threshold, generate healthy/unhealthy resource flow units) are used to determine the health status of a node.

Describe alternatives you've considered
N/A

Additional context
The logic to decide whether the node level rca healthiness is not decided yet, we will talk with other teams (e.g. SearchBackPressure team) to discuss the final threshold for RCA, and the logic will likely change in near future.

Check List

• New functionality includes testing.
• All tests pass
• New functionality has been documented.
• New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ansjcy]

nit: avoid trailing space

[CoderJeffrey]

removed

[ansjcy]

will the values of these thresholds be modified during runtime? If not we can make them final as well.

[CoderJeffrey]

Yes, these values like minTaskHeapCancellationPercentageThreshold will read values from config file in runtime thus we cannot mark them as final

minHeapDecreasePercentageThreshold = conf.readRcaConfig( CONFIG_NAME, RCA_CONF_KEY_CONSTANTS.MAX_HEAP_USAGE_DECREASE_FIELD, DEFAULT_MIN_HEAP_DECREASE_THRESHOLD, (s) -> s >= 0 && s <= 100, Integer.class);

[ansjcy]

It might be better to use enum instead of public static class here to define a list of related constants.
In general, enums offer better readability, type safety, and if we need to iterating over those constants we can simply use the values() method. This is more straightforward than manually iterating over constants in a public static final class.
you can do something like:

    public enum RCA_CONF_KEY_CONSTANTS {
        MAX_HEAP_USAGE_INCREASE_FIELD("max-heap-usage-increase"),
        MAX_SHARD_HEAP_CANCELLATION_PERCENTAGE_FIELD("max-shard-heap-cancellation-percentage");
        // ...

        private final String value;
        RCA_CONF_KEY_CONSTANTS(final String value) {
            this.value = value;
        }
        @Override
        public String toString() {
            return value;
        }
    }

[CoderJeffrey]

Thank you so much for the suggestion. I have changed the design from static class to Enum.

[ansjcy]

nit: use TODO instead of To Do for get better IDE support

[CoderJeffrey]

Fixed

[ansjcy]

nit: olg -> old

[CoderJeffrey]

fixed

[ansjcy]

This variable pollFirstCondition is not used. We can remove it

[CoderJeffrey]

removed

[ansjcy]

operate() is used for local build

is this related to this function?

[CoderJeffrey]

the operate() line is a dead comment for that function, removed.

[ansjcy]

we can change some of the LOG.info messages to LOG.debug. LOG.info should only contains essential information to store in the service log.

[CoderJeffrey]

Change unnecessary log message level to debug