MB-29928: Switch fragmentation scoring/weighting to use high-water mark

The justification for this change is based on current KV behaviour
and some input from a few 'random' real life log snapshots.

1) Current behaviour of KV's item-pager is to try and keep memory
below the high-water mark, at times of memory pressure it will attempt
to move memory to the low-water mark. The defragmenter cannot affect
mem_used, but it can affect the bucket's resident memory. Here we
allow the defragmenter's sleep to become most aggressive when
the resident memory is close/above the high-water mark.

2) It's not uncommon for users to have their mem_used between
the low and high water marks. This change means that the defragmenter
won't just be at it's most 'aggressive' for the users that are happily
between the two, only when they approach the high-water mark will
the scoring/weighting become higher.
Change-Id: Id942db14fe76ca6e18265fb66e379c18eecb6fab

Change-Id: I1f8d9098edf38dbd780b746a85da6a1177850482
Reviewed-on: http://review.couchbase.org/c/kv_engine/+/156643
Reviewed-by: Dave Rigby <[email protected]>
Tested-by: Build Bot <[email protected]>

engines/ep/src/defragmenter.cc

@@ -242,11 +242,12 @@ DefragmentVisitor& DefragmenterTask::getDefragVisitor() {
 
 float DefragmenterTask::getScoredFragmentation(
         const cb::FragmentationStats& fragStats) const {
-    auto lowWater = stats.mem_low_wat.load();
-    auto rss = fragStats.getResidentBytes() > lowWater
-                       ? lowWater
+    const auto highWater = stats.mem_high_wat.load();
+    auto rss = fragStats.getResidentBytes() > highWater
+                       ? highWater
                        : fragStats.getResidentBytes();
-    return fragStats.getFragmentationRatio() * (double(rss) / double(lowWater));
+    return fragStats.getFragmentationRatio() *
+           (double(rss) / double(highWater));
 }
 
 DefragmenterTask::SleepTimeAndRunState DefragmenterTask::calculateSleepLinear(

engines/ep/src/defragmenter.h

@@ -154,18 +154,29 @@ class DefragmenterTask : public GlobalTask {
     virtual float stepPid(float pv);
 
     /**
-     * The auto controller modes don't use raw fragmentation, but a 'scored'
-     * value using low-water mark and RSS. This function gets that value from
-     * fragStats.
-     * The current implementation of this returns the following example values
-     * Consider a raw fragmentation of 23% where allocated=500 and rss=650.
-     * Then with a low-water mark of n this function returns (score):
+     * The auto controller modes do not use raw fragmentation, but a 'scored'
+     * value that is weighted by the ratio of RSS/high-water mark.
+     *
+     * This is done because bucket fragmentation alone is not a great guide for
+     * how aggressive we would want to defragment the hash-table. For example
+     * an empty or very low utilised bucket, can easily reach high fragmentation
+     * percentages, yet running the defragmenter at high frequency has little
+     * effect. Using the RSS/high-water mark ratio means helps to guide our
+     * defragmentation sleep changes only when:
+     * 1) There are items to defragment
+     * 2) When the bucket is actually using a good chunk of its quota
+     *
+     * The current implementation of this returns the following example values:
+     * Here we consider allocated:=500 and rss:=650, this gives a fragmentation
+     * ratio of 0.23.
+     *
+     * Then with a high-water mark of n this function returns:
      *    n    | score
-     *    600  | 23   (note this case, rss exceeds n).
-     *    1000 | 14.95
-     *    2000 | 7.4
-     *    3000 | 4.98
-     *    5000 | 2.99
+     *    600  | 0.23 (note this case, rss exceeds n, we fix weight to 1).
+     *    1000 | 0.1495
+     *    2000 | 0.074
+     *    3000 | 0.0498
+     *    5000 | 0.0299
      *
      * @return the scored fragmentation
      */

engines/ep/tests/module_tests/defragmenter_test.cc

@@ -443,8 +443,8 @@ std::chrono::milliseconds MockDefragmenterTask::MockDefragmenterTaskClock::step;
 // Test that as the PID runs (and we move time forwards) that it assists in the
 // recalculation of sleep time, reducing whilst fragmentation is above threshold
 TEST_F(DefragmenterTaskTest, autoCalculateSleep_PID) {
-    // set lww to some low number
-    engine->getEpStats().setLowWaterMark(750);
+    // set hwm to some low number
+    engine->getEpStats().setHighWaterMark(750);
     auto task = std::make_unique<MockDefragmenterTask>(engine.get(),
                                                        engine->getEpStats());
 
@@ -507,8 +507,8 @@ TEST_F(DefragmenterTaskTest, autoCalculateSleep_PID) {
 }
 
 TEST_F(DefragmenterTaskTest, autoCalculateSleep) {
-    // set lww to some low number
-    engine->getEpStats().setLowWaterMark(750);
+    // set hwm to some low number
+    engine->getEpStats().setHighWaterMark(750);
     auto task = std::make_unique<MockDefragmenterTask>(engine.get(),
                                                        engine->getEpStats());
     auto& conf = engine->getConfiguration();