improve the accuracy of memory occupation estimate of InMemoryDelayedDeliveryTracker.

[thetumbled]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Now, the memory usage of InMemoryDelayedDeliveryTracker is based on org.roaringbitmap.longlong.Roaring64Bitmap#getLongSizeInBytes, referenced in org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTracker#getBufferMemoryUsage.

According to the comment of the method,

  /**
   * Estimate of the memory usage of this data structure. This can be expected to be within 1% of
   * the true memory usage in common usage scenarios.
   * If exact measures are needed, we recommend using dedicated libraries
   * such as ehcache-sizeofengine.
   *
   * In adversarial cases, this estimate may be 10x the actual memory usage. For example, if
   * you insert a single random value in a bitmap, then over a 100 bytes may be used by the JVM
   * whereas this function may return an estimate of 32 bytes.
   *
   * The same will be true in the "sparse" scenario where you have a small set of
   * random-looking integers spanning a wide range of values.
   *
   * These are considered adversarial cases because, as a general rule,
   * if your data looks like a set
   * of random integers, Roaring bitmaps are probably not the right data structure.
   *
   * Note that you can serialize your Roaring Bitmaps to disk and then construct
   * ImmutableRoaringBitmap instances from a ByteBuffer. In such cases, the Java heap
   * usage will be significantly less than
   * what is reported.
   *
   * If your main goal is to compress arrays of integers, there are other libraries
   * that are maybe more appropriate
   * such as JavaFastPFOR.
   *
   * Note, however, that in general, random integers (as produced by random number
   * generators or hash functions) are not compressible.
   * Trying to compress random data is an adversarial use case.
   *
   * @see <a href="https://github.com/lemire/JavaFastPFOR">JavaFastPFOR</a>
   *
   *
   * @return estimated memory usage.
   */
  @Override
  public long getLongSizeInBytes() {
    // 'serializedSizeInBytes' is a better than nothing estimation of the memory footprint
    // It would generally be an optimistic estimator (by underestimating the size in memory)
    return serializedSizeInBytes();
  }

We can see that, the essitimate value is within 1% of the true memory usage in common usage scenarios, while in adversarial cases, this estimate may be 10x the actual memory usage, which range wide.

As delayed message queue consume a lot memory, we need to measure it more accurate.

Solution

According to the comment of method, a lib called ehcache-sizeofengine is recommended.
https://github.com/ehcache/sizeof
I wonder whether it is the best solution, is there any lib better than it?

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[thetumbled]

Do you have any idea on it? @lhotari @dao-jun @poorbarcode @codelipenghui @BewareMyPower

[lhotari]

As delayed message queue consume a lot memory, we need to measure it more accurate.

@thetumbled We first need to understand the use case and problem to be solved. Please provide more context of what this measurement information is used for. This could help us understand why this would be important.

For RoaringBitmaps, there's also a method trim() which could reduce the amount of allocated memory after bits have been cleared in the data structure.
Just wondering if the solution would be more about knowning when it would be useful to call trim() to reduce the memory consumption. One possible naive solution could be to track the number of bits unset and simply call trim() if a certain threshold is reached. (Let's say after 1000 bits have been unset, call trim())

I checked the RoaringBitmaps source code and it seems that there isn't currently an API to calculate the total size of allocated memory. What trim() does is that it reduces the array sizes of the internal data structure to the actual used size.

According to the comment of method, a lib called ehcache-sizeofengine is recommended.
https://github.com/ehcache/sizeof

Where is this recommended? This is a very heavy weight solution. I wouldn't recommend it at all for this use case.

[thetumbled]

As delayed message queue consume a lot memory, we need to measure it more accurate.

@thetumbled We first need to understand the use case and problem to be solved. Please provide more context of what this measurement information is used for. This could help us understand why this would be important.

When using delayed message queue feature, broker face with the risk of OOM as client may set enormous number of entries to be delayed due to incorrect usage or logical bug. We need to monitor the memory occupation of each user task to distinguish the bad task.

For RoaringBitmaps, there's also a method trim() which could reduce the amount of allocated memory after bits have been cleared in the data structure. Just wondering if the solution would be more about knowning when it would be useful to call trim() to reduce the memory consumption. One possible naive solution could be to track the number of bits unset and simply call trim() if a certain threshold is reached. (Let's say after 1000 bits have been unset, call trim())

I checked the RoaringBitmaps source code and it seems that there isn't currently an API to calculate the total size of allocated memory. What trim() does is that it reduces the array sizes of the internal data structure to the actual used size.

IMO, there is no need to use trim() method to reduce the size of unused space of bitmap for the reason that we will remove the total bitmap most of the time. Decrement the size of one single bitmap only release few of space, which is insignificant.

According to the comment of method, a lib called ehcache-sizeofengine is recommended.
https://github.com/ehcache/sizeof

Where is this recommended? This is a very heavy weight solution. I wouldn't recommend it at all for this use case.

The comment of method recommend it:

If exact measures are needed, we recommend using dedicated libraries such as ehcache-sizeofengine.

The stars of this repo is few and it is not the best solution seemingly, so i search for help from the community to find out the best one.