The result of ea_conditioning is nan

[yuyinw]

Hello
My entropy is CPU Jitter ： hin =64 nin= 4096 nw =64 nout=64,

   when i use ea_conditioning, the result :
     [wy@localhost cpp]$ ./ea_conditioning -n 4096 64 64 64 1
    n_in: 4096.000000
    n_out: 64.000000
    nw: 64.000000
    h_in: 64.000000
    h': 1.000000
    (Non-vetted) h_out: nan

[joshuaehill]

The calculation for p_low underflows to 0. This is fixed in the arbitrary precision PR #136, where the output is instead:

./ea_conditioning -n 4096 64 64 64 1
n_in = 4096
n_out = 64
nw = 64
h_in = 64
h' = 1
Output_Entropy(*) = 63
0.999 * n_out = 63.93599999999999999866
h' * n_out = 64
(Non-vetted) h_out = 63

[yuyinw]

Than you joshuaehill.
Expect this problem to be fixed.
I see 90B have a Health Test, but SP800-90B_EntropyAssessmentseems have no corresponding tools

[joshuaehill]

The health tests are intended to be integrated into your entropy source. The NIST tools are used in the testing of such entropy sources, but they don't include an implementation of an entropy source.

[yuyinw]

Thank you joshuaehill

[joshuaehill]

@celic, if you would like, I can investigate further and see if there's some alternate approach that wouldn't require moving the entire calculation over to some arbitrary precision library. I've looked into this enough to know precisely where the underflow occurs, and to verify that the arbitrary-precision PR #136 solves the issue, but I haven't spent any time trying to determine if there is some less-completest approach that might also solve the problem.

[celic]

You're welcome to take a look. I'd prefer as few additional libraries as possible.

…

On Thu, Sep 9, 2021, 1:05 PM Joshua E. Hill ***@***.***> wrote: @celic <https://github.com/celic>, if you would like, I can investigate further and see if there's some alternate approach that wouldn't require moving the entire calculation over to some arbitrary precision library. I've looked into this enough to know precisely the underflow occurs, and to verify that the arbitrary-precision PR #136 <#136> solves the issue, but I haven't spent any time trying to determine if there is some less-completest approach that might also solve the problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATQXEI2KXPUAOHLUDG4P6DUBDSNFANCNFSM5DQI5UJA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[joshuaehill]

I see a number of problems, and I don't know an easy way of dealing with them.

1. Nearly every input is treated as an exponent in some sense, and the values (particularly n_in, as in this case) may be large. The biggest/smallest exponent we can represent is on the order of 1024, but sometimes you want bigger values (as in this bug report). Traditionally, this would be dealt with by putting everything in terms of logs, but all the calculations feature addition, so that wouldn't be that helpful.
2. Even when we can represent the integer, to work with P_low, we need values that look like 2^(big number) - 1. Due to the way that floating point numbers are represented, depending on the rounding mode, this will be either equal to 2^(big number) (i.e., unchanged) or 1 ULP lower (where the size of the ULP is on the order of 2^(big number-52). This isn't likely to make a big difference for most large parameter choices. In this case, we need to overestimate P_low, so I guess we should round toward 0 here, even though this clearly results in a less accurate answer.

If we want to deal with basically arbitrary inputs, the you're going to need to be really clever in some presently obscure way. Anyone else have any ideas? (I mean, other than using an arbitrary-precision library, which is what PR #136 already does).

[joshuaehill]

@celic Now that PR #136 has been merged, I think that this issue can be closed.