-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"rec_cnt" for all possible distributions will need to support numbers greater than what unsigned4 would allow #81
Comments
@dcamper need to sanity check the viz when going from 32bit -> 64bit numbers (JS doesn't support them and they end up being strings) |
@GordonSmith good point. What would be the best workaround for values exceeding 2^32? (Or is it 2^31?) |
Its actually converted to a decimal with precision 2^56 (from memory), so you start to lose the insignificant values - but - it gets sent as a string which sometimes catches us out. |
@ManjunathVenkataswamy: Would an UNSIGNED6 suffice for your needs? That would give you a maximum value of 281,474,976,710,655 (2^48 - 1) and yet comfortably remain a number for visualization purposes. |
yes it would. |
@ManjunathVenkataswamy: If you have the ability, please check out the candidate-1.10.0 branch (https://github.com/hpcc-systems/DataPatterns/tree/candidate-1.10.0) and test it. Once accepted, I can get it merged into the platform's Std ECL library. |
I haven't yet been able to test this. But will do so within 2 days. |
@ManjunathVenkataswamy Would it be possible for you to confirm the rec_cnt change in the candidate-1.10.0 branch? |
Merged into v1.10.0 |
The 'rec_cnt' that comes out in the response might need to be changed from being a unsigned4 to unsigned. We have noticed it created an issue with a pattern count that was greater than what a unsgined4 could hold. And down the lane, it ended up coming out as a junk number that made no sense.
Here is an example of how this would cause issues that depend on datapattern library for profiling the data.
if you run a code like this:
unsigned4 test := 4449418834;
test;
you would get 154451538 !!!
Discussed this need with Dan and creating this ticket.
The text was updated successfully, but these errors were encountered: