Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"rec_cnt" for all possible distributions will need to support numbers greater than what unsigned4 would allow #81

Closed
ManjunathVenkataswamy opened this issue Apr 29, 2024 · 9 comments
Assignees

Comments

@ManjunathVenkataswamy
Copy link

The 'rec_cnt' that comes out in the response might need to be changed from being a unsigned4 to unsigned. We have noticed it created an issue with a pattern count that was greater than what a unsgined4 could hold. And down the lane, it ended up coming out as a junk number that made no sense.
Here is an example of how this would cause issues that depend on datapattern library for profiling the data.
if you run a code like this:
unsigned4 test := 4449418834;
test;
you would get 154451538 !!!

Discussed this need with Dan and creating this ticket.

@ManjunathVenkataswamy ManjunathVenkataswamy changed the title rec_cnt for all possible distributions will need to support numbers greater than what unsigned4 would allow "rec_cnt" for all possible distributions will need to support numbers greater than what unsigned4 would allow Apr 29, 2024
@dcamper dcamper self-assigned this Apr 29, 2024
@GordonSmith
Copy link
Member

@dcamper need to sanity check the viz when going from 32bit -> 64bit numbers (JS doesn't support them and they end up being strings)

@dcamper
Copy link
Collaborator

dcamper commented Apr 30, 2024

@GordonSmith good point. What would be the best workaround for values exceeding 2^32? (Or is it 2^31?)

@GordonSmith
Copy link
Member

Its actually converted to a decimal with precision 2^56 (from memory), so you start to lose the insignificant values - but - it gets sent as a string which sometimes catches us out.

@dcamper
Copy link
Collaborator

dcamper commented May 1, 2024

@ManjunathVenkataswamy: Would an UNSIGNED6 suffice for your needs? That would give you a maximum value of 281,474,976,710,655 (2^48 - 1) and yet comfortably remain a number for visualization purposes.

@ManjunathVenkataswamy
Copy link
Author

yes it would.

@dcamper
Copy link
Collaborator

dcamper commented May 2, 2024

@ManjunathVenkataswamy: If you have the ability, please check out the candidate-1.10.0 branch (https://github.com/hpcc-systems/DataPatterns/tree/candidate-1.10.0) and test it. Once accepted, I can get it merged into the platform's Std ECL library.

@ManjunathVenkataswamy
Copy link
Author

I haven't yet been able to test this. But will do so within 2 days.

@dcamper
Copy link
Collaborator

dcamper commented Sep 24, 2024

@ManjunathVenkataswamy Would it be possible for you to confirm the rec_cnt change in the candidate-1.10.0 branch?

@dcamper
Copy link
Collaborator

dcamper commented Sep 27, 2024

Merged into v1.10.0

@dcamper dcamper closed this as completed Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants