-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unpacked values from scaled/offset dataset are all NaN. #1312
Comments
@rschmunk just to clarify, the place where Panoply is getting incorrect data is from a call to |
Actually
where If you're wondering why I acquire the dataset in two steps, it's actually because there's also a step where I acquire an unenhanced version in order to see the original CDL description. |
@haileyajohnson, I tracked this down, and it has to do with the changes made to the TL;DR - the unpacked data values from the sample dataset are all much smaller than Long version... As noted above, the scale and offset values in the sample dataset are really small. Tiny even. The scale factor is of order 1E-12 and the offset of order 1E-7, and after unpacking, the output data results are of order 1E-8. The dataset also specifies that the The problem then is that So it seems like a smaller difference value needs to be used in the test when dealing with such tiny scale factors and offsets. |
Thanks for tracking that down! We did change the fuzzyequals comparison to use a relative diff instead of absolute diff, and that's what broke this case - I'm planning on reverting that change. But another issue I think is that we need to do the missing and fill value comparisons with unpacked data, so that the fuzyequals implementation doesn't matter. I should have a fix up shortly. Thanks for all your help! |
@haileyajohnson, I thought about trying to hack up a test of modifying |
I downloaded a fresh snapshot and built it into Panoply. It plots the sample data as expected, so it looks like #1315 was what was needed. |
Versions impacted by the bug
Any build of version 5.5.4 from the past month.
What went wrong?
A user wrote that a plot of his data was coming up blank when using a just-downloaded Panoply. A quick check revealed that Panoply was seeing all NaN values even though an ncdump showed good values. Some further testing demonstrated that this was also occurring with a copy of Panoply from a month ago, but that a copy from the very end of December 2023 was plotting as expected. In all three cases, the Panoply release was using an up-to-date build of the netcdf-Java 5.x-maint snapshot.
The dataset packs the data as shorts and specifies a scale value and an offset. Panoply acquires the dataset in enhanced mode and accepts whatever values netCDF-Java says are there.
If I modify Panoply to acquire the dataset in NOT enhanced mode, then netCDF-Java reports the unpacked short values and a plot is successfully made. If I build a copy of Panoply using a copy of netcdfAll that I built on December 28 and tell it to acquire the dataset in enhanced mode, the unpacked data values are correctly reported and a plot is successfully made.
Looking at the commit history on the 5.x-maint branch, it looks like there were some modifications to unpacking data made about January 12, which seems a likely point where the problem is occurring. Nothing I noticed in the changes then jumps out at me as the obvious cause, but I haven't dug very deep yet.
Not sure if it's at all meaningful, but the scale and offset values are reeeaally small.
A copy of the example dataset that revealed the issue can be downloaded from Dropbox. It's a CF-1.6 dataset with lon-lat data on a relatively small array covering the re4gion around Mexico City.
Relevant stack trace
No response
Relevant log messages
No response
If you have an example file that you can share, please attach it to this issue.
If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.
Yes
Code of Conduct
The text was updated successfully, but these errors were encountered: