FIX: Avoid loss of precision when casting in packet loading #781
Conversation
| y = x.astype(dtype, copy=False) | ||
| # We need to compare the arrays to see if we trimmed any values by | ||
| # casting to a smaller datatype (e.g. float64 to uint8, 2.1 to 2) | ||
| if np.array_equal(x, y): |
There was a problem hiding this comment.
This is all somewhat unfortunate in how many "arrays" we'll be creating here. If anyone has ideas for a better way to do this I'm interested in hearing it.
We are potentially creating 3 separate arrays:
- initial asarray from a list -> creates an int64
- Trying to cast to a smaller dtype -> creates a uint8
- Doing this array comparison in a higher-order dtype space.
They are all happening in numpy c-level functions so shouldn't be a huge hit, but it would be nice to not do this array comparison just to check loss of precision.
There was a problem hiding this comment.
My thought would be to modify the _get_minimum_numpy_datatype function to take the derived value logic into account and return the appropriate datatype.
There was a problem hiding this comment.
Really good suggestion, thanks.
I opened up: #786
with an alternative approach, which is actually able to get rid of the numpy try/except array creation too. The one downside is there isn't a great way to test all of the XTCE stuff currently :-/ I did test it on the new SWE XTCE file that was failing and it looks good, but I don't have a great way to parameterize that PR like we do with this one.
greglucas
force-pushed
the
min-dtype-fix
branch
from
4c8d505
to
7e2508b
Compare
When using derived values there can be situations where a linear conversion factor is applied to a uint8 value to turn a raw measurement into a float temperature value for instance. These are represented as a small uint datatype onboard, but need to be represented as a float or larger integer datatype on the ground so we don't lose precision. Previously we were getting 2.1 cast to 2 after the derived types were attempted to be cast to their onboard types.
greglucas
force-pushed
the
min-dtype-fix
branch
from
7e2508b
to
af73099
Compare
|
Closing in favor of the other PR |
Change Summary
Overview
When using derived values there can be situations where a linear conversion factor is applied to a uint8 value to turn a raw measurement into a float temperature value for instance. These are represented as a small uint datatype onboard, but need to be represented as a float or larger integer datatype on the ground so we don't lose precision. Previously we were getting 2.1 cast to 2 after the derived types were attempted to be cast to their onboard types.
closes #780