Public Service Announcement: Skywalker and floating point digits

[jeff-cohere]

Back in the summer of '22, @overfelt and I decided to put a limit on the number of digits output by Skywalker. These limits are:

• single precision: 6 digits after the decimal point
• double precision: 10 digits after the decimal point

You can find the relevant line in Skywalker here.

Up until now, these precision limits haven't really come up as an issue with our cross validation against Fortran implementations. However, today @pressel encountered a case in validating the water uptake parameterization in which the Fortran YAML and Python data files that we compare against actually exceed the limits. It was confusing for both of us until we realized that Skywalker was capping the number of digits when writing out the mam4xx data module.

It's kind of surprising that it took us this long to encounter this kind of situation. Kyle and I both feel that this is not a big deal and that we can continue our work with the current limits in place within Skywalker. It's something that we should all be aware of, though, at the very least, so we're documenting it here and in Kyle's above-linked PR.

Let us know if you have questions or concerns about this!

@overfelt @pbosler @singhbalwinder @jaelynlitz @mjs271 @odiazib @cameronrutherford

[singhbalwinder]

This may be due to a recent change in E3SM-generated yaml/py files where we started printing in double precision. I thought it would be helpful in tracking down places where we see small differences blowing up to big numbers. @overfelt and I went through an exercise where the double precision numbers helped him track down at least one such calculation in convective parameterization.

Previously, I was also limiting the double precision to 10 significant digits (don't remember why I decided to choose that limit 😕 )

[jeff-cohere]

Thanks, @singhbalwinder . Do you and/or @overfelt feel like we should update Skywalker to write out floating point numbers in full precision?

[singhbalwinder]

I think it will be helpful to capture the full precision. Leaving trailing digits sometimes can make a big difference. We should not go back and redo our analysis but we can use the full precision moving forward.

[jeff-cohere]

Update and clarification: Skywalker actually reads input data in at the level of precision in the YAML file and does the calculation at that full precision, so the only difference is its truncated output. @singhbalwinder has since said to me that he agrees with me and @pressel that we can continue the way we're working, given that the loss in precision is just an artifact of the output.