ahi obs2ioda remove dimension nstring

[BenjaminRuston]

Description

the nstring dimension is non-standard and is removed

Issue(s) addressed

Resolves #1719

Impact

IODA file without dimension nstring

Checklist

• I have performed a self-review of my own code
• I have run the unit tests before creating the PR

[BenjaminRuston]

@ibanos90 do you know why the nstring dimension is included? Do not believe we use it, and causes issues if added for radiance data. Similarly the attributes max_datetime and min_datetime we don't use and will remove this can be determined from the MetaData/dateTime themselves

@amstokely if you remember anything please let us know but may remove the nstring dimension and attributes max_datetime and min_datetime

[ibanos90]

@ibanos90 do you know why the nstring dimension is included? Do not believe we use it, and causes issues if added for radiance data. Similarly the attributes max_datetime and min_datetime we don't use and will remove this can be determined from the MetaData/dateTime themselves

@amstokely if you remember anything please let us know but may remove the nstring dimension and attributes max_datetime and min_datetime

Hi @BenjaminRuston, I believe the nstring dimension and max_datetime and min_datetime variables may be inherited from older IODA versions and kept in the files when we used the IODA upgraders, but we do not use in any of our filters. I believe they can be safely removed.

[amstokely]

@BenjaminRuston @ibanos90 I believe this is a legacy attribute from when fixed-length character arrays were used for strings. Since we now use the NetCDF string type, nstring is no longer needed for data representation.

That said, much of the legacy obs2ioda code relies on the number of attributes, variables, and dimensions for indexing (unfortunately). Removing nstring would reduce the dimension count and could break parts of the code that depend on it. I ran into a similar issue before when removing unused variables caused indexing errors.

I'm not sure what level of testing ioda-converters currently has for obs2ioda, but the test suite I wrote back when NCAR maintained it caught these issues quickly. It may also affect analysis scripts that use index-based loops dependent on the number of dimensions.

[BenjaminRuston]

@amstokely my preference is to try to keep the attribute in memory where possible and just not write to the final file

I spoke with @ibanos90 and we believe it can be removed from the output, and we'll try to monitor the output in our applications. My preference is to move forward with this shiny new version and make changes as needed.

From everything I've seen so far it is safe to remove nstring from the files written, will also remove the min/max dateTime attributes

[fcvdb]

Thanks Ben!

[BenjaminRuston]

@ibanos90 think this is ready, would be good to have confirmation on your side that it is a lateral move

@amstokely you were right there is something buried in there somewhere appears to be conventional data related so only skipping the nstring dimension for non-conventional data types

[fabiolrdiniz]

Thanks, @BenjaminRuston!

[rajichidamb]

This file size seem to be bigger than original.