Enforcing IODA v3 Schema Compliance in `obs2ioda`

[amstokely]

Summary

The obs2ioda library currently parses the IODA ObsSpace.yaml schema to:

• Rename deprecated variable, attribute, and dimension names.
• Define and add required dimension variables.

However, it does not enforce broader IODA v3 format compliance. As a result, output files may pass through production unnoticed but still fail validation when checked with the official schema validation tool, ioda-validate.x.

Currently, obs2ioda does not generate any warnings or errors when files are out of compliance; it simply corrects known naming issues. While this avoids runtime failures, it can silently promote outdated conventions and result in downstream validation failures or silent scientific errors.

---

IODA Compliance Criteria

The IODA schema defines the expected structure, units, dimensions, and attributes for valid observation NetCDF files. ioda-validate.x enforces many of these rules and identifies compliance issues.

Enforced Criteria

• Required groups must exist (e.g., MetaData, ObsValue, ObsError).
• Required dimensions must exist .
• Required variables must be present in their respective groups.
• Variable names must match the schema; deprecated variables must be removed.
• Variables must have correct dimensions attached to valid dimension scales.
• Required and optional attributes must match schema expectations.
• units must be valid, convertible, and present when required.
• Variables not allowed in MetaData (canBeMetadata: false) must not appear there.

Planned (TODO) Criteria

• Value range checks (e.g., min/max bounds)
• _FillValue presence and correctness
• Chunking and compression configuration
• Variable type enforcement
• Dimension type validation

---

Current Validation Violations

When processed by ioda-validate.x, all major output files from obs2ioda (e.g., sfc_obs_2018041500.h5, sondes_obs_2018041500.h5, amsua_n15_obs_2018041500.h5, etc.) produce the same errors and warnings.

Errors

• MetaData/variable_names: This deprecated variable is still present and must be removed.

Warnings

• MetaData variables are missing required units.
• Root-level attributes are not defined in the schema.
• Several attributes exist in the file but have no corresponding YAML spec or are not marked as required/optional.

---

Planned Subissues

• Validate units attribute