variable/*.json: root terms carry tile-specific or operation-specific long_names

[JanStreffing]

a handful of root variable/*.json entries have a long_name that's specific to one PFT tile or one daily-extreme operation, even though the term itself is the generic root. branded variants then fail the wcrp_cmip7 ATTR004 long_name registry check on real files even when the file's long_name is correct for its branding.

example, current state in variable/cveg.json:

{
    "id": "cveg",
    "long_name": "Carbon Mass in Vegetation on Grass Tiles",
    "description": "..."
}

while variable/cveggrass.json / cvegshrub.json / cvegtree.json all have "long_name": null and the per-tile text sitting in description. so a CMIP7 file with branded variable cveg_tavg-u-hxy-shb (Shrub) and long_name = "Carbon Mass in Vegetation on Shrub Tiles" (the correct text for its tile) fails the registry check because the registry says cveg.long_name = "Carbon Mass in Vegetation on Grass Tiles".

same shape affects several other root terms - all carry a long_name that belongs to a specific variant:

root	current long_name
cveg	"Carbon Mass in Vegetation on Grass Tiles"
hurs	"Daily Minimum Near-Surface Relative Humidity over Crop Tile"
tas	"Daily Minimum Near-Surface Air Temperature"
mrsol	"Mean soil water content at a depth of 1 m"
gpp	"Carbon Mass Flux out of Atmosphere Due to Gross Primary Production on Land [kgC m-2 s-1]"
npp	"Net Primary Production on Grass Tiles as Carbon Mass Flux [kgC m-2 s-1]"
ra	"Autotrophic Respiration on Shrub Tiles as Carbon Mass Flux [kgC m-2 s-1]"
rh	"Heterotrophic Respiration on Shrub Tiles as Carbon Mass Flux [kgC m-2 s-1]"

proposed fix on the registry side:

1. root term long_name becomes the generic form (e.g. cveg.long_name = "Carbon Mass in Vegetation"), or is set to null like the other generic descriptors.
2. tile/operation variants (cveggrass, cvegshrub, cvegtree, hursmin, hursmax, tasmin, tasmax, etc.) get the specific long_name populated - usually the text already sitting in their description field.
3. for cases where there's no separate variant term (e.g. gpp is itself the root), generalize the long_name so it doesn't carry a tile token.

related plugin issue: cc-plugin-wcrp's variable long_name lookup uses variable_id.lower() (the root term) rather than the branded-variant id, so even after the registry is split per variant, the plugin will need to look up the variant first. happy to file a sister PR upstream once this side is settled.

observed against esgvoc cmip7@1.2.6 / WCRP-universe (registry-source DB).

[taylor13]

Thanks @JanStreffing for clearly identifying the problem and offering reasonable solutions.

I'm surprised that QC/QA should raise an error. The guidance for long_name found in this document states:

long_name = a character string that might serve as a title for the variable on plots (e.g., "Surface Air Temperature"). Suggested "long names" can be found in the CMOR tables (long_name key) and in the "Title" column of the data request's variables listing. A few long_name(s) in the data request listing are inconsistent with those specified in the CMOR tables, so these names are not fully standardized in CMIP7.

Thus, the long_name is not required to be consistent with the data request (although that is encouraged). Perhaps @glevava and @sashakames can modify the QC/QA to issue a warning rather than an error?

[JanStreffing]

Thanks @taylor13. One clarification: can a file with 0 HIGH but N MEDIUM findings be published to ESGF as-is, or does AFT acceptance also require MEDIUM to clear? I'd been driving every MEDIUM to zero before submitting; if that's not required, #191 / #194 stay nice cleanup but stop being blockers.

[sashakames]

MEDIUM does not affect publishing. Is the compliance-check output HIGH, if so that will block esgpublish.

[JanStreffing]

Good to know. I will correct all issues where I had claimed medium issues are blocking publication.

[glevava]

Hi @JanStreffing
Although updating the long_name values makes sense, this cannot be done directly in the CV Universe because this information must remain synchronized with the Data Request. That said, the need originates from the fact that the long_name values (which are actually already present in the Data Request and distinguish tile-specific or operation-specific variants) were excluded from the known_branded_variables because they were not originally considered to be “standard” and in some edge-cases one branded variable could have several long_name (the relationship is not 1:1 for all DR entry).

However, even if their validation through QA/QC with a MEDIUM severity level does not block publication, we believe it would be more appropriate to reintroduce them.

As a consequence, we are currently working on a new Pydantic model for the known_branded_variables, which will extend the current model by adding information such as long_name, with the objective of enabling the necessary validations on the QA/QC side.

The goal is for the known_branded_variables data descriptor in the CV to include all the information required by downstream tools involved in ESGF publication.

As a result, this PR will no longer be necessary, but it has helped highlight the need and ultimately improve the structure of the CV — so thank you again!

[JanStreffing]

@glevava thanks for the plan on adding long_name to the known_branded_variables Pydantic model. Rough timeline for when that lands? I'm trying to figure out whether to keep #191 open as an interim fix or close it and wait for the model work.

[glevava]

End of the month it will be released.

[JanStreffing]

Ty!

[taylor13]

To be sure are these statements correct:

1. Currently the QC/QA assigns a single long_name to each root name, which is at odds with the data request.
2. You plan, instead, to assign a single long_name to each branded_variable name, which will reduce the number of warnings raised if the data applies the long_name foundin the data request.
3. There may still be some warnings issued when a different long_name is applied to a requested variable depending on the frequency or region it is requested for. (I think there are a few variables like this in the data request, which have been discussed before.)

thanks,
Karl

[glevava]

Here are some details about the plan.

1. I would like to recall once again that, regarding long_name, and in line with what was agreed with the CMIP-IPO and the WIP, the QA/QC reports an error with severity MEDIUM, which corresponds to what we call a WARNING. It is considered non-blocking and it is left to users to decide whether or not they want to align with the Data Request.
2. The cause of this warning being raised is probably that long_name is currently available in the CV through the variable data descriptor, and not through the known_branded_variable data descriptor, from which they were unfortunately removed. There are therefore probably long_name in this data descriptor that are no longer synchronised with the Data Request, in particular for terms carrying tile-specific information, which have since been corrected in the DR.
3. The QA/QC does not assign anything. It only checks information against the CV, via esgvoc. Until now, the checked long_name has therefore been the one assigned in the variable data descriptor. This explains the PR proposed by @JanStreffing which aligns the long_name in CV with the DR. But this should not be necessary, and points, at best, to a weakness in the system, and at worst to the consequence of the fact that, from the beginning, we have been managing two separate sources for very closely related information: the CVs and the DR.

That's why I have proposed here b4c9a0a#top to modify the Pydantic model of the known_branded_variable data descriptor, after first checking with several power users that this would not affect their use cases, in particular the generation of the CMIP7 CMOR tables. Some of the changes I propose are the following:

• Reintroduce long_name into the model, using the value provided by the DR. This way, tile-specific long names are natively supported. Even if some inconsistencies remain in the DR long name, we need to keep the sources of information aligned with each other; otherwise, the QA/QC has no real purpose.
• Add coordinates to the model, again based on the DR. I have therefore added a coordinate data descriptor, and the branded variables now include pointers to the coordinates they use, based on the spatial and temporal shapes from the DR. This will allow the QA/QC to check that the file coordinates match what is expected by the DR, and what CMOR or XIOS is supposed to have produced, without duplicating this information on the QA/QC side.
• Convert some attributes into lists of strings, so that they can also aggregate information from the DR in cases where there are duplicate entries for the same branded name. This applies to cell_methods, cell_measures, realm, region, and description. Once all inconsistencies have been resolved in the DR over time, with a view to producing the Variable Registry, we will be able to easily synchronise the known_branded_variable in the CV and remove the lists if the relationships become 1:1 again between DR entries and branded names.
• Add the optional attributes compound_name, table_id, and frequency, which are used only at project-CV level — this information is therefore not in the Universe — in order to map the old CMIP6 or CORDEX variable identifiers to the corresponding branded variable. This allows the QA/QC to also support legacy projects without having to maintain a mapping table on its side, which would create a third source of information, even though this information belongs in the CV, whose structure was designed for this purpose.

In this way, without changing the current model too much, the known_branded_variable data descriptor contains all the necessary variable-related information, in agreement with the DR. If a value is incorrect, it should first be corrected in the Data Request and/or in the future Variable Registry. esgvoc is only a structural translation of this information, allowing downstream tools, through the known_branded_variable data descriptor, to access all the information through a single interface.