Add xml to yaml converter, update Markdown and Yaml files automatically when pull requests are merged #117

climbfuji · 2025-06-13T15:08:50Z

Description

This pull request adds the capability to render the XML standard names dictionary to Yaml, in addition to the existing rendering to Markdown. I decided to render comments into long strings that the Python Yaml library then wraps correctly. I validated the resulting Yaml file using https://yamlchecker.com and it is valid.

This pull request further updates the existing GitHub action for consistency and standard Yaml formatting (2-space indentation), and replaces the check if the Markdown is updated with tests to render the xml to both md and yaml.

Lastly, a second GitHub workflow is added that automatically renders the xml to md and yaml and commits the files to the repository. This action is only run when the authoritative branch (main) is updated (push = merge a pull request). I tested this action in my fork of ESM standard names (see https://github.com/climbfuji/CCPPStandardNames/actions/runs/15637497788/job/44056347886). What I did not test is how this works with branch protection (my main branch was not protected). We will find out if additional settings/changes are needed once this PR is approved and merged.

Closes #113

YAML; still todo: comment formatting

…atted

…argument

…d yaml

…into feature/xml2yaml

climbfuji · 2025-06-13T15:13:07Z

Metadata-standard-names.md

@@ -1,4 +1,4 @@
-# Earth System Modeling Standard Name Library
+# Earth System Modeling Standard Name Library - add some chunk here to test auto update


Ignore this - to prove that the auto update works. The file will be re-rendered from the XML and submitted to the repository when this PR is merged.

climbfuji · 2025-06-13T15:13:47Z

Metadata-standard-names.yaml

@@ -0,0 +1,5689 @@
+library_name: Earth System Modeling Standard Name Library - add some chunk here to test auto update


Ignore this ("- add some chunk here to test auto update") - to prove that the auto update works. The file will be re-rendered from the XML and submitted to the repository when this PR is merged.

tools/write_standard_name_table.py

climbfuji · 2025-06-13T15:15:45Z

tools/write_standard_name_table.py

-            if stdn_longname is None:
-                sdict = {'standard_name':stdn_name}
-                stdn_longname = standard_name_to_long_name(sdict)
+    # Step through the sections


No changes to parse_section other than adjusting the indentation to 4 whitespaces

gold2718

One question and not the most thorough review but this looks okay to me.

gold2718 · 2025-06-14T01:38:20Z

.github/workflows/commit_to_main.yml

+          echo "The following changes will be committed (git diff Metadata-standard-names.yaml):"
+          git diff Metadata-standard-names.yaml
+          git add Metadata-standard-names.yaml
+          git commit -m "Update Metadata-standard-names.yaml from standard_names.xml" || echo "No changes to commit"


It seems likely that if the markdown is unchanged then YAML is also unhanged. Why not write both files, then run git?

True. A little bit of reorg, but happy to make the change.

Done in 9097066

nusbaume

Looks good to me, but did have a couple questions and one (optional) request.

.github/workflows/pull_request_ci.yml

tools/write_standard_name_table.py

…ack in, add release/* to branches

nusbaume

Looks great to me now. Thanks!

dustinswales

@climbfuji This is really slick!

climbfuji · 2025-06-30T15:55:11Z

@mkavulich Shall we go ahead and merge? If we run into trouble with the github bot not being able to commit the updated xml and yamls, then I can look into alternative options before the next meeting.

mkavulich · 2025-06-30T17:04:03Z

@climbfuji It occurs to me that we can ensure the actions will work before merging by adding the "workflow_dispatch" option and kicking off the file creation manually; is there any downside to including this manual option for the new GitHub action?

climbfuji · 2025-06-30T20:22:59Z

@climbfuji It occurs to me that we can ensure the actions will work before merging by adding the "workflow_dispatch" option and kicking off the file creation manually; is there any downside to including this manual option for the new GitHub action?

The code lives in a branch in my fork and you can't trigger a workflow from a branch in a different fork.

mkavulich · 2025-07-01T15:30:10Z

Thanks for the explanation. I will go ahead and merge then, and keep an eye on the result.

mkavulich · 2025-07-01T15:42:59Z

@climbfuji The workflow is failing due to branch protection rules that prevent direct pushes to main. It appears as if there are options available that do not involve removing branch protections, so I am looking into those.

climbfuji · 2025-07-01T17:00:14Z

@climbfuji The workflow is failing due to branch protection rules that prevent direct pushes to main. It appears as if there are options available that do not involve removing branch protections, so I am looking into those.

Thanks @mkavulich . Happy to work with you on that. I was thinking that we could also add the commit to the PR action itself, but trigger it only when a label is added. If we commit the updated files every time a developer pushes a commit, the developers have to pull the updates into their local clone (annoying). The downside of the label solution is that we have to remember adding the label to run the action before we merge. But maybe a big and fat check box at the top of the PR template could help with that.

Because it is a bit of a hassle for PR authors to have to make changes to multiple files, #117 attempted to automate the process using the `tools/write_standard_name_table.py` tool. However, this new GitHub action could not run automatically due to branch protection rules on the `main` branch. Rather than remove those restrictions, this PR reverts to the old method of manual updates, but provides more extensive documentation and user prompts to do this updating properly.

Add xml to yaml converter, update Markdown and Yaml files automatically when pull requests are merged (ESCOMP#117) This pull request adds the capability to render the XML standard names dictionary to Yaml, in addition to the existing rendering to Markdown. I decided to render comments into long strings that the Python Yaml library then wraps correctly. I validated the resulting Yaml file using https://yamlchecker.com and it is valid. This pull request further updates the existing GitHub action for consistency and standard Yaml formatting (2-space indentation), and replaces the check if the Markdown is updated with tests to render the xml to both md and yaml. Lastly, a second GitHub workflow is added that automatically renders the xml to md and yaml and commits the files to the repository. This action is only run when the authoritative branch (main) is updated (push = merge a pull request). I tested this action in my fork of ESM standard names (see https://github.com/climbfuji/CCPPStandardNames/actions/runs/15637497788/job/44056347886). What I did not test is how this works with branch protection (my main branch was not protected). We will find out if additional settings/changes are needed once this PR is approved and merged. Closes ESCOMP#113

New procedure for updating .md and .yaml files (ESCOMP#122) Because it is a bit of a hassle for PR authors to have to make changes to multiple files, ESCOMP#117 attempted to automate the process using the `tools/write_standard_name_table.py` tool. However, this new GitHub action could not run automatically due to branch protection rules on the `main` branch. Rather than remove those restrictions, this PR reverts to the old method of manual updates, but provides more extensive documentation and user prompts to do this updating properly.

climbfuji and others added 11 commits June 12, 2025 20:52

Update tools/write_standard_name_table.py: linting, basic output to

a8f1110

YAML; still todo: comment formatting

Update tools/write_standard_name_table.py: comments now properly form…

fb2ab9e

…atted

Remove Metadata-standard-names.md

265773e

Add Metadata-standard-names.yaml and Metadata-standard-names.md

8557580

In tools/write_standard_name_table.py, make output format a required …

6f915bf

…argument

Update .github/workflows/pull_request_ci.yml: test rendering to md an…

e743db1

…d yaml

Update Metadata-standard-names.md

7216533

Add .github/workflows/commit_to_main.yml

1e7b451

Merge branch 'main' of https://github.com/climbfuji/ccppstandardnames …

9354ba3

…into feature/xml2yaml

Add comments to GitHub action workflows that explain maining of git diff

fcb7a39

Clean up .github/workflows/commit_to_main.yml and update README.md

7c613d0

climbfuji requested review from cacraigucar, dustinswales, gold2718, grantfirl, mattldawson, mkavulich, mwaxmonsky, nusbaume, peverwhee, ss421 and svahl991 as code owners June 13, 2025 15:08

climbfuji commented Jun 13, 2025

View reviewed changes

Update tools/write_standard_name_table.py

653c8d2

climbfuji self-assigned this Jun 13, 2025

climbfuji mentioned this pull request Jun 13, 2025

Support a yaml-based format for Standard Names dictionary #113

Closed

gold2718 approved these changes Jun 14, 2025

View reviewed changes

Only commit once from GitHub actions, just before push

9097066

nusbaume reviewed Jun 22, 2025

View reviewed changes

.github/workflows/pull_request_ci.yml Show resolved Hide resolved

tools/write_standard_name_table.py Show resolved Hide resolved

Update .github/workflows/pull_request_ci.yml: add workflow_dispatch b…

97c9736

…ack in, add release/* to branches

nusbaume approved these changes Jun 28, 2025

View reviewed changes

dustinswales approved these changes Jun 30, 2025

View reviewed changes

mkavulich merged commit 40158a8 into ESCOMP:main Jul 1, 2025
3 checks passed

climbfuji deleted the feature/xml2yaml branch July 1, 2025 16:58

climbfuji mentioned this pull request Jul 9, 2025

Fix GitHub action that submits the updated rendered files when a PR is merged to main #121

Closed

mkavulich mentioned this pull request Jul 17, 2025

New procedure for updating .md and .yaml files #122

Merged

		@@ -1,4 +1,4 @@
		# Earth System Modeling Standard Name Library
		# Earth System Modeling Standard Name Library - add some chunk here to test auto update

		@@ -0,0 +1,5689 @@
		library_name: Earth System Modeling Standard Name Library - add some chunk here to test auto update

Add xml to yaml converter, update Markdown and Yaml files automatically when pull requests are merged #117

Add xml to yaml converter, update Markdown and Yaml files automatically when pull requests are merged #117

Uh oh!

Conversation

climbfuji commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

climbfuji Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

climbfuji Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

climbfuji Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

gold2718 left a comment

Choose a reason for hiding this comment

Uh oh!

gold2718 Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

climbfuji Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

climbfuji Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nusbaume left a comment

Choose a reason for hiding this comment

Uh oh!

dustinswales left a comment

Choose a reason for hiding this comment

Uh oh!

climbfuji commented Jun 30, 2025

Uh oh!

mkavulich commented Jun 30, 2025

Uh oh!

climbfuji commented Jun 30, 2025

Uh oh!

mkavulich commented Jul 1, 2025

Uh oh!

Uh oh!

mkavulich commented Jul 1, 2025

Uh oh!

climbfuji commented Jul 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

climbfuji commented Jun 13, 2025 •

edited

Loading