Transmute script output testing coverage #755

Are there any linked Issues or Pull Requests?

Umbrella: #424
PR: #795

Brief description

Whilst Transmute script do log their output when transformations fail, it is unclear when these are genuine and new or existing.

We are moving to a space where changes to Science source, will require changes to Transmute scripts.
As we have no metadata in source, more complex scripts which target specific nodes, may begin to fail, if these nodes move in the psyir tree, when source changes occur, like adding ort removing ifblocks, calls or loops.

Also, a number of the logged transformations fails, are expected, as noted above, we just try transformations generally on all nodes in the tree, so often these logged 'errors' are in fact expected.

A combined overhaul of these is needed, either to reduce or overhaul the logging output, and or a method to capture known logging messages, and filter out new ones.

The UM already does this with https://github.com/MetOffice/um/blob/23785d7b7fc65364d4160e211f3593d5ec45a08c/rose-stem/bin/compiler_warnings.py#L89
I expect LFRic would benefit from this, and this would be a good fit for Transmute, allowing logging, but suppressing know and accepted messages.

This way, if any changes to source or PSyclone cause new messages (and thus the scripts are likely failing), we can more easily see and fix.

We could also 'kgo' the generated source with a generated vs stored hash check, but this would add steps to the build system potentially unnecessarily.

Further details of the issue.

No response