From 5d971434435b7287583913856743a45d3fe86c20 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Tue, 5 Dec 2023 17:28:14 -0700 Subject: [PATCH 01/21] Updating doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst for mksurfdata_esmf --- .../using-clm-tools/what-are-the-clm-tools.rst | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst index dd650a79e9..a1ebe87101 100644 --- a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst +++ b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst @@ -14,19 +14,15 @@ The list of generally important scripts and programs are as follows. 1. *./mkmapgrids* to create SCRIP grid data files from old CLM format grid files that can then be used to create new CLM datasets (deprecated). There is also a NCL script (``./mkmapgrids/mkscripgrid.ncl`` to create SCRIP grid files for regular latitude/longitude grids. -#. *./mkmapdata* to create SCRIP mapping data file from SCRIP grid files (uses ESMF). - -#. *mksurfdata_map* to create surface datasets from grid datasets (clm4_0 and |version| versions). +#. *mksurfdata_esmf* to create surface datasets from grid datasets that we refer to as raw datasets (ctsm5_2 and newer versions). #. *./mkprocdata_map* to interpolate output unstructured grids (such as the CAM HOMME dy-core "ne" grids like ne30np4) into a 2D regular lat/long grid format that can be plotted easily. Can be used by either clm4_0 or |version|. -#. *$CIMEROOT/tools/mapping/gen_domain_files/gen_domain* to create a domain file for datm from a mapping file. The domain file is then used by BOTH datm AND CLM to define the grid and land-mask. - #. *$CIMEROOT/tools/cprnc* to compare two NetCDF files. In the sections to come we will go into detailed description of how to use each of these tools in turn. First, however we will discuss the common environment variables and options that are used by all of the FORTRAN tools. Second, we go over the outline of the entire file creation process for all input files needed by CLM for a new resolution, then we turn to each tool. In the last section we will discuss how to customize files for particular observational sites. -The FORTRAN tools (mksurfdata_map and mkprocdata_map) run, with a namelist (mksurfdata_map) to provide options, or with command line arguments (mkprocdata_map). +The FORTRAN tools (mksurfdata_esmf and mkprocdata_map) run with a namelist (mksurfdata_map) to provide options or with command line arguments (mkprocdata_map). In the following sections, we will outline how to make these files available for build-namelist so that you can easily create simulations that include them. In the chapter on single-point and regional datasets we also give an alternative way to enter new datasets without having to edit files. @@ -34,23 +30,19 @@ In the following sections, we will outline how to make these files available for Running FORTRAN tools with namelists ------------------------------------ -**mksurfdata_map** runs with a namelist that is read from standard input. Hence, you create a namelist and then run them by redirecting the namelist file into standard input as follows: +**mksurfdata_esmf** runs with a namelist that is read from standard input. First you create a namelist, then you create a jobscript that runs mksurfdata_esmf by redirecting the namelist file into standard input as follows: :: ./program < namelist -There is a sample namelist called ``$CTSMROOT/tools/mksurfdata_map/mksurfdata_map.namleist`` that shows you what the namelist should look like. **mksurfdata_map** also has a script that creates the namelist and runs the program for you. Namelists that you create should be similar to the example namelist. The namelist values are also documented along with the other namelists in the: +There is a tool that generates the namelist called ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_namelist.py``. The namelist contains information gathered from the file ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_namelist.xml``. There is also a tool that generates a jobscript for running, and this is called ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_jobscript_single.py``. :: - $CTSMROOT/bld/namelist_files/namelist_definition.xml`` file - and default values in the: - $CTSMROOT/bld/namelist_files/namelist_defaults_clm_tools.xml`` file. - ----------------------------------------------- Running FORTRAN tools with command line options ----------------------------------------------- -**gen_domain**, mkprocdata_map, and **cprnc** run with command line arguments. The detailed sections below will give you more information on the command line arguments specific to each tool. Also running the tool without any arguments will give you a general synopsis on how to run the tool. +mkprocdata_map, and **cprnc** run with command line arguments. The detailed sections below will give you more information on the command line arguments specific to each tool. Also running the tool without any arguments will give you a general synopsis on how to run the tool. ----------------------------------------- Running FORTRAN tools built with SMP=TRUE From 42098c5f864fae168a1a1c9cf88a01fa0e7efd27 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 29 May 2025 15:23:36 -0700 Subject: [PATCH 02/21] Draft update documentation for generating fsurdat/landuse files --- .../creating-surface-datasets.rst | 216 +++++++++++++++--- 1 file changed, 189 insertions(+), 27 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst index 8394f17b3f..3d7d75387d 100644 --- a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst +++ b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst @@ -6,55 +6,217 @@ Creating Surface Datasets =========================== -When just creating a replacement file for an existing one, the relevant tool should be used directly to create the file. When you are creating a set of files for a new resolution there are some dependencies between the tools that you need to keep in mind when creating them. The main dependency is that you MUST create a SCRIP grid file first as the SCRIP grid dataset is then input into the other tools. Also look at Table :numref:`reqd-files-table` which gives information on the files required and when. :numref:`Figure Data_Flow` shows an overview of the general data-flow for creation of the fsurdat datasets. + +## mksurfdata_esmf Purpose + +This tool is intended to generate fsurdat files (surface datasets) and landuse files for the +CTSM. It can generate global, regional, and single-point fsurdat files, as long +as a mesh file is available for the grid. -.. _Figure Data_Flow: +The subset_data tool allows users to make fsurdat files from existing fsurdat +files when a mesh file is unavailable. Generally, users are encouraged to use the +subset_data tool for generating regional and single-point fsurdat files (see note about regional grids at the end). -.. figure:: mkmapdata_mksurfdata.jpeg + +## Building + - Data Flow for Creation of Surface Datasets from Raw SCRIP Grid Files + +### Build Requirements + -Starting from a SCRIP grid file that describes the grid you will run the model on, you first run ```mkmapdata.sh`` to create a list of mapping files. See :numref:`Figure mkmapdata.sh` for a more detailed view of how ``mkmapdata.sh`` works. The mapping files tell ``mksurfdata_esmf`` how to map between the output grid and the raw datasets that it uses as input. The output of ``mksurfdata_esmf`` is a surface dataset that you then use for running the model. See :numref:`Figure Workflow of CLM5 Land Use Data Tool and mksurfdata_esmf Tool` for a more detailed view of how ``mksurfdata_esmf`` works. +mksurfdata_esmf is a distributed memory parallel program (using Message Passing +Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) +for regridding as well as PIO (Parallel I/O) and NetCDF output. As +such, libraries must be built for the following: -:numref:`Figure Data_Flow_Legend` is the legend for this figure (:numref:`Figure Data_Flow`) and other figures in this chapter (:numref:`Figure Global-Domain` and :numref:`Figure mknoocnmap.pl`). +1. MPI +2. NetCDF +3. PIO +4. ESMF -.. _Figure Data_Flow_Legend: +In addition for the build: python, bash-shell, CMake and GNU-Make are required -.. figure:: LegendCLMToolDataFlow.jpeg +These libraries need to be built such that they can all work together in the +same executable. Hence, the above order may be required in building them. - Legend for Data Flow Figures +CTSM submodules that are required are: cime and ccs_config. See [Building](#building-the-executable) on getting +those. A python environment that includes particular packages is also required +we demonstrate how to use the ctsm_pylib environment that we support in CTSM. -Green arrows define the input to a program, while red arrows define the output. Cylinders define files that are either created by a program or used as input for a program. Boxes are programs. +Note, PNETCDF is an optional library that can be used, but is NOT required. -You start with a description of a SCRIP grid file for your output grid file and then create mapping files from the raw datasets to it. Once, the mapping files are created ``mksurfdata_esmf`` is run to create the surface dataset to run the model. +#### Use cime to manage the build requirements -Creating a Complete Set of Files for Input to CLM -------------------------------------------------- +See [IMPORTANT NOTE](important note-only-working-on-derecho-currently) -1. Create SCRIP grid datasets (if NOT already done) +For users working on cime machines you can use the build script to build the +tool. On other machines you'll need to do a port to cime and tell how to build +for that machine. That's talked about in the cime documentation. +And you'll have to make some modifications to the build script. - First you need to create a descriptor file for your grid, that includes the locations of cell centers and cell corners. There is also a "mask" field, but in this case the mask is set to one everywhere (i.e. all of the masks for the output model grid are "nomask"). An example SCRIP grid file is: ``$CSMDATA/lnd/clm2/mappingdata/grids/SCRIPgrid_10x15_nomask_c110308.nc``. The ``mkmapgrids`` and ``mkscripgrid.ncl`` NCL script in the ``$CTSMROOT/tools/mkmapgrids`` directory can help you with this. SCRIP grid files for all the standard CLM grids are already created for you. See the Section called Creating an output SCRIP grid file at a resolution to run the model on for more information on this. +https://github.com/ESMCI/cime/wiki/Porting-Overview -.. todo:: - Update the below, as domain files aren't needed with nuopc. +Machines that already run CTSM or CESM have been ported to cime. So if you can +run the model on your machine, you will be able to build the tool there. -2. Create domain dataset (if NOT already done) +To get a list of the machines that have been ported to cime: - Next use ``gen_domain`` to create a domain file for use by DATM and CLM. This is required, unless a domain file was already created. See the Section called Creating a domain file for CLM and DATM for more information on this. +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory +cd ../../cime/scripts # or ../../../../cime/scripts for a CESM checkout +./query_config --machines +``` -3. Create mapping files for ``mksurfdata_esmf`` (if NOT already done) +#### NOTE: +In addition to having a port to cime, the machine also needs to have PIO built +and able to be referenced with the env variable PIO which will need to be in +the porting instructions for the machine. An independent PIO library +is available on supported CESM machines. - Create mapping files for ``mksurfdata_esmf`` with ``mkmapdata.sh`` in ``$CTSMROOT/tools/mkmapdata``. See the Section called Creating mapping files that ``mksurfdata_esmf`` will use for more information on this. + +#### IMPORTANT NOTE: ONLY WORKING ON DERECHO CURRENTLY + -4. Create surface datasets - Next use ``mksurfdata_esmf`` to create a surface dataset, using the mapping datasets created on the previous step as input. There is a version for either clm4_0 or |version| for this program. See the Section called Using ``mksurfdata_esmf`` to create surface datasets from grid datasets for more information on this. +> [!IMPORTANT] +> Currently we have run and tested mksurfdata_esmf on Derecho. Please see this github issue about mksurfdata_esmf on other CESM machines: -5. Enter the new datasets into the ``build-namelist`` XML database - The last optional thing to do is to enter the new datasets into the ``build-namelist`` XML database. See Chapter 3 for more information on doing this. This is optional because the user may enter these files into their namelists manually. The advantage of entering them into the database is so that they automatically come up when you create new cases. +https://github.com/ESCOMP/CTSM/issues/2341 -The ``$CTSMROOT/tools/README`` goes through the complete process for creating input files needed to run CLM. We repeat that file here: + +### Building the executable + -.. include:: ../../../../tools/README + Before starting, be sure that you have run + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./bin/git-fleximod update # Assuming at the top level of the CTSM/CESM checkout +``` + +This will bring in CIME and ccs_config which are required for building. + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./gen_mksurfdata_build # For machines with a cime build +``` + + Note: The pio_iotype value gets set and written to a simple .txt file + by this build script. The value depends on your machine. If not running + on derecho, casper, or izumi, you may need to update this, though + a default value does get set for other machines. + + +## Running for a single submission + + +### Setup ctsm_pylib + Work in the ctsm_pylib environment, which requires the following steps when + running on Derecho. On other machines it will be similar but might be different + in order to get conda in your path and activate the ctsm_pylib environment. + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + module load conda + cd ../.. # or ../../../.. for a CESM checkout) + ./py_env_create # Assuming at the top level of the CTSM/CESM checkout + conda activate ctsm_pylib +``` + +to generate your target namelist: + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./gen_mksurfdata_namelist --help +``` + +for example try --res 1.9x2.5 --start-year 1850 --end-year 1850: + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./gen_mksurfdata_namelist --res --start-year --end-year +``` + +> [!TIP] +> **IF FILES ARE MISSING FROM** /inputdata, a target namelist will be generated +> but with a generic name and with warning to run `./download_input_data` next. +> **IF A SMALLER SET OF FILES IS STILL MISSING AFTER RUNNING** `./download_input_data` +> and rerunning `./gen_mksurfdata_namelist`, then rerun +> `./gen_mksurfdata_namelist with your options needed. +> and rerun `./download_input_data` until +> `./gen_mksurfdata_namelist` finds all files. + + Example, to generate your target jobscript (again use --help for instructions): + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./gen_mksurfdata_jobscript_single --number-of-nodes 2 --tasks-per-node 128 --namelist-file target.namelist + qsub mksurfdata_jobscript_single.sh +``` + + Read note about regional grids at the end. + + +## Running for the generation of multiple datasets + + Work in the ctsm_pylib environment, as explained in earlier section. + gen_mksurfdata_jobscript_multi runs `./gen_mksurfdata_namelist` for you + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + ./gen_mksurfdata_jobscript_multi --number-of-nodes 2 --scenario global-present + qsub mksurfdata_jobscript_multi.sh +``` + + If you are looking to generate all (or a large number of) the datasets or the + single-point (1x1) datasets, you are best off using the Makefile. For example + +``` shell +# Assuming pwd is the tools/mksurfdata_esmf directory + make all # ...or + make all-subset +``` + + As of 2024/9/12 one needs to generate NEON and PLUMBER2 fsurdat files by + running ./neon_surf_wrapper and ./plumber2_surf_wrapper manually in the + /tools/site_and_regional directory. + + +## NOTES + + +# Guidelines for input datasets to mksurfdata_esmf + +> [!TIP] +> ALL raw datasets \*.nc **FILES MUST NOT BE NetCDF4**. + +Example to convert to CDF5 + +``` shell +nccopy -k cdf5 oldfile newfile +``` + +> [!TIP] +> The LAI raw dataset \*.nc **FILE MUST HAVE** an "unlimited" time dimension + +Example to change time to unlimted dimension using the NCO operator ncks. + +``` shell +ncks --mk_rec_dmn time file_with_time_equals_12.nc -o file_with_time_unlimited.nc +``` + +### IMPORTANT KNOWN PROBLEMS + +- See github issue https://github.com/ESCOMP/CTSM/issues/3141 +- In general we recommend using subset_data and/or fsurdat_modifier for regional grids + + +## FINAL COMMENT + + +The ``$CTSMROOT/tools/mksurfdata_esmf/README.md`` documents the same instructions for creating fsurdat/landuse files needed to run CLM. We repeat that file here: + +.. include:: ../../../../tools/mksurfdata_esmf/README.md :literal: From 36839798e3993e6ad154947a31413f82a2f7dd3a Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Mon, 23 Mar 2026 17:43:40 -0600 Subject: [PATCH 03/21] Remove creating-domain-files.rst as not needed with nuopc --- .../using-clm-tools/creating-domain-files.rst | 40 ------------------- 1 file changed, 40 deletions(-) delete mode 100644 doc/source/users_guide/using-clm-tools/creating-domain-files.rst diff --git a/doc/source/users_guide/using-clm-tools/creating-domain-files.rst b/doc/source/users_guide/using-clm-tools/creating-domain-files.rst deleted file mode 100644 index 90814e1927..0000000000 --- a/doc/source/users_guide/using-clm-tools/creating-domain-files.rst +++ /dev/null @@ -1,40 +0,0 @@ -.. include:: ../substitutions.rst - -.. _creating-domain-files: - -***************************** - Creating CLM domain files -***************************** - -.. todo:: - Delete this page? Domain files aren't needed with nuopc. - -``gen_domain`` to create a domain file for datm from a mapping file. ``gen_domain`` is a tool that is a part of CIME. The domain file is then used by BOTH DATM AND CLM to define the grid and land-mask. The general data flow is shown in two figures. :numref:`Figure mkmapdata.sh` shows the general flow for a general global case (or for a regional grid that DOES include ocean). :numref:`Figure mknoocnmap.pl` shows the use of ``mknoocnmap.pl`` (see :ref:`using-mkocnmap`) to create a regional or single-point map file that is then run through ``gen_domain`` to create the domain file for it. As stated before :numref:`Figure Data_Flow_Legend` is the legend for both of these figures. See `the $CIMEROOT/tools/mapping/gen_domain_files/README `_ file for more help on ``gen_domain``. - -Here we create domain files for a regular global domain. - -Global Domain file creation -=========================== - -.. _Figure Global-Domain: - -.. figure:: GlobalDomain.jpeg - - Global Domain file creation - -Starting from SCRIP grid files for both your atmosphere and ocean, you use ``$CIMEROOT/tools/mapping/gen_mapping_files/gen_cesm_maps.sh`` to create a mapping file between the atmosphere and ocean. That mapping file is then used as input to ``gen_domain`` to create output domain files for both atmosphere and ocean. The atmosphere domain file is then used by both CLM and DATM for I compsets, while the ocean domain file is ignored. For this process you have to define your SCRIP grid files on your own. For a regional or single-point case that doesn't include ocean see :numref:`Figure mknoocnmap.pl`. (See :numref:`Figure Global-Domain` for the legend for this figure.) - -Note that the SCRIP grid file used to start this process is also used in ``mkmapdata.sh`` (see :ref:`using-mkocnmap`). Next we create domain files for a single-point or regional domain. - -Domain file creation using mknoocnmap.pl -======================================== - -.. _Figure mknoocnmap.pl: - -.. figure:: mknoocnmap.jpeg - - Domain file creation using mknoocnmap.pl - -For a regular latitude/longitude grid that can be used for regional or single point simulations -- you can use ``mknoocnmap.pl``. It creates a SCRIP grid file that can then be used as input to ``mkmapdata.sh`` as well as a SCRIP mapping file that is then input to ``gen_domain``. The output of ``gen_domain`` is a atmosphere domain file used by both CLM and DATM and a ocean domain file that is ignored. (See :numref:`Figure mknoocnmap.pl` for the legend for this figure.) - -In this case the process creates both SCRIP grid files to be used by ``mkmapdata.sh`` as well as the domain files that will be used by both CLM and DATM. From 0f16d7d0922ffd0aae4bf59f7f138cc930b45b9e Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Mon, 23 Mar 2026 17:50:51 -0600 Subject: [PATCH 04/21] Remove references to creating-domain-files --- .../setting-ctsm-runtime-options.rst | 2 +- doc/source/users_guide/using-clm-tools/index.rst | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst b/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst index acb1cad9be..eea7c74db8 100644 --- a/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst +++ b/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst @@ -106,7 +106,7 @@ The first set of options in this file specifies key file names: - ``lnd_domain_file`` must be specified. This file specifies CTSM's grid and land mask. The general process for creating this file is described in section - :numref:`creating-domain-files`. + creating-domain-files (deprecated). - ``fsurdat`` also must be specified. This file specifies a variety of spatially-varying properties. This file is grid-specific, but can be created from grid-independent files diff --git a/doc/source/users_guide/using-clm-tools/index.rst b/doc/source/users_guide/using-clm-tools/index.rst index b721b3e7d6..9dbd00fe32 100644 --- a/doc/source/users_guide/using-clm-tools/index.rst +++ b/doc/source/users_guide/using-clm-tools/index.rst @@ -19,7 +19,6 @@ Using CLM tools using-ctsm-pylib.rst creating-input-for-surface-dataset-generation.rst creating-surface-datasets.rst - creating-domain-files.rst observational-sites-datasets.rst cprnc.rst paramfile-tools.md From 9e0062265bec84828c5095185f9cefcafa88928b Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Mon, 23 Mar 2026 18:02:11 -0600 Subject: [PATCH 05/21] Remove reference to mknoocnmap.pl as obsolete --- ...g-input-for-surface-dataset-generation.rst | 29 ------------------- 1 file changed, 29 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst index 66a65adbde..6a0aaf86e3 100644 --- a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst +++ b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst @@ -13,35 +13,6 @@ The utility ``mkmapdata.sh`` requires SCRIP format input files to describe the i SCRIP grid files for all the standard model resolutions and the raw surface datasets have already been done and the files are in the XML database. Hence, this step doesn't need to be done -- EXCEPT WHEN YOU ARE CREATING YOUR OWN GRIDS. -.. _using-mkocnmap: - -Using mknocnmap.pl to create grid and maps for single-point regional grids --------------------------------------------------------------------------- - -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -If you want to create a regular latitude/longitude single-point or regional grid, we suggest you use ``mknoocnmap.pl`` in ``$CTSMROOT/tools/mkmapdata`` which will create both the SCRIP grid file you need (using ``$CTSMROOT/tools/mkmapgrids/mkscripgrid.ncl``) AND an identity mapping file assuming there is NO ocean in your grid domain. If you HAVE ocean in your domain you could modify the mask in the SCRIP grid file for ocean, and then use ``ESMF_RegridWeightGen`` to create the mapping file, and ``gen_domain`` to create the domain file. Like other tools, ``./mkmapdata/mknoocnmap.pl`` has a help option with the following: -:: - - SYNOPSIS - mknoocnmap.pl [options] Gets map and grid files for a single land-only point. - REQUIRED OPTIONS - -centerpoint [or -p] Center latitude,longitude of the grid to create. - -name [-or -n] Name to use to describe point - - OPTIONS - -dx Size of total grid in degrees in longitude direction - (default is 0.1) - -dy Size of total grid in degrees in latitude direction - (default is 0.1) - -silent [or -s] Make output silent - -help [or -h] Print usage to STDOUT. - -verbose [or -v] Make output more verbose. - -nx Number of longitudes (default is 1) - -ny Number of latitudes (default is 1) - -See :numref:`Figure mknoocnmap.pl` for a visual representation of this process. Creating mapping files for mksurfdata_esmf ============================================== From 27c3f3ead8dece08526a2a7af72fcde4bf460c61 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 2 Apr 2026 17:31:23 -0600 Subject: [PATCH 06/21] Rm mention of mapping files, replace SCRIP and corresp. tool with mesh --- ...g-input-for-surface-dataset-generation.rst | 72 +------ tools/mkmapgrids/README | 22 -- tools/mkmapgrids/mkscripgrid.ncl | 188 ------------------ 3 files changed, 3 insertions(+), 279 deletions(-) delete mode 100644 tools/mkmapgrids/README delete mode 100644 tools/mkmapgrids/mkscripgrid.ncl diff --git a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst index 6a0aaf86e3..5685224c93 100644 --- a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst +++ b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst @@ -6,76 +6,10 @@ Creating input for surface dataset generation ********************************************* -Generating SCRIP grid files +Generating ESMF mesh files ================================== -The utility ``mkmapdata.sh`` requires SCRIP format input files to describe the input and output grids that maps are generated for. CLM provides a utility, ``mkmapgrids`` that generates those files. The program converts old formats of CAM or CLM grid files to SCRIP grid format. There is also a NCL script (``mkscripgrid.ncl``) to create regular latitude longitude regional or single-point grids at the resolution the user desires. +The ``mksurfdata_esmf`` tool requires ESMF mesh files to describe the input and output grids used for generating fsurdat and landuse files. CLM? provides the ``mesh_maker`` tool for generating such files. -SCRIP grid files for all the standard model resolutions and the raw surface datasets have already been done and the files are in the XML database. Hence, this step doesn't need to be done -- EXCEPT WHEN YOU ARE CREATING YOUR OWN GRIDS. +ESMF mesh files for all the standard model resolutions and the raw surface datasets already exist and the files are in the XML database. Hence, you may skip this step -- UNLESS YOU ARE CREATING YOUR OWN GRIDS. - -Creating mapping files for mksurfdata_esmf -============================================== - -``mkmapdata.sh`` uses the above SCRIP grid input files to create SCRIP mapping data files (uses ESMF). - -The bash shell script ``$CTSMROOT/tools/mkmapgrids/mkmapdata.sh`` uses ``ESMF_RegridWeightGen`` to create a list of maps from the raw datasets that are input to ``mksurfdata_esmf``. Each dataset that has a different grid, or land-mask needs a different mapping file for it, but many different raw datasets share the same grid/land-mask as other files. Hence, there doesn't need to be a different mapping file for EACH raw dataset---just for each raw dataset that has a DIFFERENT grid or land-mask. See :numref:`Figure mkmapdata.sh` for a visual representation of how this works. The bash script figures out which mapping files it needs to create and then runs ``ESMF_RegridWeightGen`` for each one. You can then either enter the datasets into the XML database (see Chapter :numref:`adding-new-resolutions-section`), or leave the files in place and use the ``-res usrspec -usr_gname -usr_gdate`` options to ``mksurfdata_esmf``. ``mkmapdata.sh`` has a help option with the following -:: - - ../../tools/mkmapdata/mkmapdata.sh - - ********************** - usage on cheyenne:Figure mkmapdata.sh - ./mkmapdata.sh - - valid arguments: - [-f|--gridfile ] - Full pathname of model SCRIP grid file to use - This variable should be set if this is not a supported grid - This variable will override the automatic generation of the - filename generated from the -res argument - the filename is generated ASSUMING that this is a supported - grid that has entries in the file namelist_defaults_clm.xml - the -r|--res argument MUST be specied if this argument is specified - [-r|--res ] - Model output resolution (default is 10x15) - [-t|--gridtype ] - Model output grid type - supported values are [regional,global], (default is global) - [-b|--batch] - Toggles batch mode usage. - If you want to run in batch mode - you need to have a separate batch script for a supported machine - that calls this script interactively - you cannot submit this - script directory to the batch system - [-l|--list] - List mapping files required (use check_input_data to get them) - also writes data to clm.input_data_list - [-d|--debug] - Toggles debug-only (don't actually run mkmapdata just echo what would happen) - [-h|--help] - Displays this help message - [-v|--verbose] - Toggle verbose usage -- log more information on what is happening - - You can also set the following env variables: - ESMFBIN_PATH - Path to ESMF binaries - (default is /contrib/esmf-5.3.0-64-O/bin) - CSMDATA ------ Path to CESM input data - (default is /glade/p/cesm/cseg/inputdata) - MPIEXEC ------ Name of mpirun executable - (default is mpirun.lsf) - REGRID_PROC -- Number of MPI processors to use - (default is 8) - - **pass environment variables by preceding above commands - with 'env var1=setting var2=setting ' - ********************** - -.. _Figure mkmapdata.sh: - -.. figure:: mkmapdata_details.jpeg - - Details of running mkmapdata.sh - -Each of the raw datasets for ``mksurfdata_esmf`` needs a mapping file to map from the output grid you are running on to the grid and land-mask for that dataset. This is what ``mkmapdata.sh`` does. To create the mapping files you need a SCRIP grid file to correspond with each resolution and land mask that you have a raw data file in ``mksurfdata_esmf``. Some raw datasets share the same grid and land mask -- hence they can share the same SCRIP grid file. The output maps created here go into ``mksurfdata_esmf`` see :numref:`Figure Workflow of CLM5 Land Use Data Tool and mksurfdata_esmf Tool`. diff --git a/tools/mkmapgrids/README b/tools/mkmapgrids/README deleted file mode 100644 index cc6d2cb644..0000000000 --- a/tools/mkmapgrids/README +++ /dev/null @@ -1,22 +0,0 @@ -$CTSMROOT/tools/mkmapgrids/README June08/2018 - -NCL script to create a SCRIP grid file for a regular lat/lon grid. - -To use the script, set the following environment variables - -Required (or defaults to a single point over Boulder Colorado) - -PTNAME ! name of your grid -S_LAT ! Southern latitude corner -N_LAT ! Northern latitude corner -E_LON ! Eastern longitude corner -W_LON ! Western longitude corner - -Optional: - -NX ! Number of grid points along longitude (default 1) -NY ! Number of grid points along latitude (default 1) -IMASK ! 0 or 1, mask to use if all points are active or not (default active) -PRINT ! TRUE/FALSE do extra verbose printing or not (default FALSE) -GRIDFILE ! Output filename - diff --git a/tools/mkmapgrids/mkscripgrid.ncl b/tools/mkmapgrids/mkscripgrid.ncl deleted file mode 100644 index 0cbd1a8960..0000000000 --- a/tools/mkmapgrids/mkscripgrid.ncl +++ /dev/null @@ -1,188 +0,0 @@ -; -; mkscripgrid.ncl -; -; Create SCRIP grid and mapping file for a land-only point or region. -; Requires NCL 6.1.0 or later for the ESMF regridding functions -; -; Erik Kluzek -; Dec/07/2011 -; -load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" -load "$NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl" -begin - ; =========================================================================================================== - ; Set a few constants needed later - cdate = systemfunc( "date +%y%m%d" ); - ldate = systemfunc( "date" ); - ; - ; IMPORTANT NOTE: EDIT THE FOLLOWING TO CUSTOMIZE or use ENV VARIABLE SETTINGS - ; Edit the following as needed to interpolate to a new resolution. - ; - ; Input resolution and position - ; - name = getenv("PTNAME"); ; Get name of this point - - latS = stringtodouble( getenv("S_LAT") ); ; Get south latitude from env variable - latN = stringtodouble( getenv("N_LAT") ); ; Get north latitude from env variable - lonE = stringtodouble( getenv("E_LON") ); ; Get east longitude from env variable - lonW = stringtodouble( getenv("W_LON") ); ; Get west longitude from env variable - - nx = stringtointeger( getenv("NX" ) ); ; Get number of grids along longitude lines - ny = stringtointeger( getenv("NY" ) ); ; Get number of grids along latitude lines - - imask = stringtointeger( getenv("IMASK") ); ; Get imask to use from env variable - - print_str = getenv("PRINT"); ; Do Extra printing for debugging - - outfilename = getenv("GRIDFILE"); ; Get filename from env variable - - gitdescribe = getenv("GITDES"); ; Git describe from the source clone - - if ( ismissing(nx) )then - nx = 1; - end if - if ( ismissing(ny) )then - ny = 1; - end if - if ( ismissing(imask) )then - imask = 1; - end if - if ( ismissing(name) )then - name = nx+"x"+ny+"pt_US-UMB"; - end if - if ( ismissing(latS) )then - latS = 45.5098; - end if - if ( ismissing(latN) )then - latN = 45.6098; - end if - if ( ismissing(lonW) )then - lonW = 275.2362; - end if - if ( ismissing(lonE) )then - lonE = 275.3362; - end if - if ( ismissing(print_str) )then - printn = False; - else - if ( print_str .eq. "TRUE" )then - printn = True; - else - printn = False; - end if - end if - - if ( ismissing(outfilename) )then - if ( imask .eq. 1 )then - outfilename = "SCRIPgrid_"+name+"_nomask_c"+cdate+".nc"; - else - if ( imask .eq. 0 )then - outfilename = "SCRIPgrid_"+name+"_noocean_c"+cdate+".nc"; - else - outfilename = "SCRIPgrid_"+name+"_mask_c"+cdate+".nc"; - end if - end if - end if - - if ( ismissing(gitdescribe) )then - gitdescribe = systemfunc( "git describe" ) - end if - - system( "/bin/rm -f "+outfilename ); - if ( printn )then - print( "output file: "+outfilename ); - end if - -function fspan1up( fbegin [*]:double, fend [*]:double, number:integer ) -; -; An "fspan" that can handle size of 1 and up. -; Do fspan for arrays of two or more, or average of end points for array of one. -; -local farray; -begin - if ( number .eq. 1) then - farray = (/ (fbegin + fend) / 2.0d00 /); - else - farray = fspan( fbegin, fend, number ); - end if - return( farray ); -end - - ; - ; Compute derived quantities - ; - - delX = (lonE - lonW) / int2dble(nx); - delY = (latN - latS) / int2dble(ny); - - lonCenters = fspan1up( (lonW + delX/2.d0), (lonE - delX/2.d0), nx) - latCenters = fspan1up( (latS + delY/2.d0), (latN - delY/2.d0), ny) - lon = new( (/ny, nx/), "double" ); - lat = new( (/ny, nx/), "double" ); - if ( (nx .eq. 1) .or. (ny .eq. 1) )then - if ( printn )then - print( "Calculate corners" ) - end if - lonCorners = new( (/ny, nx, 4/), "double" ); - latCorners = new( (/ny, nx, 4/), "double" ); - else - if ( printn )then - print( "Have NCL calculate corners" ) - end if - end if - do i = 0, nx-1 - lat(:,i) = latCenters; - if ( (nx .eq. 1) .or. (ny .eq. 1) )then - latCorners(:,i,0) = latCenters - delY/2.d0; - latCorners(:,i,1) = latCenters - delY/2.d0; - latCorners(:,i,2) = latCenters + delY/2.d0; - latCorners(:,i,3) = latCenters + delY/2.d0; - end if - end do - do j = 0, ny-1 - lon(j,:) = lonCenters; - if ( (nx .eq. 1) .or. (ny .eq. 1) )then - lonCorners(j,:,0) = lonCenters - delX/2.d0; - lonCorners(j,:,1) = lonCenters + delX/2.d0; - lonCorners(j,:,2) = lonCenters + delX/2.d0; - lonCorners(j,:,3) = lonCenters - delX/2.d0; - end if - end do - - ; for some reason, "No_FillValue" isn't working in the case where imask=1 - Mask2D = new( (/ny,nx/), "integer", "No_FillValue" ) - Mask2D(:,:) = imask - gridSize = delX+"x"+delY - - ; - ; Create SCRIP grid file - ; - - Opt = True - Opt@Mask2D = Mask2D - if ( (nx .eq. 1) .or. (ny .eq. 1) )then - Opt@GridCornerLat = latCorners - Opt@GridCornerLon = lonCorners - end if - Opt@Title = "SCRIP grid file for "+name - if (printn) then - Opt@Debug = True - end if - curvilinear_to_SCRIP(outfilename, lat, lon, Opt) - - ; - ; Add global attributes to file - ; - - nc = addfile( outfilename, "w" ); - nc@history = ldate+": create using mkscripgrid.ncl"; - nc@comment = "Ocean is assumed to be non-existant in this region"; - nc@Version = gitdescribe; - if ( printn )then - print( "================================================================================================" ); - print( "Successfully created SCRIP grid file: "+outfilename); - end if - - ; =========================================================================================================== - -end From 162a793e4de093a648b124f3aa4a8b8dc0b92563 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 2 Apr 2026 18:14:53 -0600 Subject: [PATCH 07/21] Add reference to Section 1.9 _how-to-make-mesh --- .../creating-input-for-surface-dataset-generation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst index 5685224c93..0949236164 100644 --- a/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst +++ b/doc/source/users_guide/using-clm-tools/creating-input-for-surface-dataset-generation.rst @@ -9,7 +9,7 @@ Creating input for surface dataset generation Generating ESMF mesh files ================================== -The ``mksurfdata_esmf`` tool requires ESMF mesh files to describe the input and output grids used for generating fsurdat and landuse files. CLM? provides the ``mesh_maker`` tool for generating such files. +The ``mksurfdata_esmf`` tool requires ESMF mesh files to describe the input and output grids used for generating fsurdat and landuse files. CLM provides the ``mesh_maker`` tool for generating such files. See section :numref:`how-to-make-mesh` for details. ESMF mesh files for all the standard model resolutions and the raw surface datasets already exist and the files are in the XML database. Hence, you may skip this step -- UNLESS YOU ARE CREATING YOUR OWN GRIDS. From 547ffe9102c24630691106874bf8b39330fd484a Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 2 Apr 2026 18:24:36 -0600 Subject: [PATCH 08/21] Remove some figures not in use by the new documentation --- doc/source/users_guide/using-clm-tools/GlobalDomain.jpeg | 3 --- .../users_guide/using-clm-tools/LegendCLMToolDataFlow.jpeg | 3 --- doc/source/users_guide/using-clm-tools/mkmapdata_details.jpeg | 3 --- .../users_guide/using-clm-tools/mkmapdata_mksurfdata.jpeg | 3 --- doc/source/users_guide/using-clm-tools/mknoocnmap.jpeg | 3 --- doc/source/users_guide/using-clm-tools/mksurfdata_details.jpeg | 3 --- 6 files changed, 18 deletions(-) delete mode 100644 doc/source/users_guide/using-clm-tools/GlobalDomain.jpeg delete mode 100644 doc/source/users_guide/using-clm-tools/LegendCLMToolDataFlow.jpeg delete mode 100644 doc/source/users_guide/using-clm-tools/mkmapdata_details.jpeg delete mode 100644 doc/source/users_guide/using-clm-tools/mkmapdata_mksurfdata.jpeg delete mode 100644 doc/source/users_guide/using-clm-tools/mknoocnmap.jpeg delete mode 100644 doc/source/users_guide/using-clm-tools/mksurfdata_details.jpeg diff --git a/doc/source/users_guide/using-clm-tools/GlobalDomain.jpeg b/doc/source/users_guide/using-clm-tools/GlobalDomain.jpeg deleted file mode 100644 index 7e2df5ad90..0000000000 --- a/doc/source/users_guide/using-clm-tools/GlobalDomain.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9000cdb3c7761a9838e9af059f36dfcc204c7a02bdab6d5206da0de1bf3051ac -size 38541 diff --git a/doc/source/users_guide/using-clm-tools/LegendCLMToolDataFlow.jpeg b/doc/source/users_guide/using-clm-tools/LegendCLMToolDataFlow.jpeg deleted file mode 100644 index e0f1559cbd..0000000000 --- a/doc/source/users_guide/using-clm-tools/LegendCLMToolDataFlow.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d71ecdf638e4582f66f1c2b101b6548bd9d47fe2cb3de0f58386048f6ce6cfbc -size 15711 diff --git a/doc/source/users_guide/using-clm-tools/mkmapdata_details.jpeg b/doc/source/users_guide/using-clm-tools/mkmapdata_details.jpeg deleted file mode 100644 index b8d33578b8..0000000000 --- a/doc/source/users_guide/using-clm-tools/mkmapdata_details.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:56399b80b49aa4705b27a0e98758b120f78c4eb62d993f85944db2c1d2122d0f -size 51018 diff --git a/doc/source/users_guide/using-clm-tools/mkmapdata_mksurfdata.jpeg b/doc/source/users_guide/using-clm-tools/mkmapdata_mksurfdata.jpeg deleted file mode 100644 index 7e59cee7f3..0000000000 --- a/doc/source/users_guide/using-clm-tools/mkmapdata_mksurfdata.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d85feb3d5c6aba70963f51b10406cd45ff334c60f2daf965ccb9b1abbc158985 -size 41951 diff --git a/doc/source/users_guide/using-clm-tools/mknoocnmap.jpeg b/doc/source/users_guide/using-clm-tools/mknoocnmap.jpeg deleted file mode 100644 index 308fde0e1c..0000000000 --- a/doc/source/users_guide/using-clm-tools/mknoocnmap.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2c4d6c677d6e68761f9c6677be4bc94d93cd75c26989c834cd3e022db432359d -size 33914 diff --git a/doc/source/users_guide/using-clm-tools/mksurfdata_details.jpeg b/doc/source/users_guide/using-clm-tools/mksurfdata_details.jpeg deleted file mode 100644 index 9d6e829891..0000000000 --- a/doc/source/users_guide/using-clm-tools/mksurfdata_details.jpeg +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:32104479195bd2612b09df1322b97c5b2db220d8d0115e32e6020e17ade597e0 -size 41837 From af56462b7219adaa46ac330b3cc405cd3c4fb174 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 3 Apr 2026 15:14:31 -0600 Subject: [PATCH 09/21] Remove mknoocnmap.pl and mkunitymap.ncl --- tools/site_and_regional/README.md | 11 - tools/site_and_regional/mknoocnmap.pl | 302 ------------------------- tools/site_and_regional/mkunitymap.ncl | 166 -------------- 3 files changed, 479 deletions(-) delete mode 100755 tools/site_and_regional/mknoocnmap.pl delete mode 100644 tools/site_and_regional/mkunitymap.ncl diff --git a/tools/site_and_regional/README.md b/tools/site_and_regional/README.md index e2ec7cda95..56f0ff451c 100644 --- a/tools/site_and_regional/README.md +++ b/tools/site_and_regional/README.md @@ -51,14 +51,3 @@ neon_gcs_upload Script to rename and upload NEON site finidat files to NEON server for use in transient startup cases -## DEPRECATED SCRIPTS: - -mknoocnmap.pl - Script to create unity mapping dataset for single-point - or regional studies over land-only (no ocean). - -NCL Scripts available: - -mkunitymap.ncl - NCL script to create a unity map -- ran by above script (mknoocnmap.pl) - diff --git a/tools/site_and_regional/mknoocnmap.pl b/tools/site_and_regional/mknoocnmap.pl deleted file mode 100755 index 5438c8bcd3..0000000000 --- a/tools/site_and_regional/mknoocnmap.pl +++ /dev/null @@ -1,302 +0,0 @@ -#!/usr/bin/env perl -# -# mknoocnmap.pl Erik Kluzek -# Dec/07/2011 -# -# Create SCRIP grid and mapping files for a single-point or region -# that is assumed to be a land land-only region. -# -use Cwd; -use strict; -use English; -use IO::File; -use Getopt::Long; - -# -# Global constants -# -my $degsiz = 0.1; - -#----------------------------------------------------------------------------------------------- -# Set the directory that contains this scripts. If the command was issued using a -# relative or absolute path, that path is in $ProgDir. Otherwise assume the -# command was issued from the current working directory. - -(my $ProgName = $0) =~ s!(.*)/!!; # name of this script -my $ProgDir = $1; # name of directory containing this script -- may be a - # relative or absolute path, or null if the script - # is in - # the user's PATH -my $cmdline = "@ARGV"; # Command line arguments to script -my $cwd = getcwd(); # current working directory -my $scrdir; # absolute pathname of directory that contains this script -my $nm = "${ProgName}::"; # name to use if script dies -if ($ProgDir) { - $scrdir = absolute_path($ProgDir); -} else { - $scrdir = $cwd; -} - - -#----------------------------------------------------------------------------------------------- - -sub usage { - die < Center latitude,longitude of the grid to create. - -name [-or -n] Name to use to describe point - -OPTIONS - -dx Size of total grid in degrees in longitude direction - (default is $degsiz) - -dy Size of total grid in degrees in latitude direction - (default is $degsiz) - -silent [or -s] Make output silent - -help [or -h] Print usage to STDOUT. - -verbose [or -v] Make output more verbose. - -nx Number of longitudes (default is 1) - -ny Number of latitudes (default is 1) -EOF -} - -#----------------------------------------------------------------------------------------------- - -sub get_latlon { -# -# Return the latitude and longitude of the input string and validate it -# - my $string = shift; - my $desc = shift; - my $dx = shift; - my $dy = shift; - - my $lat = undef; - my $lon = undef; - my $valreal1 = "[+-]?[0-9]*\.?[0-9]*[EedDqQ]?[0-9+-]*"; - - if ( $string =~ /^($valreal1)\s*,\s*($valreal1)$/ ) { - $lat = $1; - $lon = $2; - } else { - die <<"EOF"; -** $ProgName - Error in entering latitude/longitude for $desc ** -EOF - } - if ( $dx <= 0.0 || $dx > 360. ) { - die <<"EOF"; -** $ProgName - Bad value for dx (=$dx) for $desc ** - } - if ( $dy <= 0.0 || $dy > 180. ) { - die <<"EOF"; -** $ProgName - Bad value for dy (=$dy) for $desc ** - } - if ( ($lat < -90.+$dy/2.0) || ($lat > 90.0-$dy/2.0) ) { - die <<"EOF"; -** $ProgName - Bad value for latitude (=$lat) for $desc ** -EOF - } - if ( ($lon < $dx/2.0) || ($lon > 360.0-$dx/2.0) ) { - die <<"EOF"; -** $ProgName - Bad value for longitude (=$lat) for $desc ** -EOF - } - return( $lat, $lon ); - -} - -#------------------------------------------------------------------------------- - -sub absolute_path { -# -# Convert a pathname into an absolute pathname, expanding any . or .. characters. -# Assumes pathnames refer to a local filesystem. -# Assumes the directory separator is "/". -# - my $path = shift; - my $cwd = getcwd(); # current working directory - my $abspath; # resulting absolute pathname - -# Strip off any leading or trailing whitespace. (This pattern won't match if -# there's embedded whitespace. - $path =~ s!^\s*(\S*)\s*$!$1!; - -# Convert relative to absolute path. - - if ($path =~ m!^\.$!) { # path is "." - return $cwd; - } elsif ($path =~ m!^\./!) { # path starts with "./" - $path =~ s!^\.!$cwd!; - } elsif ($path =~ m!^\.\.$!) { # path is ".." - $path = "$cwd/.."; - } elsif ($path =~ m!^\.\./!) { # path starts with "../" - $path = "$cwd/$path"; - } elsif ($path =~ m!^[^/]!) { # path starts with non-slash character - $path = "$cwd/$path"; - } - - my ($dir, @dirs2); - my @dirs = split "/", $path, -1; # The -1 prevents split from stripping trailing nulls - # This enables correct processing of the input "/". - - # Remove any "" that are not leading. - for (my $i=0; $i<=$#dirs; ++$i) { - if ($i == 0 or $dirs[$i] ne "") { - push @dirs2, $dirs[$i]; - } - } - @dirs = (); - - # Remove any "." - foreach $dir (@dirs2) { - unless ($dir eq ".") { - push @dirs, $dir; - } - } - @dirs2 = (); - - # Remove the "subdir/.." parts. - foreach $dir (@dirs) { - if ( $dir !~ /^\.\.$/ ) { - push @dirs2, $dir; - } else { - pop @dirs2; # remove previous dir when current dir is .. - } - } - if ($#dirs2 == 0 and $dirs2[0] eq "") { return "/"; } - $abspath = join '/', @dirs2; - return( $abspath ); -} - -#------------------------------------------------------------------------------- - -# Process command-line options - -my %opts = ( - ctr => undef, - help => undef, - name => undef, - nx => 1, - ny => 1, - dx => $degsiz, - dy => $degsiz, - silent => 0, - verbose => 0, - ); - -GetOptions( - "p|centerpoint=s" => \$opts{'ctr'}, - "n|name=s" => \$opts{'name'}, - "nx=i" => \$opts{'nx'}, - "ny=i" => \$opts{'ny'}, - "dx=f" => \$opts{'dx'}, - "dy=f" => \$opts{'dy'}, - "h|help" => \$opts{'help'}, - "s|silent" => \$opts{'silent'}, - "v|verbose" => \$opts{'verbose'}, -) or usage(); - -# Check for unparsed arguments -if (@ARGV) { - print "ERROR: unrecognized arguments: @ARGV\n"; - usage(); -} - -if ( $opts{'verbose'} && $opts{'silent'} ) { - print "ERROR: Can NOT set both silent and verbose at once!\n"; - usage(); -} -my $printlev; -if ( $opts{'verbose'} ) { - $printlev = 2; -} elsif ( $opts{'silent'} ) { - $printlev = 0; -} else { - $printlev = 1; -} - -if ( ! defined($opts{'ctr'}) ) { - print "ERROR: MUST set the center point\n"; - usage(); -} -if ( ! defined($opts{'name'}) ) { - print "ERROR: MUST set the name of the point\n"; - usage(); -} -my $name = $opts{'name'}; - -my ($lat,$lon) = get_latlon( $opts{'ctr'}, $name, $opts{'dx'}, $opts{'dy'} ); -my $S_lat = $lat - $opts{'dy'}/2.0; -my $N_lat = $lat + $opts{'dy'}/2.0; -my $W_lon = $lon - $opts{'dx'}/2.0; -my $E_lon = $lon + $opts{'dx'}/2.0; - -my $nx = $opts{'nx'}; -my $ny = $opts{'ny'}; -if ( $opts{'nx'} < 1 ) { - print "ERROR: nx MUST be greater than or equal to 1\n"; - usage(); -} -if ( $opts{'ny'} < 1 ) { - print "ERROR: ny MUST be greater than or equal to 1\n"; - usage(); -} - -#----------------------------------------------------------------------------------------------- -my $print; -if ( $printlev > 1 ) { - $print = "PRINT=TRUE"; -} - -# Creation date -my $cdate = `date +%y%m%d`; chomp( $cdate ); - -if ( $printlev > 0 ) { - print "\n\nCreate SCRIP grid and mapping files for a single-point\n"; -} -# land grid... -my $GITDES = `cd $scrdir; git describe; cd -`; -$GITDES =~ s/\n//g; -my $grddir = absolute_path( "$scrdir/../mkmapgrids" ); -my $cwd = getcwd(); -my $datdir = $grddir; -if ( $cwd ne $scrdir ) { - $datdir = $cwd; -} -my $grid1 = "$datdir/SCRIPgrid_${name}_nomask_c${cdate}.nc"; -my $cmdenv = "env S_LAT=$S_lat W_LON=$W_lon N_LAT=$N_lat E_LON=$E_lon " . - "NX=$nx NY=$ny PTNAME=$name GITDES=$GITDES $print "; - -my $cmd = "$cmdenv GRIDFILE=$grid1 ncl $scrdir/../mkmapgrids/mkscripgrid.ncl"; -if ( $printlev > 0 ) { - print "Create land SCRIP gridfile\n"; - print "Execute: $cmd\n"; -} -system( $cmd ); - -# ocean grid... -my $grid2 = "$datdir/SCRIPgrid_${name}_noocean_c${cdate}.nc"; -my $cmd = "$cmdenv GRIDFILE=$grid2 IMASK=0 ncl $scrdir/../mkmapgrids/mkscripgrid.ncl"; -if ( $printlev > 0 ) { - print "Create ocean SCRIP gridfile\n"; - print "Execute: $cmd\n"; -} -system( $cmd ); - -# Now create a unity mapping between the two... -# Note reversal of grid1 & grid2, because we want an ocean -> land -# mapping file -my $mapfile = "$datdir/map_${name}_noocean_to_${name}_nomask_aave_da_${cdate}.nc"; -my $cmd = "env GRIDFILE1=$grid2 GRIDFILE2=$grid1 MAPFILE=$mapfile " . - "GITDES=$GITDES $print ncl $scrdir/mkunitymap.ncl"; - -if ( $printlev > 0 ) { - print "Create unity mapping file between the two gridfile\n"; - print "Execute: $cmd\n"; -} -system( $cmd ); - -if ( $printlev > 0 ) { - print "\n\nSuccessfully created grid/mapping files for single-point\n"; -} diff --git a/tools/site_and_regional/mkunitymap.ncl b/tools/site_and_regional/mkunitymap.ncl deleted file mode 100644 index 7570f7eedd..0000000000 --- a/tools/site_and_regional/mkunitymap.ncl +++ /dev/null @@ -1,166 +0,0 @@ -; -; mkunitymap.ncl -; -; Create a unity map file either between two identical grids or between two -; grids that do NOT intersect at all. -; -; Erik Kluzek -; Dec/07/2011 -; -load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" -begin - ; Set a few constants needed later - cdate = systemfunc( "date +%y%m%d" ); - ldate = systemfunc( "date" ); - ; =========================================================================================================== - ; - ; IMPORTANT NOTE: EDIT THE FOLLOWING TO CUSTOMIZE or use ENV VARIABLE SETTINGS - ; Edit the following as needed to interpolate to a new resolution. - gridfile1 = getenv("GRIDFILE1"); ; Get name of the first SCRIP grid file - gridfile2 = getenv("GRIDFILE2"); ; Get name of the second SCRIP grid file - - outfilename = getenv("MAPFILE"); ; Get name of the output mapping file - - print_str = getenv("PRINT"); ; Do Extra printing for debugging - - gitdescribe = getenv("GITDES"); ; Git describe from the source clone - - if ( ismissing(gridfile1) )then - print( "ERROR: GRIDFILE1 is missing!" ); - exit - end if - if ( ismissing(gridfile2) )then - print( "ERROR: GRIDFILE2 is missing!" ); - exit - end if - if ( ismissing(outfilename) )then - print( "ERROR: MAPFILE is missing!" ); - exit - end if - if ( ismissing(print_str) )then - printn = False; - else - if ( print_str .eq. "TRUE" )then - printn = True; - else - printn = False; - end if - end if - - if ( ismissing(gitdescribe) )then - gitdescribe = systemfunc( "git describe" ) - end if - - ; - ; Open up the input grid files - ; - nca = addfile( gridfile1, "r" ); - ncb = addfile( gridfile2, "r" ); - - system( "/bin/rm -f "+outfilename ); - if ( printn )then - print( "output mapping file to create: "+outfilename ); - end if - nc = addfile( outfilename, "c" ); - ; - ; Define dimensions - ; - n_a = dimsizes( nca->grid_center_lat ); - n_b = dimsizes( ncb->grid_center_lat ); - if ( n_a .ne. n_b )then - print( "ERROR: dimensions of input SCRIP grid files is NOT the same!" ); - exit - end if - if ( any(ncb->grid_imask .ne. 1.0d00) )then - print( "ERROR: the mask of the second file isn't identically 1!" ); - print( "(second file should be land grid file)"); - exit - end if - chkvars = (/ "grid_center_lat", "grid_center_lon", "grid_corner_lat", "grid_corner_lon" /); - do i = 1, dimsizes(chkvars)-1 - if ( any(nca->$chkvars(i)$ .ne. ncb->$chkvars(i)$) )then - print( "ERROR: the grid variables are different between the two files!: "+chkvars(i) ); - exit - end if - end do - n_s = n_a; - dimnames = (/ "n_a", "n_b", "n_s", "nv_a", "nv_b", "num_wgts", "src_grid_rank", "dst_grid_rank" /); - dsizes = (/ n_a, n_b, n_a, 4, 4, 1, 2, 2/); - is_unlim = (/ False, False, False, False, False, False, False, False /); - filedimdef( nc, dimnames, dsizes, is_unlim ); - - ; - ; Define grid dimensions - ; - filevardef( nc, "src_grid_dims", "integer", (/ "src_grid_rank" /)) - nc->src_grid_dims = (/nca->grid_dims/) - filevardef( nc, "dst_grid_dims", "integer", (/ "dst_grid_rank" /)) - nc->dst_grid_dims = (/ncb->grid_dims/) - - ; - ; Define variables - ; - cvars = (/ "yc", "xc", "yv", "xv", "mask" /); - gvars = (/ "grid_center_lat", "grid_center_lon", "grid_corner_lat", "grid_corner_lon", "grid_imask" /); - - do i = 0, dimsizes(cvars)-1 - var = cvars(i)+"_a"; - if ( cvars(i) .eq. "yv" .or. cvars(i) .eq. "xv" )then - dnamesa = (/ "n_a", "nv_a" /); - dnamesb = (/ "n_b", "nv_b" /); - else - dnamesa = (/ "n_a" /); - dnamesb = (/ "n_b" /); - end if - filevardef ( nc, var, typeof(nca->$gvars(i)$), dnamesa ); - filevarattdef ( nc, var, nca->$gvars(i)$ ); - nc->$var$ = (/ nca->$gvars(i)$ /); - var = cvars(i)+"_b"; - filevardef ( nc, var, typeof(nca->$gvars(i)$), dnamesb ); - filevarattdef ( nc, var, ncb->$gvars(i)$ ); - nc->$var$ = (/ ncb->$gvars(i)$ /); - delete( dnamesa ); - delete( dnamesb ); - end do - filevardef ( nc, "area_a", "double", (/ "n_a" /) ); - filevardef ( nc, "area_b", "double", (/ "n_b" /) ); - filevardef ( nc, "frac_a", "double", (/ "n_a" /) ); - filevardef ( nc, "frac_b", "double", (/ "n_b" /) ); - ; - ; Attributes - ; - nc->area_a@units = "square radians"; - nc->frac_a@units = "unitless"; - nc->area_b@units = nc->area_a@units; - nc->frac_b@units = nc->frac_a@units; - nc@conventions = "NCAR-CESM"; - nc@domain_a = gridfile1; - nc@domain_b = gridfile2; - nc@grid_file_src = gridfile1; - nc@grid_file_dst = gridfile2; - nc@title = "SCRIP mapping file between identical grids without ocean"; - nc@history = ldate+": create using mkunitymap.ncl"; - nc@Version = gitdescribe; - - ; - ; Fraction - ; - nc->frac_a = int2dble( (/nc->mask_a/) ); - nc->frac_b = int2dble( (/nc->mask_b/) ); - ; - ; Area - ; - nc->area_a = gc_qarea( nc->yv_a(:,:), nc->xv_a(:,:) ); - nc->area_b = gc_qarea( nc->yv_b(:,:), nc->xv_b(:,:) ); - ; - ; Weights - ; - filevardef ( nc, "col", "integer", (/ "n_s" /) ); - filevardef ( nc, "row", "integer", (/ "n_s" /) ); - filevardef ( nc, "S", "double", (/ "n_s" /) ); - - nc->col = ispan( 1, n_s, 1 ); - nc->row = nc->col; - nc->S = 1.0d00; - -end From 029f1c745b840f4e274e52fb79ab4e14f4d1b505 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 3 Apr 2026 15:15:36 -0600 Subject: [PATCH 10/21] wrf-tools.rst: replace SCRIP with mesh, rm refs to deprecated scripts --- .../lilac/specific-atm-models/wrf-tools.rst | 49 +++---------------- 1 file changed, 6 insertions(+), 43 deletions(-) diff --git a/doc/source/lilac/specific-atm-models/wrf-tools.rst b/doc/source/lilac/specific-atm-models/wrf-tools.rst index 1222a16f10..1caf0a24d7 100644 --- a/doc/source/lilac/specific-atm-models/wrf-tools.rst +++ b/doc/source/lilac/specific-atm-models/wrf-tools.rst @@ -16,61 +16,24 @@ Before this step, make sure you have successfully created geo_em* files for your specific WRF domain using WPS. Instructions on how to run ``geogrid.exe`` is described in here. -1. Create SCRIP grid file from WRF ``geo_em*`` files, using the following ncl - script:: +1. Create ESMF mesh file from WRF ``geo_em*`` files, using the make_mesh tool:: - ncl create_scrip_file.ncl + See section :numref:`how-to-make-mesh` for details. - This creates two files that are complements of each other only in the mask field +2. Create surface datasets in ``tools/mksurfdata_esmf``:: -2. Create mapping files by using ``mkmapdata`` code under - ``CTSM/tools/mkmapdata/``. + See section :numref:`creating-surface-datasets` for details. - Using environment variables set the following environment varibales needed - by ``mkunitymap.ncl`` code:: - - setenv GRIDFILE1 wrf2clm_ocean_noneg.nc - setenv GRIDFILE2 wrf2clm_land_noneg.nc - setenv MAPFILE wrf2clm_mapping_noneg.nc - setenv PRINT TRUE - - ncl mkunitymap.ncl - -.. warning:: - - This will throw some git errors if not run in a repository. - -3. Create ESMF mapping files by running ``regridbatch.sh``:: - - qsub regridbatch.sh - -4. In your ctsm repository directory, build:: - - ../../../configure --macros-format Makefile --mpilib mpi-serial - -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -5. Generate CTSM domain files using ``get_domain`` tool:: - - ./gen_domain -m /glade/work/$USER/ctsm/nldas_grid/scrip/wrf2clm_mapping_noneg.nc -o wrf2clm_ocn_noneg -l wrf2clm_lnd_noneg - -.. todo:: - Update the below, as ``mksurfdata.pl`` no longer exists. - -6. Create surface datasets in ``tools/mksurfdata_esmf``:: - - ./mksurfdata.pl -res usrspec -usr_gname "nldas" -usr_gdate "190124" -usr_mapdir "/glade/work/$USER/ctsm/nldas_grid/map" -y 2000 -exedir "/glade/u/home/$USER/src/ctsm/ctsm_surfdata/tools/mksurfdata_esmf" -no-crop Merge WRF initial conditions into an existing CTSM initial condition file -------------------------------------------------------------------------- -The following procedure is if you'd wish to merget WRF inital conditions from +The following procedure is if you'd wish to merge WRF inital conditions from ``wrfinput`` file into CTSM initial condition file :: ncl transfer_wrfinput_to_ctsm_with_snow.ncl 'finidat="the_existing_finidat_file.nc"' 'wrfinput="your_wrfinput_file"' 'merged="the_merged_finidat_file.nc"' .. todo:: - Sam, can you please make the above ncl script available. + Versions of the transfer_wrfinput ncl script are available in /glade/work/slevis/git_wrf/ctsm_init/. From 577d2bc8c1a957f7d6044fe0923dd76e86a57888 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 3 Apr 2026 15:43:08 -0600 Subject: [PATCH 11/21] Remove unnecessary references to SCRIP, domain, and mapping files --- README | 1 - .../what-are-the-clm-tools.rst | 11 +--- python/ctsm/test/test_sys_mesh_modifier.py | 2 +- tools/README.md | 2 - tools/contrib/README.md | 2 - tools/contrib/create_scrip_file.ncl | 58 ------------------- 6 files changed, 2 insertions(+), 74 deletions(-) delete mode 100644 tools/contrib/create_scrip_file.ncl diff --git a/README b/README index deca3cd8d2..b5a11c5ed6 100644 --- a/README +++ b/README @@ -135,7 +135,6 @@ cime_config/usermods_dirs/clm ---- Directories of sets of user-modification subd tools/mksurfdata_esmf --------- Directory to build program to create surface dataset at any resolution. -tools/mkmapgrids -------------- NCL script to create a SCRIP grid file for a regular lat/lon grid (deprecated) tools/crop_calendars ---------- Tools to process and process and create crop calendar datasets for CTSM tools/modify_input_files ------ Script to modify existing CTSM input datasets in standard ways tools/site_and_regional ------- Scripts to create input datasets for single site and regional diff --git a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst index dec0fb67c5..dd179ec233 100644 --- a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst +++ b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst @@ -15,16 +15,7 @@ The tools produce files that can be used for CLM4.5 and |version|. They do **NOT The list of generally important scripts and programs are as follows. -1. ``./mkmapgrids/mkscripgrid.ncl`` to create SCRIP grid files for regular latitude/longitude grids. - -#. ``./mkmapdata`` to create SCRIP mapping data file from SCRIP grid files (uses ESMF). - -#. ``mksurfdata_esmf`` to create surface datasets from gridded datasets that we refer to as raw datasets (ctsm5_2 and newer versions). - -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -#. ``$CIMEROOT/tools/mapping/gen_domain_files/gen_domain`` to create a domain file for datm from a mapping file. The domain file is then used by BOTH datm AND CLM to define the grid and land-mask. +1. ``mksurfdata_esmf`` to create surface datasets from gridded datasets that we refer to as raw datasets (ctsm5_2 and newer versions). #. ``$CIMEROOT/tools/cprnc`` to compare two NetCDF files. diff --git a/python/ctsm/test/test_sys_mesh_modifier.py b/python/ctsm/test/test_sys_mesh_modifier.py index 69ec8afdac..b28a77e7a5 100755 --- a/python/ctsm/test/test_sys_mesh_modifier.py +++ b/python/ctsm/test/test_sys_mesh_modifier.py @@ -81,7 +81,7 @@ def createScripGridAndMask(self, fsurdat_in): # The mask will be idnetically 1, no matter the input grid (you can, change it, but you have to get it from a mapping file) # Since, the mask is going to be changed later, it's fine that the mask at this point is identically 1. - # This could also alturnatively be done, by using the stored SCRIP grid file for the resolution under CESM inputdata + # This could also alternatively be done, by using the stored SCRIP grid file for the resolution under CESM inputdata ncks_cmd = ( f"ncks --rgr infer --rgr scrip={self.scrip_file} {fsurdat_in} {self.metadata_file}" ) diff --git a/tools/README.md b/tools/README.md index 3e74fd6f3e..d36290729b 100644 --- a/tools/README.md +++ b/tools/README.md @@ -11,8 +11,6 @@ I. General directory structure: crop_calendars --- Regrid and process GGCMI sowing and harvest date files for use in CTSM. - mkmapgrids ------- Create regular lat/lon SCRIP grid files - site_and_regional Scripts for handling input datasets for site and regional cases. These scripts both help with creation of datasets using the standard process as well as subsetting existing datasets and overwriting diff --git a/tools/contrib/README.md b/tools/contrib/README.md index c53f807ffe..d7a4c55ebb 100644 --- a/tools/contrib/README.md +++ b/tools/contrib/README.md @@ -21,8 +21,6 @@ abm_raw.ncl add_tillage_to_paramsfile.py Add tillage data to the parameter file Can this be deleted since we have modify paramfile scripts? EBK 3/12/2026 -create_scrip_file.ncl - Create a SCRIP grid file needed for running with WRF CRUJRA_antarctica.ipynb Jupyter notebook to add forcing data over Antarctica for the CRUJRA forcing CRUJRA_greenland.ipynb diff --git a/tools/contrib/create_scrip_file.ncl b/tools/contrib/create_scrip_file.ncl deleted file mode 100644 index c2d911743b..0000000000 --- a/tools/contrib/create_scrip_file.ncl +++ /dev/null @@ -1,58 +0,0 @@ -load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl" -load "$NCARG_ROOT/lib/ncarg/nclscripts/esmf/ESMF_regridding.ncl" -begin - -nums = (/"00","01","02","03","04","05","06","07","08","09", \ - "10","11","12","13","14","15","16","17","18","19", \ - "20","21","22","23","24","25","26","27","28","29", \ - "30","31","32","33","34","35","36","37","38","39"/) - -;************************************************ -; get WRF fields -;************************************************ - -;change this to point to the relevant geo_em file. -wrf_file = addfile("geo_em.d01.nc", "r") - - -lat2d = wrf_file->XLAT_M(0,:,:) -lon2d = wrf_file->XLONG_M(0,:,:) -lon2d = where(lon2d.lt.0, lon2d+360.0, lon2d) -landmask = wrf_file->LANDMASK(0,:,:) - -ns_corner_lat = wrf_file->XLAT_V(0,:,:) -ns_corner_lon = wrf_file->XLONG_U(0,:,:) -ew_corner_lat = wrf_file->XLAT_U(0,:,:) -ew_corner_lon = wrf_file->XLONG_U(0,:,:) - -print(lat2d(0,0)) -print(lon2d(0,0)) - -print(ns_corner_lat(0:1,0:1)) -print(ns_corner_lon(0:1,0:1)) -print(ew_corner_lat(0:1,0:1)) -print(ew_corner_lon(0:1,0:1)) - -opt = True -opt@GridMask = landmask -opt@ForceOverwrite = True - -curvilinear_to_SCRIP("wrf2clm_land.nc",lat2d,lon2d,opt) - -opt@GridMask = where(landmask.eq.0,1,0) -curvilinear_to_SCRIP("wrf2clm_ocean.nc",lat2d,lon2d,opt) - -infile = addfile("wrf2clm_land.nc","r") - -lat1d = infile->grid_center_lat -lon1d = infile->grid_center_lon - -lat1dc = infile->grid_corner_lat -lon1dc = infile->grid_corner_lon - -print(lat1d(0)) -print(lon1d(0)) -print(lat1dc(0,:)) -print(lon1dc(0,:)) - -end From 2c1640b30cc4fb340d188c52e80a6cc07ab6fb38 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 3 Apr 2026 15:49:58 -0600 Subject: [PATCH 12/21] Remove some references to gen_domain --- .../using-clm-tools/building-the-clm-tools.rst | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst b/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst index b0d30d11c7..82b3bbe84b 100644 --- a/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst +++ b/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst @@ -4,10 +4,7 @@ .. include:: ../substitutions.rst -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -The tools **cprnc** and **gen_domain** use the CIME configure/build system which is described in the next section. +The tool **cprnc** uses the CIME configure/build system which is described in the next section. The only CLM FORTRAN tool is mksurfdata_esmf which has it's own build system that takes advantage of the cime build. @@ -15,10 +12,7 @@ The only CLM FORTRAN tool is mksurfdata_esmf which has it's own build system tha Building the CLM tools that use the CIME configure/build system ================================================================ -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -``cprnc`` and ``gen_domain`` both use the CIME configure/build system rather than the CLM specific version described above. +``cprnc`` uses the CIME configure/build system rather than the CLM specific version described above. See `CIME documentation on adding grids `_ for more information on adding grids, creating mapping files, and running ``gen_domain``. Also see the CIME file: ``$CTSMROOT/tools/mapping/gen_domain_files/INSTALL`` for how to build ``gen_domain``. From f78930fc609babba4814f62fa15186fb33f3123a Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Wed, 15 Apr 2026 13:13:53 -0600 Subject: [PATCH 13/21] Mostly formatting improvements to wrf-tools.rst --- doc/source/lilac/specific-atm-models/wrf-tools.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/doc/source/lilac/specific-atm-models/wrf-tools.rst b/doc/source/lilac/specific-atm-models/wrf-tools.rst index 1caf0a24d7..ab9ef318b2 100644 --- a/doc/source/lilac/specific-atm-models/wrf-tools.rst +++ b/doc/source/lilac/specific-atm-models/wrf-tools.rst @@ -16,13 +16,9 @@ Before this step, make sure you have successfully created geo_em* files for your specific WRF domain using WPS. Instructions on how to run ``geogrid.exe`` is described in here. -1. Create ESMF mesh file from WRF ``geo_em*`` files, using the make_mesh tool:: +1. Create ESMF mesh file from WRF ``geo_em*`` files, using the make_mesh tool. Details in section :numref:`how-to-make-mesh`. - See section :numref:`how-to-make-mesh` for details. - -2. Create surface datasets in ``tools/mksurfdata_esmf``:: - - See section :numref:`creating-surface-datasets` for details. +2. Create surface datasets in ``tools/mksurfdata_esmf``. Details in section :numref:`creating-surface-datasets`. Merge WRF initial conditions into an existing CTSM initial condition file @@ -31,6 +27,7 @@ Merge WRF initial conditions into an existing CTSM initial condition file The following procedure is if you'd wish to merge WRF inital conditions from ``wrfinput`` file into CTSM initial condition file :: + module load ncl ncl transfer_wrfinput_to_ctsm_with_snow.ncl 'finidat="the_existing_finidat_file.nc"' 'wrfinput="your_wrfinput_file"' 'merged="the_merged_finidat_file.nc"' .. todo:: From a364961b6be0a33ffaf9cbfd0da492b6543c144f Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 16 Apr 2026 17:08:08 -0600 Subject: [PATCH 14/21] Formatting corrections --- .../creating-surface-datasets.rst | 61 +++++++++---------- tools/mksurfdata_esmf/README.md | 4 +- 2 files changed, 33 insertions(+), 32 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst index 16f8a6d1a4..feeeab220b 100644 --- a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst +++ b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst @@ -2,28 +2,25 @@ .. _creating-surface-datasets: -=========================== - Creating Surface Datasets -=========================== +Creating Surface Datasets +========================= + +mksurfdata_esmf purpose +----------------------- - -## mksurfdata_esmf Purpose - This tool is intended to generate fsurdat files (surface datasets) and landuse files for the CTSM. It can generate global, regional, and single-point fsurdat files, as long as a mesh file is available for the grid. The subset_data tool allows users to make fsurdat files from existing fsurdat files when a mesh file is unavailable. Generally, users are encouraged to use the -subset_data tool for generating regional and single-point fsurdat files (see note about regional grids at the end). +subset_data tool for generating regional and single-point fsurdat files. - -## Building - +Building +-------- - -### Build Requirements - +Build Requirements +^^^^^^^^^^^^^^^^^^ mksurfdata_esmf is a distributed memory parallel program (using Message Passing Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) @@ -46,9 +43,11 @@ we demonstrate how to use the ctsm_pylib environment that we support in CTSM. Note, PNETCDF is an optional library that can be used, but is NOT required. -#### Use cime to manage the build requirements +.. rubric:: Use cime to manage the build requirements + +.. important:: -See [IMPORTANT NOTE](important note-only-working-on-derecho-currently) + CURRENTLY WORKS ONLY ON DERECHO IN CTSM (not CESM) CHECKOUTS For users working on cime machines you can use the build script to build the tool. On other machines you'll need to do a port to cime and tell how to build @@ -62,30 +61,30 @@ run the model on your machine, you will be able to build the tool there. To get a list of the machines that have been ported to cime: -``` shell -# Assuming pwd is the tools/mksurfdata_esmf directory -cd ../../cime/scripts # or ../../../../cime/scripts for a CESM checkout -./query_config --machines -``` +.. code-block:: + + # Assuming pwd is your CTSM or CESM checkout + cd cime/scripts + ./query_config --machines -#### NOTE: -In addition to having a port to cime, the machine also needs to have PIO built -and able to be referenced with the env variable PIO which will need to be in -the porting instructions for the machine. An independent PIO library -is available on supported CESM machines. +.. note:: - -#### IMPORTANT NOTE: ONLY WORKING ON DERECHO CURRENTLY - + In addition to having a port to cime, the machine also needs to have PIO built and able to be referenced with the env variable PIO which will need to be in the porting instructions for the machine. An independent PIO library is available on supported CESM machines. +.. important:: -> [!IMPORTANT] -> Currently we have run and tested mksurfdata_esmf on Derecho. Please see this github issue about mksurfdata_esmf on other CESM machines: + Currently we have run and tested mksurfdata_esmf on Derecho. Please see this github issue about mksurfdata_esmf on other CESM machines: https://github.com/ESCOMP/CTSM/issues/2341 -The ``$CTSMROOT/tools/README.md`` goes through the complete process for creating input files needed to run CLM. We repeat that file here: +The complete process +-------------------- + +If you have read the previous section, you are ready to proceed. The ``$CTSMROOT/tools/README.md`` goes through the complete process for creating input files needed to run CLM. The ``$CTSMROOT/tools/mksurfdata_esmf/README.md`` specifically goes through the complete process of generating surface and landuse datasets. We repeat those files here: .. include:: ../../../../tools/README.md :code: markdown +.. include:: ../../../../tools/mksurfdata_esmf/README.md + :code: markdown + diff --git a/tools/mksurfdata_esmf/README.md b/tools/mksurfdata_esmf/README.md index 95064a9839..dffa49a4a9 100644 --- a/tools/mksurfdata_esmf/README.md +++ b/tools/mksurfdata_esmf/README.md @@ -1,4 +1,7 @@ # Instructions for Using mksurfdata_esmf to Create Surface Datasets +#### $CTSMROOT/tools/mksurfdata_esmf/README + +Testing ## Table of contents 1. [Purpose](#purpose) @@ -217,4 +220,3 @@ ncks --mk_rec_dmn time file_with_time_equals_12.nc -o file_with_time_unlimited.n In general we recommend using subset_data and/or fsurdat_modifier for regional grids. - From 94051adff322a82307b6c23d65f57646290e137d Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 16 Apr 2026 17:18:47 -0600 Subject: [PATCH 15/21] Add reference to building-the-clm-tools for gen_domain --- .../setting-ctsm-runtime-options.rst | 2 +- .../users_guide/using-clm-tools/building-the-clm-tools.rst | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst b/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst index eea7c74db8..79fd8d5eba 100644 --- a/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst +++ b/doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst @@ -106,7 +106,7 @@ The first set of options in this file specifies key file names: - ``lnd_domain_file`` must be specified. This file specifies CTSM's grid and land mask. The general process for creating this file is described in section - creating-domain-files (deprecated). + :numref:`building-the-clm-tools`. - ``fsurdat`` also must be specified. This file specifies a variety of spatially-varying properties. This file is grid-specific, but can be created from grid-independent files diff --git a/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst b/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst index 82b3bbe84b..2093b410e6 100644 --- a/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst +++ b/doc/source/users_guide/using-clm-tools/building-the-clm-tools.rst @@ -1,9 +1,11 @@ +.. include:: ../substitutions.rst + +.. _building-the-clm-tools: + ======================== Building the CLM tools ======================== -.. include:: ../substitutions.rst - The tool **cprnc** uses the CIME configure/build system which is described in the next section. The only CLM FORTRAN tool is mksurfdata_esmf which has it's own build system that takes advantage of the cime build. From d68088172de157ccd1cdcd5995527ed5dbcc7633 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Thu, 16 Apr 2026 17:26:38 -0600 Subject: [PATCH 16/21] Remove debugging comment --- tools/mksurfdata_esmf/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/tools/mksurfdata_esmf/README.md b/tools/mksurfdata_esmf/README.md index dffa49a4a9..7152a5fdbe 100644 --- a/tools/mksurfdata_esmf/README.md +++ b/tools/mksurfdata_esmf/README.md @@ -1,8 +1,6 @@ # Instructions for Using mksurfdata_esmf to Create Surface Datasets #### $CTSMROOT/tools/mksurfdata_esmf/README -Testing - ## Table of contents 1. [Purpose](#purpose) 1. [Building](#building) From 2a90905f7dbdbfbd97eaa7b36e1b839278a14b00 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 8 May 2026 14:55:15 -0600 Subject: [PATCH 17/21] Introduce wrapping lines in this file --- .../creating-surface-datasets.rst | 28 +++++-------------- 1 file changed, 7 insertions(+), 21 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst index feeeab220b..57e0f5b818 100644 --- a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst +++ b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst @@ -8,13 +8,9 @@ Creating Surface Datasets mksurfdata_esmf purpose ----------------------- -This tool is intended to generate fsurdat files (surface datasets) and landuse files for the -CTSM. It can generate global, regional, and single-point fsurdat files, as long -as a mesh file is available for the grid. +This tool is intended to generate fsurdat files (surface datasets) and landuse files for the CTSM. It can generate global, regional, and single-point fsurdat files, as long as a mesh file is available for the grid. -The subset_data tool allows users to make fsurdat files from existing fsurdat -files when a mesh file is unavailable. Generally, users are encouraged to use the -subset_data tool for generating regional and single-point fsurdat files. +The subset_data tool allows users to make fsurdat files from existing fsurdat files when a mesh file is unavailable. Generally, users are encouraged to use the subset_data tool for generating regional and single-point fsurdat files. Building -------- @@ -22,10 +18,7 @@ Building Build Requirements ^^^^^^^^^^^^^^^^^^ -mksurfdata_esmf is a distributed memory parallel program (using Message Passing -Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) -for regridding as well as PIO (Parallel I/O) and NetCDF output. As -such, libraries must be built for the following: +mksurfdata_esmf is a distributed memory parallel program (using Message Passing Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) for regridding as well as PIO (Parallel I/O) and NetCDF output. As such, libraries must be built for the following: 1. MPI 2. NetCDF @@ -34,12 +27,9 @@ such, libraries must be built for the following: In addition for the build: python, bash-shell, CMake and GNU-Make are required -These libraries need to be built such that they can all work together in the -same executable. Hence, the above order may be required in building them. +These libraries need to be built such that they can all work together in the same executable. Hence, the above order may be required in building them. -CTSM submodules that are required are: cime and ccs_config. See [Building](#building-the-executable) on getting -those. A python environment that includes particular packages is also required -we demonstrate how to use the ctsm_pylib environment that we support in CTSM. +CTSM submodules cime and ccs_config are required, and we will show how these come in. A python environment that includes particular packages is also required. We demonstrate how to use the ctsm_pylib environment that we support in CTSM. Note, PNETCDF is an optional library that can be used, but is NOT required. @@ -49,15 +39,11 @@ Note, PNETCDF is an optional library that can be used, but is NOT required. CURRENTLY WORKS ONLY ON DERECHO IN CTSM (not CESM) CHECKOUTS -For users working on cime machines you can use the build script to build the -tool. On other machines you'll need to do a port to cime and tell how to build -for that machine. That's talked about in the cime documentation. -And you'll have to make some modifications to the build script. +For users working on cime machines you can use the build script to build the tool. On other machines you'll need to do a port to cime and tell how to build for that machine. That's talked about in the cime documentation. And you'll have to make some modifications to the build script. https://github.com/ESMCI/cime/wiki/Porting-Overview -Machines that already run CTSM or CESM have been ported to cime. So if you can -run the model on your machine, you will be able to build the tool there. +Machines that already run CTSM or CESM have been ported to cime. So if you can run the model on your machine, you will be able to build the tool there. To get a list of the machines that have been ported to cime: From f3fdafbdf4577c3c075f782032b91783469bf737 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 8 May 2026 15:19:44 -0600 Subject: [PATCH 18/21] Remove redundant sections from tools/mksurfdata_esmf/README.md --- tools/mksurfdata_esmf/README.md | 91 +++------------------------------ 1 file changed, 7 insertions(+), 84 deletions(-) diff --git a/tools/mksurfdata_esmf/README.md b/tools/mksurfdata_esmf/README.md index 7152a5fdbe..5182f03b2d 100644 --- a/tools/mksurfdata_esmf/README.md +++ b/tools/mksurfdata_esmf/README.md @@ -1,90 +1,13 @@ # Instructions for Using mksurfdata_esmf to Create Surface Datasets -#### $CTSMROOT/tools/mksurfdata_esmf/README +#### $CTSMROOT/tools/mksurfdata_esmf/README.md ## Table of contents -1. [Purpose](#purpose) -1. [Building](#building) -1. [Running a Single Submission](#running-for-a-single-submission) -1. [Running for Multiple Datasets](#running-for-the-generation-of-multiple-datasets) -1. [Notes](#notes) - - -## Purpose - -This tool is intended to generate fsurdat files (surface datasets) for the -CTSM. It can generate global, regional, and single-point fsurdat files, as long -as a mesh file is available for the grid. - -The subset_data tool allows users to make fsurdat files from existing fsurdat -files when a mesh file is unavailable. Generally, users should consider the -subset_data tool for generating regional and single-point fsurdat files. - - -## Building - - - -### Build Requirements - - -mksurfdata_esmf is a distributed memory parallel program (using Message Passing -Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) -for regridding as well as PIO (Parallel I/O) and NetCDF output. As -such, libraries must be built for the following: - -1. MPI -2. NetCDF -3. PIO -4. ESMF - -In addition for the build: python, bash-shell, CMake and GNU-Make are required - -These libraries need to be built such that they can all work together in the -same executable. Hence, the above order may be required in building them. - -CTSM submodules that are required are: cime and ccs_config. See [Building](#building-the-executable) on getting -those. A python environment that includes particular packages is also required -we demonstrate how to use the ctsm_pylib environment that we support in CTSM. - -Note, PNETCDF is an optional library that can be used, but is NOT required. - -#### Use cime to manage the build requirements - -See [IMPORTANT NOTE](important note-only-working-on-derecho-currently) - -For users working on cime machines you can use the build script to build the -tool. On other machines you'll need to do a port to cime and tell how to build -for that machine. That's talked about in the cime documentation. -And you'll have to make some modifications to the build script. - -https://github.com/ESMCI/cime/wiki/Porting-Overview - -Machines that already run CTSM or CESM have been ported to cime. So if you can -run the model on your machine, you will be able to build the tool there. - -To get a list of the machines that have been ported to cime: - -``` shell -# Assuming pwd is the tools/mksurfdata_esmf directory -cd ../../cime/scripts # or ../../../../cime/scripts for a CESM checkout -./query_config --machines -``` - -#### NOTE: -In addition to having a port to cime, the machine also needs to have PIO built -and able to be referenced with the env variable PIO which will need to be in -the porting instructions for the machine. An independent PIO library -is available on supported CESM machines. - - -#### IMPORTANT NOTE: ONLY WORKING ON DERECHO CURRENTLY - - - -> [!IMPORTANT] -> Currently we have run and tested mksurfdata_esmf on Derecho. Please see this github issue about mksurfdata_esmf on other CESM machines: - -https://github.com/ESCOMP/CTSM/issues/2341 +1. Purpose NOW IN THE USER'S GUIDE https://escomp.github.io/CTSM/users_guide/using-clm-tools/creating-surface-datasets.html#mksurfdata-esmf-purpose +2. Build Requirements NOW IN THE USER'S GUIDE https://escomp.github.io/CTSM/users_guide/using-clm-tools/creating-surface-datasets.html#build-requirements +3. [Building the executable](#building-the-executable) +4. [Running a Single Submission](#running-for-a-single-submission) +5. [Running for Multiple Datasets](#running-for-the-generation-of-multiple-datasets) +6. [Notes](#notes) ### Building the executable From e973c4fec2a02513ff61c4acb05eb2d415c6093d Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 8 May 2026 15:21:11 -0600 Subject: [PATCH 19/21] Make User's Guide sections consistent with last README changes --- .../using-clm-tools/creating-surface-datasets.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst index 57e0f5b818..f45afe9ec6 100644 --- a/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst +++ b/doc/source/users_guide/using-clm-tools/creating-surface-datasets.rst @@ -12,11 +12,8 @@ This tool is intended to generate fsurdat files (surface datasets) and landuse f The subset_data tool allows users to make fsurdat files from existing fsurdat files when a mesh file is unavailable. Generally, users are encouraged to use the subset_data tool for generating regional and single-point fsurdat files. -Building --------- - Build Requirements -^^^^^^^^^^^^^^^^^^ +------------------ mksurfdata_esmf is a distributed memory parallel program (using Message Passing Interface -- MPI) that utilizes both ESMF (Earth System Modelling Framework) for regridding as well as PIO (Parallel I/O) and NetCDF output. As such, libraries must be built for the following: From 2af2e5df8acbda81045d696096f6dbd7826b76bb Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 8 May 2026 15:22:09 -0600 Subject: [PATCH 20/21] Update all README references to include the .md suffix --- tools/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/README.md b/tools/README.md index d36290729b..29c8ce48f6 100644 --- a/tools/README.md +++ b/tools/README.md @@ -1,5 +1,5 @@ # CTSM Tools for Preprocessing of Input Datasets or Postprocessing of History Output -#### $CTSMROOT/tools/README +#### $CTSMROOT/tools/README.md CTSM tools for analysis of CTSM history files -- or for creation or modification of CTSM input files. @@ -35,7 +35,7 @@ II. Notes on building/running for each of the above tools: mkmapgrids and site_and_regional only contain scripts that do not need build files. - Some tools have copies of files from other directories -- see the README.filecopies + Some tools have copies of files from other directories -- see the README.filecopies.md file for more information on this. Tools may also have files with the directory name followed by namelist to provide sample namelists. @@ -43,7 +43,7 @@ II. Notes on building/running for each of the above tools: .namelist ------ Namelist to create a global file. These files are also used by the test scripts to test the tools (see the - README.testing) file. + README.testing.md) file. > [!NOTE] > Be sure to change the path of the datasets referenced by these namelists to From 0d7fb5026b2da74fcef133b1b66b96cdd4ce5292 Mon Sep 17 00:00:00 2001 From: Samuel Levis Date: Fri, 8 May 2026 15:27:59 -0600 Subject: [PATCH 21/21] Remove a couple of TODO placeholders --- .../using-clm-tools/what-are-the-clm-tools.rst | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst index dd179ec233..ce721ebbef 100644 --- a/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst +++ b/doc/source/users_guide/using-clm-tools/what-are-the-clm-tools.rst @@ -6,9 +6,6 @@ What are the CLM tools ======================== -.. todo:: - Remove references to mkprocdata_map? - There are several tools provided with CLM that allow you to create your own input datasets at resolutions you choose, or to interpolate initial conditions to a different resolution, or used to compare CLM history files between different cases. The tools are all available in the ``$CTSMROOT/tools`` directory. Most of the tools are FORTRAN stand-alone programs in their own directory, but there is also a suite of NCL scripts in the ``$CTSMROOT/tools//ncl_scripts`` directory, and some of the tools are scripts that may also call the ESMF regridding program. Some of the NCL scripts are very specialized and not meant for general use, and we won't document them here. They still contain documentation in the script itself and the README file in the tools directory. The tools produce files that can be used for CLM4.5 and |version|. They do **NOT** produce files that can be used for CLM4.0. If you need files for CLM4.0, you'll need to use a previous version of CLM. @@ -35,11 +32,3 @@ Running FORTRAN tools with namelists There is a tool that generates the namelist called ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_namelist.py``. The namelist contains information gathered from the file ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_namelist.xml``. There is also a tool that generates a jobscript for running, and this is called ``$CTSMROOT/tools/mksurfdata_esmf/gen_mksurfdata_jobscript_single.py``. ------------------------------------------------ -Running FORTRAN tools with command line options ------------------------------------------------ - -.. todo:: - Update the below, as domain files aren't needed with nuopc. - -**gen_domain** and **cprnc** run with command line arguments. The detailed sections below will give you more information on the command line arguments specific to each tool. Also running the tool without any arguments will give you a general synopsis on how to run the tool.