diff --git a/README.SSP370 b/README.SSP370 new file mode 100644 index 0000000..360fe70 --- /dev/null +++ b/README.SSP370 @@ -0,0 +1,69 @@ +------------ +# fix EAM settings for v2 +------------ +cd /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/eam/bld/namelist_files/use_cases +cp /global/cfs/cdirs/e3sm/wlin/share/fix_ssp370_ssp585_use_case_files/SSP370_eam_CMIP6.xml . +cp /global/cfs/cdirs/e3sm/wlin/share/fix_ssp370_ssp585_use_case_files/SSP585_eam_CMIP6.xml . + +------------ +May 30 2022 +------------ +The v2 SSP370 is supposed to have the same atm physics parameter settings as for the historical simulations. The only differences should be inputdata files related. + +With the current master, I created a historical (WCYCL20TR) and a SSP370 (WCYCLSSP370) cases. Due to the wrong use_case setting, the atm physics parameters is a mixture of EAMv1 and EAMv2. The atm_in files for the two test cases, as follows, differ in more than just the input files. +WCYCL20TR: /global/cscratch1/sd/wlin/E3SM_testings/E3SMv2/20220527.WCYCL20TR.ne30pg2_EC30to60E2r2/run/atm_in +WCYCLSSP370old : /global/cscratch1/sd/wlin/E3SM_testings/E3SMv2/20220527.WCYCLSSP370old.ne30pg2_EC30to60E2r2/run/atm_in +I think atm_in in your SSP370 run is the same as in WCYCLSSP370old. + +A PR to fix the use_cases for the E3SM master is in. atm_in for a new WCYCLSSP370 test case created off it is +/global/cscratch1/sd/wlin/E3SM_testings/E3SMv2/20220527.WCYCLSSP370new.ne30pg2_EC30to60E2r2/run/atm_in +The use_case file was created using a template from V1. The forcing files were correctly updated. All the settings for the tuning parameters should have been removed but were not. I am terribly sorry for this negligence. + +To rerun with the same code base but with the correct atm settings, need to replace the following file +components/eam/bld/namelist_files/use_cases/SSP370_eam_CMIP6.xml +with the one in the above bug-fix PR. I also copied the new use_case files to the following directory on NERSC +/global/cfs/cdirs/e3sm/wlin/share/fix_ssp370_ssp585_use_case_files + +----------- +ELM settings: +I'm curious about the changes to the ELM settings. Do I also need to make these changes or will the previous settings I used be ok, since they were validated in the original SSP370 PR for ELM. + + +< lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_simyr1850_2015_c211105.nc +> lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc + +> +> .false. +> .true. +> .true. + +--------------------- +Wuyin Lin: +Thanks for bringing up the ELM settings. The previous settings you used are ok. + +In general, it looks somewhat more consistent in terms of simyr starting time if using surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc. + +The two files are essentially the same and either one is ok for the currently SSP runs, as long as the same file is used for all ensemble members. They are different in five variables, four of which -- PCT_NATVEG, PCT_LAKE, PCT_GLACIER, and PCT_URBAN at 1e-7 level. The other one, PCT_NAT_PFT, has large differences, but during run time, the same variable from landuse.timeseries will be used. + +Please note that, for hybrid runs as for the SSP370 simulations, the surface data recorded in the restart file (elm.r) are supposed to take precedence over what are in fsurdat. But the elm initialization code does have an issue and it is not implemented exactly that way. As a result, simulation results would be non-BFB using one vs the other. Alan Di Vittorio is working on the fix. + +--------------------- +Jim Benedict: +All, + +A summary of the ELM bug and its implications for v2 SSP* runs can be found here: +https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3417112603/Statement+on+ELM+land+use+behavior+for+future+projection+simulations + +The bug impacts were found to be negligible (but not zero) for v2 SSP* runs, as Wuyin mentioned. Use of the "2015" fsurdat file in the updated ELM use case file should eliminate any issues. + +Jim + +------------------- +Jim Benedict: +Hi Nan, + + +Many (many!) apologies for the error in the SSP370 EAM config! When I started putting together the SSP370 configuration last year, I asked about the EAM parameter settings and the guidance then was that the values would not change, so I put it aside. I should have brought it up again once the other aerosol/GHG/etc inputs were finalized. The problem, I believe, was that the EAM parameter settings had not yet been updated to be consistent with the v2 DECK at the time I started and those changes never made it into my workflow. + + +The small silver lining is that your new runs will use the "2015" ELM fsurdat file that is more consistent with a (2015-2100) SSP simulation, based on our discovery of the small ELM bug that Wuyin noted. diff --git a/README.e3sm-setup b/README.e3sm-setup new file mode 100644 index 0000000..65f8cf0 --- /dev/null +++ b/README.e3sm-setup @@ -0,0 +1,25 @@ +Running E3SM on NERSC Cori + +directions: https://e3sm.org/model/running-e3sm/e3sm-quick-start/ +monitoring jobs: https://docs.nersc.gov/jobs/monitoring/ + +squeue -u sglanvil + +get source code + mkdir /global/homes/s/sglanvil/sourceCode/ + git clone -b maint-1.0 --recursive https://github.com/E3SM-Project/E3SM.git + +create new case +cd /global/homes/s/sglanvil/sourceCode/E3SM/cime/scripts + +./create_newcase --case MCSP_CMT_5_coupled --compset A_WCYCL20TRS_CMIP6 --res ne30_oECv3_ICG --pecount L --handle-preexisting-dirs u --mach cori-knl --output-root /global/cscratch1/sd/sglanvil/ --script-root /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled -project mp9 + + +cd /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled +./case.setup +./case.build +./case.submit + +Compsets: +cd E3SM/cime/scripts/ +./query_config --compsets diff --git a/README.samStevenson b/README.samStevenson new file mode 100644 index 0000000..63a3aa1 --- /dev/null +++ b/README.samStevenson @@ -0,0 +1,12 @@ +https://docs.google.com/document/d/1t7j5xXWrbGzSSWb0HXSksQxpKQM7FpKfpYKnqPErz_0/edit +https://docs.google.com/document/d/1mgTeLXYe2niMRloIW9cXG3GvoZmcVkKn0pVNZSO4FC8/edit + + +======= +PI control start years: +REF_CASE:20180129.DECKv1b_piControl.ne30_oEC.edison + 210 + 320 + 365 + 410 + 480 diff --git a/README.user_nl_cam b/README.user_nl_cam new file mode 100644 index 0000000..94df2c9 --- /dev/null +++ b/README.user_nl_cam @@ -0,0 +1,140 @@ +Below is a namelist that Nan has put together for the E3SMv1 simulations with additional output - could you please have a look through and make sure that it has all the needed MJO and MJO-QBO interactions output? (In particular, Jack: can you run all your diags from this? & Julie: are all the variables here that you used for the QBO-MJO plots?) + +!! h0, h1, h2, h3, h4 + + nhtfrq = 0, -24, -24, -6, -6 + + + +! monthly (h0) + + fincl1 = 'IEFLX:A' + + + +! daily 2D (h1) - Averaged + + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + +'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','OLR:A', 'RHREFHT:A', + +'UBOT:A', 'VBOT:A', 'U10:A', 'Z200:A', 'Z500:A', 'U200:A', 'V200:A', 'U850:A', 'V850:A' + + + +! daily Full-column (h2) - Averaged + + fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + + + +! 6hrly 2D (h3) + + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + + 'IVT:I','uIVT:I','vIVT:I' + + + +MAYBE: if this would be useful to you please let us know as this is very space intensive! + +! 6 hrly Full-column (h4) + + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + + +Thanks, +Yaga + + +Chih-Chieh-Jack Chen +Oct 28, 2021, 3:41 PM (4 days ago) +to Jadwiga, Julie, Richard, Brian, me + +Hi Yaga, + +Yes, the MJO diagnostics I have been running require daily FLUT, PRECT, U850, V850, U200, and V200. + +They are all in fincl2. + +Jack + + +Julie Caron +Oct 28, 2021, 3:51 PM (4 days ago) +to Jadwiga, Richard, Brian, Chih-Chieh-Jack, me + +yes, this looks like it has the mjo fields i need too ( same list as jack mentioned). i also have some code that grabs U at 50mb for the qbo i believe. but i think yaga can speak to what frequency is needed and if we want to include some standard plots of qbo in general do we have all the upper level winds at the right frequency? +thanks, +julie + + +Jadwiga (Yaga) Richter +Oct 28, 2021, 3:55 PM (4 days ago) +to Julie, Richard, Brian, Chih-Chieh-Jack, me + +Thanks Jack & Julie. + +For the QBO we mostly need monthly mean wind. For the MJO-QBO paper we used daily wind at U(50 hPa) which we could get out of the 3D daily U in h2. + +Nan: if we could add daily U(at 50 hPa) without much overhead, that would make things easier for the analysis, + +Yaga + + +Richard Neale +Oct 28, 2021, 3:57 PM (4 days ago) +to Jadwiga, Julie, Brian, Chih-Chieh-Jack, me + +A few of the tendency fields on the h0 files would be useful: ZMDT, ZMDQ, STEND_CLUBB, RVMTEND_CLUBB, MPDT, MPDQ, DCQ and DTCOND. + +I know there are already lots of fincls, but 3 hourly PRECT and maybe PRECC are always useful. + +Rich +-- +-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- + Rich Neale - rneale@ucar.edu - http://www.cgd.ucar.edu/cms/rneale - + Section Head - Atmospheric Modeling & Predictability, CGD, NCAR + P.O. Box 3000, Boulder, CO 80307-3000 + Tel (303) 497-1380 Fax (303) 497-1324 +-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- + + +Brian Medeiros +Oct 28, 2021, 4:05 PM (4 days ago) +to Jadwiga, Julie, Richard, Chih-Chieh-Jack, me + +This looks reasonable for standard MJO diagnostics. + +I was just replying similarly to Rich… the question for the QBO-MJO analysis is what kind of process-level diagnostics will be needed? I’m wondering about heating rates and measures of stability. The tendencies Rich mentioned take care of heating…. stability might be hard. The daily fields might be enough, I guess… though QRS and QRL might be needed too. + +.brian + + +Jadwiga (Yaga) Richter +Oct 28, 2021, 4:07 PM (4 days ago) +to Brian, Julie, Richard, Chih-Chieh-Jack, me + +Daily fields should be enough for the QBO-MJO. If we can add QRS and QRL to the daily's that would be great, +Yaga + + +Richard Neale +Oct 28, 2021, 4:10 PM (4 days ago) +to Brian, Jadwiga, Julie, Chih-Chieh-Jack, me + +Good point Brian, we'll need daily convective heating, moistening; copies of my h0 additions. I think daily measures of tropopause height/pressure would be good also if we don't have them. +Rich + +Not sure what you need answers for? +I say yes to the h0 variables and 3 hourly variables. +Do you need the actual tropopause variables? How about these 3 daily to capture the tropopause. + +tropopause.F90: call addfld('TROPF_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (cold point)', flag_xyfill=.True.) +tropopause.F90: call addfld('TROPF_T', horiz_only, 'A', 'K', 'Tropopause Temperature (cold point)', flag_xyfill=.True.) +tropopause.F90: call addfld('TROPF_Z', horiz_only, 'A', 'm', 'Tropopause Height (cold point)', flag_xyfill=.True.) + +All good? +Rich diff --git a/SourceMods/src.cam/README b/SourceMods/src.cam/README deleted file mode 100644 index 9fc96e6..0000000 --- a/SourceMods/src.cam/README +++ /dev/null @@ -1,9 +0,0 @@ -Put source mods for the cam library in this directory. - -WARNING: SourceMods are not kept under version control, and can easily -become out of date if changes are made to the source code on which they -are based. We only recommend using SourceMods for small, short-term -changes that just apply to one or two cases. For larger or longer-term -changes, including gradual, incremental changes towards a final -solution, we highly recommend making changes in the main source tree, -leveraging version control (git or svn). diff --git a/SourceMods/src.cam/cam_diagnostics.F90 b/SourceMods/src.cam/cam_diagnostics.F90 index 5c51ca3..5b1e9b1 100644 --- a/SourceMods/src.cam/cam_diagnostics.F90 +++ b/SourceMods/src.cam/cam_diagnostics.F90 @@ -4,24 +4,26 @@ module cam_diagnostics ! Module to compute a variety of diagnostics quantities for history files !--------------------------------------------------------------------------------- -use shr_kind_mod, only: r8 => shr_kind_r8 -use camsrfexch, only: cam_in_t, cam_out_t -use cam_control_mod, only: moist_physics -use physics_types, only: physics_state, physics_tend -use ppgrid, only: pcols, pver, begchunk, endchunk -use physics_buffer, only: physics_buffer_desc, pbuf_add_field, dtype_r8 -use physics_buffer, only: dyn_time_lvls, pbuf_get_field, pbuf_get_index, pbuf_old_tim_idx - -use cam_history, only: outfld, write_inithist, hist_fld_active, inithist_all -use constituents, only: pcnst, cnst_name, cnst_longname, cnst_cam_outfld -use constituents, only: ptendnam, dmetendnam, apcnst, bpcnst, cnst_get_ind -use dycore, only: dycore_is -use phys_control, only: phys_getopts -use wv_saturation, only: qsat, qsat_water, svp_ice -use time_manager, only: is_first_step - -use scamMod, only: single_column, wfld -use cam_abortutils, only: endrun +use shr_kind_mod, only: r8 => shr_kind_r8 +use camsrfexch, only: cam_in_t, cam_out_t +use physics_types, only: physics_state, physics_tend +use ppgrid, only: pcols, pver, pverp, begchunk, endchunk +use physics_buffer, only: physics_buffer_desc, pbuf_add_field, dtype_r8, dyn_time_lvls, & + pbuf_get_field, pbuf_get_index, pbuf_old_tim_idx + + + +use cam_history, only: outfld, write_inithist, hist_fld_active +use constituents, only: pcnst, cnst_name, cnst_longname, cnst_cam_outfld, ptendnam, dmetendnam, apcnst, bpcnst, & + cnst_get_ind +use chemistry, only: chem_is +use dycore, only: dycore_is +use phys_control, only: phys_getopts +use wv_saturation, only: qsat, qsat_water, svp_ice +use time_manager, only: is_first_step + +use scamMod, only: single_column, wfld +use cam_abortutils, only: endrun implicit none private @@ -30,21 +32,24 @@ module cam_diagnostics ! Public interfaces public :: & - diag_readnl, &! read namelist options - diag_register, &! register pbuf space - diag_init, &! initialization - diag_allocate, &! allocate memory for module variables - diag_deallocate, &! deallocate memory for module variables - diag_conv_tend_ini, &! initialize convective tendency calcs - diag_phys_writeout, &! output diagnostics of the dynamics - diag_phys_tend_writeout, &! output physics tendencies - diag_state_b4_phys_write, &! output state before physics execution - diag_conv, &! output diagnostics of convective processes - diag_surf, &! output diagnostics of the surface - diag_export, &! output export state - diag_physvar_ic, & - nsurf - + diag_register, &! register pbuf space + diag_init, &! initialization + diag_allocate, &! allocate memory for module variables + diag_deallocate, &! deallocate memory for module variables + diag_conv_tend_ini, &! initialize convective tendency calcs + diag_phys_writeout, &! output diagnostics of the dynamics + diag_phys_tend_writeout, & ! output physics tendencies + diag_state_b4_phys_write,& ! output state before physics execution + diag_conv, &! output diagnostics of convective processes + diag_surf, &! output diagnostics of the surface + diag_export, &! output export state + diag_physvar_ic, & + diag_readnl ! read namelist options + +logical, public :: inithist_all = .false. ! Flag to indicate set of fields to be + ! included on IC file + ! .false. include only required fields + ! .true. include required *and* optional fields ! Private data @@ -60,25 +65,16 @@ module cam_diagnostics character(len=8) :: diag_cnst_conv_tend = 'q_only' ! output constituent tendencies due to convection ! 'none', 'q_only' or 'all' -integer, parameter :: surf_100000 = 1 -integer, parameter :: surf_092500 = 2 -integer, parameter :: surf_085000 = 3 -integer, parameter :: surf_070000 = 4 -integer, parameter :: nsurf = 4 - logical :: history_amwg ! output the variables used by the AMWG diag package +logical :: history_verbose ! produce verbose history output logical :: history_vdiag ! output the variables used by the AMWG variability diag package logical :: history_eddy ! output the eddy variables logical :: history_budget ! output tendencies and state variables for CAM4 ! temperature, water vapor, cloud ice and cloud ! liquid budgets. integer :: history_budget_histfile_num ! output history file number for budget fields -logical :: history_waccm ! outputs typically used for WACCM -! Physics buffer indices - -integer :: psl_idx = 0 -integer :: relhum_idx = 0 +!Physics buffer indices integer :: qcwat_idx = 0 integer :: tcwat_idx = 0 integer :: lcwat_idx = 0 @@ -101,228 +97,413 @@ module cam_diagnostics integer :: tpert_idx=-1, qpert_idx=-1, pblh_idx=-1 +logical :: prog_modal_aero +contains -integer :: trefmxav_idx = -1, trefmnav_idx = -1 +! =============================================================================== -contains +subroutine diag_register + + ! Request physics buffer space for fields that persist across timesteps. + call pbuf_add_field('T_TTEND', 'global', dtype_r8, (/pcols,pver,dyn_time_lvls/), t_ttend_idx) + +end subroutine diag_register -!============================================================================== +!=============================================================================== - subroutine diag_readnl(nlfile) - use namelist_utils, only: find_group_name - use units, only: getunit, freeunit - use spmd_utils, only: masterproc, masterprocid, mpi_character, mpicom +subroutine diag_readnl(nlfile) + use namelist_utils, only: find_group_name + use units, only: getunit, freeunit + use mpishorthand + use spmd_utils, only: masterproc - character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input + character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input - ! Local variables - integer :: unitn, ierr - character(len=*), parameter :: subname = 'diag_readnl' + ! Local variables + integer :: unitn, ierr + character(len=*), parameter :: subname = 'diag_readnl' - namelist /cam_diag_opts/ diag_cnst_conv_tend - !-------------------------------------------------------------------------- + namelist /cam_diag_opts/ diag_cnst_conv_tend + !----------------------------------------------------------------------------- - if (masterproc) then + if (masterproc) then unitn = getunit() open( unitn, file=trim(nlfile), status='old' ) call find_group_name(unitn, 'cam_diag_opts', status=ierr) if (ierr == 0) then - read(unitn, cam_diag_opts, iostat=ierr) - if (ierr /= 0) then - call endrun(subname // ':: ERROR reading namelist') - end if + read(unitn, cam_diag_opts, iostat=ierr) + if (ierr /= 0) then + call endrun(subname // ':: ERROR reading namelist') + end if end if close(unitn) call freeunit(unitn) - end if - - ! Broadcast namelist variables - call mpi_bcast(diag_cnst_conv_tend, len(diag_cnst_conv_tend), mpi_character, masterprocid, mpicom, ierr) - - end subroutine diag_readnl - -!============================================================================== - - subroutine diag_register_dry() - - call pbuf_add_field('PSL', 'physpkg', dtype_r8, (/pcols/), psl_idx) - - ! Request physics buffer space for fields that persist across timesteps. - call pbuf_add_field('T_TTEND', 'global', dtype_r8, (/pcols,pver,dyn_time_lvls/), t_ttend_idx) - end subroutine diag_register_dry - - subroutine diag_register_moist() - ! Request physics buffer space for fields that persist across timesteps. - call pbuf_add_field('TREFMXAV', 'global', dtype_r8, (/pcols/), trefmxav_idx) - call pbuf_add_field('TREFMNAV', 'global', dtype_r8, (/pcols/), trefmnav_idx) - end subroutine diag_register_moist - - subroutine diag_register() - call diag_register_dry() - if (moist_physics) then - call diag_register_moist() - end if - end subroutine diag_register + end if -!============================================================================== +#ifdef SPMD + ! Broadcast namelist variables + call mpibcast(diag_cnst_conv_tend, len(diag_cnst_conv_tend), mpichar, 0, mpicom) +#endif + +end subroutine diag_readnl + +!================================================================================================ + +subroutine diag_init() + + ! Declare the history fields for which this module contains outfld calls. + + use cam_history, only: addfld, horiz_only, add_default + use constituent_burden, only: constituent_burden_init + use cam_control_mod, only: moist_physics, ideal_phys + use tidal_diag, only: tidal_diag_init + + integer :: k, m + ! Note - this is a duplication of information in ice_constants + ! Cannot put in a use statement if want to swap ice models to cice4 + integer, parameter :: plevmx = 4 ! number of subsurface levels + character(len=8), parameter :: tsnam(plevmx) = (/ 'TS1', 'TS2', 'TS3', 'TS4' /) + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: ierr + + call phys_getopts(prog_modal_aero_out = prog_modal_aero ) + + ! outfld calls in diag_phys_writeout + + call addfld ('NSTEP',horiz_only, 'A','timestep','Model timestep') + call addfld ('PHIS',horiz_only, 'I','m2/s2','Surface geopotential') + + call addfld ('PS',horiz_only, 'A','Pa','Surface pressure') + call addfld ('T',(/ 'lev' /), 'A','K','Temperature') + call addfld ('U',(/ 'lev' /), 'A','m/s','Zonal wind') + call addfld ('V',(/ 'lev' /), 'A','m/s','Meridional wind') + call addfld (cnst_name(1),(/ 'lev' /), 'A','kg/kg',cnst_longname(1)) + + ! State before physics + call addfld ('TBP',(/ 'lev' /), 'A','K','Temperature (before physics)' ) + call addfld (bpcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (before physics)') + ! State after physics + call addfld ('TAP',(/ 'lev' /), 'A','K','Temperature (after physics)' ) + call addfld ('UAP',(/ 'lev' /), 'A','m/s','Zonal wind (after physics)' ) + call addfld ('VAP',(/ 'lev' /), 'A','m/s','Meridional wind (after physics)' ) + call addfld (apcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (after physics)') + if ( dycore_is('LR') .or. dycore_is('SE') ) then + call addfld ('TFIX',horiz_only, 'A' ,'K/s','T fixer (T equivalent of Energy correction)') + call addfld ('PTTEND_RESID',(/ 'lev' /), 'A' ,'K/s',& + 'T-tendency due to BAB kluge at end of tphysac (diagnostic not part of T-budget)' ) + end if + call addfld ('TTEND_TOT',(/ 'lev' /), 'A','K/s' ,'Total temperature tendency' ) - subroutine diag_init_dry(pbuf2d) - ! Declare the history fields for which this module contains outfld calls. - - use cam_history, only: addfld, add_default, horiz_only - use cam_history, only: register_vector_field - use constituent_burden, only: constituent_burden_init - use physics_buffer, only: pbuf_set_field - use tidal_diag, only: tidal_diag_init - - type(physics_buffer_desc), pointer, intent(in) :: pbuf2d(:,:) - - integer :: k, m - integer :: ierr - - ! outfld calls in diag_phys_writeout - call addfld (cnst_name(1), (/ 'lev' /), 'A', 'kg/kg', cnst_longname(1)) - call addfld ('NSTEP', horiz_only, 'A', 'timestep', 'Model timestep') - call addfld ('PHIS', horiz_only, 'I', 'm2/s2', 'Surface geopotential') - - call addfld ('PS', horiz_only, 'A', 'Pa', 'Surface pressure') - call addfld ('T', (/ 'lev' /), 'A', 'K', 'Temperature') - call addfld ('U', (/ 'lev' /), 'A', 'm/s', 'Zonal wind') - call addfld ('V', (/ 'lev' /), 'A', 'm/s', 'Meridional wind') - - call register_vector_field('U','V') - - ! State before physics - call addfld ('TBP', (/ 'lev' /), 'A','K', 'Temperature (before physics)') - call addfld (bpcnst(1), (/ 'lev' /), 'A','kg/kg', trim(cnst_longname(1))//' (before physics)') - ! State after physics - call addfld ('TAP', (/ 'lev' /), 'A','K', 'Temperature (after physics)' ) - call addfld ('UAP', (/ 'lev' /), 'A','m/s', 'Zonal wind (after physics)' ) - call addfld ('VAP', (/ 'lev' /), 'A','m/s', 'Meridional wind (after physics)' ) - - call register_vector_field('UAP','VAP') - - call addfld (apcnst(1), (/ 'lev' /), 'A','kg/kg', trim(cnst_longname(1))//' (after physics)') - if ( dycore_is('LR') .or. dycore_is('SE') ) then - call addfld ('TFIX', horiz_only, 'A', 'K/s', 'T fixer (T equivalent of Energy correction)') - end if - call addfld ('TTEND_TOT', (/ 'lev' /), 'A', 'K/s', 'Total temperature tendency') - - call addfld ('Z3', (/ 'lev' /), 'A', 'm', 'Geopotential Height (above sea level)') - call addfld ('Z1000', horiz_only, 'A', 'm', 'Geopotential Z at 1000 mbar pressure surface') - call addfld ('Z700', horiz_only, 'A', 'm', 'Geopotential Z at 700 mbar pressure surface') - call addfld ('Z500', horiz_only, 'A', 'm', 'Geopotential Z at 500 mbar pressure surface') - call addfld ('Z300', horiz_only, 'A', 'm', 'Geopotential Z at 300 mbar pressure surface') - call addfld ('Z200', horiz_only, 'A', 'm', 'Geopotential Z at 200 mbar pressure surface') - call addfld ('Z100', horiz_only, 'A', 'm', 'Geopotential Z at 100 mbar pressure surface') - call addfld ('Z050', horiz_only, 'A', 'm', 'Geopotential Z at 50 mbar pressure surface') - - call addfld ('ZZ', (/ 'lev' /), 'A', 'm2', 'Eddy height variance' ) - call addfld ('VZ', (/ 'lev' /), 'A', 'm2/s', 'Meridional transport of geopotential height') - call addfld ('VT', (/ 'lev' /), 'A', 'K m/s ', 'Meridional heat transport') - call addfld ('VU', (/ 'lev' /), 'A', 'm2/s2', 'Meridional flux of zonal momentum' ) - call addfld ('VV', (/ 'lev' /), 'A', 'm2/s2', 'Meridional velocity squared' ) - call addfld ('OMEGAV', (/ 'lev' /), 'A', 'm Pa/s2 ', 'Vertical flux of meridional momentum' ) - call addfld ('OMGAOMGA', (/ 'lev' /), 'A', 'Pa2/s2', 'Vertical flux of vertical momentum' ) - - call addfld ('UU', (/ 'lev' /), 'A', 'm2/s2', 'Zonal velocity squared' ) - call addfld ('WSPEED', (/ 'lev' /), 'X', 'm/s', 'Horizontal total wind speed maximum' ) - call addfld ('WSPDSRFMX', horiz_only, 'X', 'm/s', 'Horizontal total wind speed maximum at the surface' ) - call addfld ('WSPDSRFAV', horiz_only, 'A', 'm/s', 'Horizontal total wind speed average at the surface' ) - - call addfld ('OMEGA', (/ 'lev' /), 'A', 'Pa/s', 'Vertical velocity (pressure)') - call addfld ('OMEGAT', (/ 'lev' /), 'A', 'K Pa/s ', 'Vertical heat flux' ) - call addfld ('OMEGAU', (/ 'lev' /), 'A', 'm Pa/s2 ', 'Vertical flux of zonal momentum' ) - call addfld ('OMEGA850', horiz_only, 'A', 'Pa/s', 'Vertical velocity at 850 mbar pressure surface') - call addfld ('OMEGA500', horiz_only, 'A', 'Pa/s', 'Vertical velocity at 500 mbar pressure surface') - - call addfld ('PSL', horiz_only, 'A', 'Pa','Sea level pressure') - - call addfld ('T1000', horiz_only, 'A', 'K','Temperature at 1000 mbar pressure surface') - call addfld ('T925', horiz_only, 'A', 'K','Temperature at 925 mbar pressure surface') - call addfld ('T850', horiz_only, 'A', 'K','Temperature at 850 mbar pressure surface') - call addfld ('T700', horiz_only, 'A', 'K','Temperature at 700 mbar pressure surface') - call addfld ('T500', horiz_only, 'A', 'K','Temperature at 500 mbar pressure surface') - call addfld ('T400', horiz_only, 'A', 'K','Temperature at 400 mbar pressure surface') - call addfld ('T300', horiz_only, 'A', 'K','Temperature at 300 mbar pressure surface') - call addfld ('T200', horiz_only, 'A', 'K','Temperature at 200 mbar pressure surface') - call addfld ('T010', horiz_only, 'A', 'K','Temperature at 10 mbar pressure surface') - - call addfld ('T7001000', horiz_only, 'A', 'K','Temperature difference 700 mb - 1000 mb') - call addfld ('TH7001000', horiz_only, 'A', 'K','Theta difference 700 mb - 1000 mb') - call addfld ('THE7001000', horiz_only, 'A', 'K','ThetaE difference 700 mb - 1000 mb') - - call addfld ('T8501000', horiz_only, 'A', 'K','Temperature difference 850 mb - 1000 mb') - call addfld ('TH8501000', horiz_only, 'A', 'K','Theta difference 850 mb - 1000 mb') - call addfld ('T9251000', horiz_only, 'A', 'K','Temperature difference 925 mb - 1000 mb') - call addfld ('TH9251000', horiz_only, 'A', 'K','Theta difference 925 mb - 1000 mb') - - call addfld ('TT', (/ 'lev' /), 'A', 'K2','Eddy temperature variance' ) - - call addfld ('U850', horiz_only, 'A', 'm/s','Zonal wind at 850 mbar pressure surface') - call addfld ('U500', horiz_only, 'A', 'm/s','Zonal wind at 500 mbar pressure surface') - call addfld ('U250', horiz_only, 'A', 'm/s','Zonal wind at 250 mbar pressure surface') - call addfld ('U200', horiz_only, 'A', 'm/s','Zonal wind at 200 mbar pressure surface') - call addfld ('U010', horiz_only, 'A', 'm/s','Zonal wind at 10 mbar pressure surface') - call addfld ('V850', horiz_only, 'A', 'm/s','Meridional wind at 850 mbar pressure surface') - call addfld ('V500', horiz_only, 'A', 'm/s','Meridional wind at 500 mbar pressure surface') - call addfld ('V250', horiz_only, 'A', 'm/s','Meridional wind at 250 mbar pressure surface') - call addfld ('V200', horiz_only, 'A', 'm/s','Meridional wind at 200 mbar pressure surface') - - call register_vector_field('U850', 'V850') - call register_vector_field('U500', 'V500') - call register_vector_field('U250', 'V250') - call register_vector_field('U200', 'V200') - - call addfld ('UBOT', horiz_only, 'A', 'm/s','Lowest model level zonal wind') - call addfld ('VBOT', horiz_only, 'A', 'm/s','Lowest model level meridional wind') - call register_vector_field('UBOT', 'VBOT') - - call addfld ('ZBOT', horiz_only, 'A', 'm','Lowest model level height') - - call addfld ('ATMEINT', horiz_only, 'A', 'J/m2','Vertically integrated total atmospheric energy ') - - if (history_amwg) then + ! column burdens for all constituents except water vapor + call constituent_burden_init + + call addfld ('Z3',(/ 'lev' /), 'A','m','Geopotential Height (above sea level)') + call addfld ('Z1000',horiz_only, 'A','m','Geopotential Z at 1000 mbar pressure surface') + call addfld ('Z975',horiz_only, 'A','m','Geopotential Z at 975 mbar pressure surface') + call addfld ('Z950',horiz_only, 'A','m','Geopotential Z at 950 mbar pressure surface') + call addfld ('Z925',horiz_only, 'A','m','Geopotential Z at 925 mbar pressure surface') + call addfld ('Z900',horiz_only, 'A','m','Geopotential Z at 900 mbar pressure surface') + call addfld ('Z850',horiz_only, 'A','m','Geopotential Z at 850 mbar pressure surface') + call addfld ('Z800',horiz_only, 'A','m','Geopotential Z at 800 mbar pressure surface') + call addfld ('Z700',horiz_only, 'A','m','Geopotential Z at 700 mbar pressure surface') + call addfld ('Z600',horiz_only, 'A','m','Geopotential Z at 600 mbar pressure surface') + call addfld ('Z500',horiz_only, 'A','m','Geopotential Z at 500 mbar pressure surface') + call addfld ('Z400',horiz_only, 'A','m','Geopotential Z at 400 mbar pressure surface') + call addfld ('Z300',horiz_only, 'A','m','Geopotential Z at 300 mbar pressure surface') + call addfld ('Z200',horiz_only, 'A','m','Geopotential Z at 200 mbar pressure surface') + call addfld ('Z100',horiz_only, 'A','m','Geopotential Z at 100 mbar pressure surface') + call addfld ('Z050',horiz_only, 'A','m','Geopotential Z at 50 mbar pressure surface') + call addfld ('Z010',horiz_only, 'A','m','Geopotential Z at 10 mbar pressure surface') + + call addfld ('ZZ',(/ 'lev' /), 'A','m2','Eddy height variance' ) + call addfld ('VZ',(/ 'lev' /), 'A','m2/s','Meridional transport of geopotential energy') + call addfld ('VT',(/ 'lev' /), 'A','K m/s ','Meridional heat transport') + call addfld ('VU',(/ 'lev' /), 'A','m2/s2','Meridional flux of zonal momentum' ) + call addfld ('VV',(/ 'lev' /), 'A','m2/s2','Meridional velocity squared' ) + + if(prog_modal_aero) then !Only for prognostic aerosols + call addfld ('bc_a1_2',(/ 'lev' /), 'A','kg2/kg2','bc_a1 squared') + call addfld ('dst_a1_2',(/ 'lev' /), 'A','kg2/kg2','dst_a1 squared') + call addfld ('dst_a3_2',(/ 'lev' /), 'A','kg2/kg2','dst_a3 squared') + call addfld ('ncl_a1_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a1 squared') + call addfld ('ncl_a2_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a2 squared') + call addfld ('ncl_a3_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a3 squared') + call addfld ('so4_a1_2',(/ 'lev' /), 'A','kg2/kg2','so4_a1 squared') + call addfld ('so4_a2_2',(/ 'lev' /), 'A','kg2/kg2','so4_a2 squared') + call addfld ('so4_a3_2',(/ 'lev' /), 'A','kg2/kg2','so4_a3 squared') + call addfld ('soa_a1_2',(/ 'lev' /), 'A','kg2/kg2','soa_a1 squared') + call addfld ('soa_a2_2',(/ 'lev' /), 'A','kg2/kg2','soa_a2 squared') + call addfld ('pom_a1_2',(/ 'lev' /), 'A','kg2/kg2','pom_a1 squared') + call addfld ('Vbc_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional bc_a1 transport') + call addfld ('Vdst_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a1 transport') + call addfld ('Vdst_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a3 transport') + call addfld ('Vncl_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a1 transport') + call addfld ('Vncl_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a2 transport') + call addfld ('Vncl_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a3 transport') + call addfld ('Vso4_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a1 transport') + call addfld ('Vso4_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a2 transport') + call addfld ('Vso4_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a3 transport') + call addfld ('Vsoa_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a1 transport') + call addfld ('Vsoa_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a2 transport') + call addfld ('Vpom_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional pom_a1 transport') + endif + call addfld ('VQ',(/ 'lev' /), 'A','m/skg/kg','Meridional water transport') + call addfld ('QQ',(/ 'lev' /), 'A','kg2/kg2','Eddy moisture variance') + call addfld ('OMEGAV',(/ 'lev' /) ,'A','m Pa/s2 ','Vertical flux of meridional momentum' ) + call addfld ('OMGAOMGA',(/ 'lev' /) ,'A','Pa2/s2','Vertical flux of vertical momentum' ) + call addfld ('OMEGAQ',(/ 'lev' /) ,'A','kgPa/kgs','Vertical water transport' ) + + call addfld ('UU',(/ 'lev' /), 'A','m2/s2','Zonal velocity squared' ) + call addfld ('WSPEED',(/ 'lev' /), 'X','m/s','Horizontal total wind speed maximum' ) + call addfld ('WSPDSRFMX',horiz_only, 'X','m/s','Horizontal total wind speed maximum at the surface' ) + call addfld ('WSPDSRFAV',horiz_only, 'A','m/s','Horizontal total wind speed average at the surface' ) + + call addfld ('OMEGA',(/ 'lev' /), 'A','Pa/s','Vertical velocity (pressure)') + call addfld ('OMEGAT',(/ 'lev' /), 'A','K Pa/s ','Vertical heat flux' ) + call addfld ('OMEGAU',(/ 'lev' /), 'A','m Pa/s2 ','Vertical flux of zonal momentum' ) + call addfld ('OMEGA1000',horiz_only, 'A','Pa/s','Vertical velocity at 1000 mbar pressure surface') + call addfld ('OMEGA975',horiz_only, 'A','Pa/s','Vertical velocity at 975 mbar pressure surface') + call addfld ('OMEGA950',horiz_only, 'A','Pa/s','Vertical velocity at 950 mbar pressure surface') + call addfld ('OMEGA925',horiz_only, 'A','Pa/s','Vertical velocity at 925 mbar pressure surface') + call addfld ('OMEGA900',horiz_only, 'A','Pa/s','Vertical velocity at 900 mbar pressure surface') + call addfld ('OMEGA850',horiz_only, 'A','Pa/s','Vertical velocity at 850 mbar pressure surface') + call addfld ('OMEGA800',horiz_only, 'A','Pa/s','Vertical velocity at 800 mbar pressure surface') + call addfld ('OMEGA700',horiz_only, 'A','Pa/s','Vertical velocity at 700 mbar pressure surface') + call addfld ('OMEGA600',horiz_only, 'A','Pa/s','Vertical velocity at 600 mbar pressure surface') + call addfld ('OMEGA500',horiz_only, 'A','Pa/s','Vertical velocity at 500 mbar pressure surface') + call addfld ('OMEGA400',horiz_only, 'A','Pa/s','Vertical velocity at 400 mbar pressure surface') + call addfld ('OMEGA300',horiz_only, 'A','Pa/s','Vertical velocity at 300 mbar pressure surface') + call addfld ('OMEGA200',horiz_only, 'A','Pa/s','Vertical velocity at 200 mbar pressure surface') + call addfld ('OMEGA100',horiz_only, 'A','Pa/s','Vertical velocity at 100 mbar pressure surface') + call addfld ('OMEGABOT',horiz_only, 'A','Pa/s','Lowest model level vertical velocity') + + call addfld ('RH1000',horiz_only, 'A','%','Relative humidity at 1000 mbar pressure surface') + call addfld ('RH975',horiz_only, 'A','%','Relative humidity at 975 mbar pressure surface') + call addfld ('RH950',horiz_only, 'A','%','Relative humidity at 950 mbar pressure surface') + call addfld ('RH925',horiz_only, 'A','%','Relative humidity at 925 mbar pressure surface') + call addfld ('RH900',horiz_only, 'A','%','Relative humidity at 900 mbar pressure surface') + call addfld ('RH850',horiz_only, 'A','%','Relative humidity at 850 mbar pressure surface') + call addfld ('RH800',horiz_only, 'A','%','Relative humidity at 800 mbar pressure surface') + call addfld ('RH700',horiz_only, 'A','%','Relative humidity at 700 mbar pressure surface') + call addfld ('RH600',horiz_only, 'A','%','Relative humidity at 600 mbar pressure surface') + call addfld ('RH500',horiz_only, 'A','%','Relative humidity at 500 mbar pressure surface') + call addfld ('RH400',horiz_only, 'A','%','Relative humidity at 400 mbar pressure surface') + call addfld ('RH300',horiz_only, 'A','%','Relative humidity at 300 mbar pressure surface') + call addfld ('RH200',horiz_only, 'A','%','Relative humidity at 200 mbar pressure surface') + call addfld ('RH100',horiz_only, 'A','%','Relative humidity at 100 mbar pressure surface') + call addfld ('RHBOT',horiz_only, 'A','%','Lowest model level relative humidity') + + call addfld ('MQ',(/ 'lev' /), 'A','kg/m2','Water vapor mass in layer') + call addfld ('TMQ',horiz_only, 'A','kg/m2','Total (vertically integrated) precipitable water') + +! cas fields + call addfld ('IVT',horiz_only, 'A', 'kg/m/s','Total (vertically integrated) vapor transport') + call addfld ('uIVT',horiz_only, 'A', 'kg/m/s','u-component (vertically integrated) vapor transport') + call addfld ('vIVT',horiz_only, 'A', 'kg/m/s','v-component (vertically integrated) vapor transport') +! end cas fields + + call addfld ('TUQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) zonal water flux') + call addfld ('TVQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) meridional water flux') + call addfld ('TUH',horiz_only, 'A','W/m', 'Total (vertically integrated) zonal MSE flux') + call addfld ('TVH',horiz_only, 'A','W/m', 'Total (vertically integrated) meridional MSE flux') + call addfld ('DTENDTH', horiz_only, 'A', 'W/m2', 'Dynamic Tendency of Total (vertically integrated) moist static energy') + call addfld ('DTENDTQ', horiz_only, 'A', 'kg/m2/s','Dynamic Tendency of Total (vertically integrated) specific humidity') + call addfld ('RELHUM',(/ 'lev' /), 'A','percent','Relative humidity') + call addfld ('RHW',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to liquid') + call addfld ('RHI',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to ice') + call addfld ('RHCFMIP',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to water above 273 K, ice below 273 K') + call addfld ('PSL',horiz_only, 'A','Pa','Sea level pressure') + + call addfld ('T850',horiz_only, 'A','K','Temperature at 850 mbar pressure surface') + call addfld ('T500',horiz_only, 'A','K','Temperature at 500 mbar pressure surface') + call addfld ('T300',horiz_only, 'A','K','Temperature at 300 mbar pressure surface') + call addfld ('T200',horiz_only, 'A','K','Temperature at 200 mbar pressure surface') + call addfld ('Q850',horiz_only, 'A','kg/kg','Specific Humidity at 850 mbar pressure surface') + call addfld ('Q500',horiz_only, 'A','kg/kg','Specific Humidity at 500 mbar pressure surface') + call addfld ('Q300',horiz_only, 'A','kg/kg','Specific Humidity at 300 mbar pressure surface') + call addfld ('Q200',horiz_only, 'A','kg/kg','Specific Humidity at 200 mbar pressure surface') + call addfld ('U1000',horiz_only, 'A','m/s','Zonal wind at 1000 mbar pressure surface') + call addfld ('U975',horiz_only, 'A','m/s','Zonal wind at 975 mbar pressure surface') + call addfld ('U950',horiz_only, 'A','m/s','Zonal wind at 950 mbar pressure surface') + call addfld ('U925',horiz_only, 'A','m/s','Zonal wind at 925 mbar pressure surface') + call addfld ('U900',horiz_only, 'A','m/s','Zonal wind at 900 mbar pressure surface') + call addfld ('U850',horiz_only, 'A','m/s','Zonal wind at 850 mbar pressure surface') + call addfld ('U800',horiz_only, 'A','m/s','Zonal wind at 800 mbar pressure surface') + call addfld ('U700',horiz_only, 'A','m/s','Zonal wind at 700 mbar pressure surface') + call addfld ('U600',horiz_only, 'A','m/s','Zonal wind at 600 mbar pressure surface') + call addfld ('U500',horiz_only, 'A','m/s','Zonal wind at 500 mbar pressure surface') + call addfld ('U400',horiz_only, 'A','m/s','Zonal wind at 400 mbar pressure surface') + call addfld ('U300',horiz_only, 'A','m/s','Zonal wind at 300 mbar pressure surface') + call addfld ('U250',horiz_only, 'A','m/s','Zonal wind at 250 mbar pressure surface') + call addfld ('U200',horiz_only, 'A','m/s','Zonal wind at 200 mbar pressure surface') + call addfld ('U100',horiz_only, 'A','m/s','Zonal wind at 100 mbar pressure surface') + call addfld ('U050',horiz_only, 'A','m/s','Zonal wind at 50 mbar pressure surface') + call addfld ('U010',horiz_only, 'A','m/s','Zonal wind at 10 mbar pressure surface') + call addfld ('V1000',horiz_only, 'A','m/s','Meridional wind at 1000 mbar pressure surface') + call addfld ('V975',horiz_only, 'A','m/s','Meridional wind at 975 mbar pressure surface') + call addfld ('V950',horiz_only, 'A','m/s','Meridional wind at 950 mbar pressure surface') + call addfld ('V925',horiz_only, 'A','m/s','Meridional wind at 925 mbar pressure surface') + call addfld ('V900',horiz_only, 'A','m/s','Meridional wind at 900 mbar pressure surface') + call addfld ('V850',horiz_only, 'A','m/s','Meridional wind at 850 mbar pressure surface') + call addfld ('V800',horiz_only, 'A','m/s','Meridional wind at 800 mbar pressure surface') + call addfld ('V700',horiz_only, 'A','m/s','Meridional wind at 700 mbar pressure surface') + call addfld ('V600',horiz_only, 'A','m/s','Meridional wind at 600 mbar pressure surface') + call addfld ('V500',horiz_only, 'A','m/s','Meridional wind at 500 mbar pressure surface') + call addfld ('V400',horiz_only, 'A','m/s','Meridional wind at 400 mbar pressure surface') + call addfld ('V300',horiz_only, 'A','m/s','Meridional wind at 300 mbar pressure surface') + call addfld ('V250',horiz_only, 'A','m/s','Meridional wind at 250 mbar pressure surface') + call addfld ('V200',horiz_only, 'A','m/s','Meridional wind at 200 mbar pressure surface') + call addfld ('V100',horiz_only, 'A','m/s','Meridional wind at 100 mbar pressure surface') + call addfld ('V050',horiz_only, 'A','m/s','Meridional wind at 50 mbar pressure surface') + call addfld ('V010',horiz_only, 'A','m/s','Meridional wind at 10 mbar pressure surface') + + call addfld ('TT',(/ 'lev' /), 'A','K2','Eddy temperature variance' ) + + call addfld ('UBOT',horiz_only, 'A','m/s','Lowest model level zonal wind') + call addfld ('VBOT',horiz_only, 'A','m/s','Lowest model level meridional wind') + call addfld ('QBOT',horiz_only, 'A','kg/kg','Lowest model level water vapor mixing ratio') + call addfld ('ZBOT',horiz_only, 'A','m','Lowest model level height') + + call addfld ('ATMEINT',horiz_only, 'A','J/m2','Vertically integrated total atmospheric energy ') + + call addfld ('T1000',horiz_only, 'A','K','Temperature at 1000 mbar pressure surface') + call addfld ('T975',horiz_only, 'A','K','Temperature at 975 mbar pressure surface') + call addfld ('T950',horiz_only, 'A','K','Temperature at 950 mbar pressure surface') + call addfld ('T925',horiz_only, 'A','K','Temperature at 925 mbar pressure surface') + call addfld ('T900',horiz_only, 'A','K','Temperature at 900 mbar pressure surface') + call addfld ('T800',horiz_only, 'A','K','Temperature at 800 mbar pressure surface') + call addfld ('T700',horiz_only, 'A','K','Temperature at 700 mbar pressure surface') + call addfld ('T600',horiz_only, 'A','K','Temperature at 600 mbar pressure surface') + call addfld ('T400',horiz_only, 'A','K','Temperature at 400 mbar pressure surface') + call addfld ('T100',horiz_only, 'A','K','Temperature at 100 mbar pressure surface') + call addfld ('T050',horiz_only, 'A','K','Temperature at 50 mbar pressure surface') + call addfld ('T010',horiz_only, 'A','K','Temperature at 10 mbar pressure surface') + call addfld ('Q1000',horiz_only, 'A','kg/kg','Specific Humidity at 1000 mbar pressure surface') + call addfld ('Q975',horiz_only, 'A','kg/kg','Specific Humidity at 975 mbar pressure surface') + call addfld ('Q950',horiz_only, 'A','kg/kg','Specific Humidity at 950 mbar pressure surface') + call addfld ('Q925',horiz_only, 'A','kg/kg','Specific Humidity at 925 mbar pressure surface') + call addfld ('Q900',horiz_only, 'A','kg/kg','Specific Humidity at 900 mbar pressure surface') + call addfld ('Q800',horiz_only, 'A','kg/kg','Specific Humidity at 800 mbar pressure surface') + call addfld ('Q700',horiz_only, 'A','kg/kg','Specific Humidity at 700 mbar pressure surface') + call addfld ('Q600',horiz_only, 'A','kg/kg','Specific Humidity at 600 mbar pressure surface') + call addfld ('Q400',horiz_only, 'A','kg/kg','Specific Humidity at 400 mbar pressure surface') + call addfld ('Q100',horiz_only, 'A','kg/kg','Specific Humidity at 100 mbar pressure surface') + call addfld ('Q050',horiz_only, 'A','kg/kg','Specific Humidity at 050 mbar pressure surface') + call addfld ('Q010',horiz_only, 'A','kg/kg','Specific Humidity at 010 mbar pressure surface') + + call addfld ('T7001000',horiz_only, 'A','K','Temperature difference 700 mb - 1000 mb') + call addfld ('TH7001000',horiz_only, 'A','K','Theta difference 700 mb - 1000 mb') + call addfld ('THE7001000',horiz_only, 'A','K','ThetaE difference 700 mb - 1000 mb') + + call addfld ('T8501000',horiz_only, 'A','K','Temperature difference 850 mb - 1000 mb') + call addfld ('TH8501000',horiz_only, 'A','K','Theta difference 850 mb - 1000 mb') + call addfld ('THE8501000',horiz_only, 'A','K','ThetaE difference 850 mb - 1000 mb') + call addfld ('T9251000',horiz_only, 'A','K','Temperature difference 925 mb - 1000 mb') + call addfld ('TH9251000',horiz_only, 'A','K','Theta difference 925 mb - 1000 mb') + call addfld ('THE9251000',horiz_only, 'A','K','ThetaE difference 925 mb - 1000 mb') + + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call addfld('QRS', (/ 'lev' /), 'A', 'K/s', 'Solar heating rate') + end if + + ! ---------------------------- + ! determine default variables + ! ---------------------------- + call phys_getopts(history_amwg_out = history_amwg , & + history_verbose_out = history_verbose , & + history_vdiag_out = history_vdiag , & + history_eddy_out = history_eddy , & + history_budget_out = history_budget , & + history_budget_histfile_num_out = history_budget_histfile_num) + + if (history_amwg) then call add_default ('PHIS ' , 1, ' ') call add_default ('PS ' , 1, ' ') call add_default ('T ' , 1, ' ') call add_default ('U ' , 1, ' ') call add_default ('V ' , 1, ' ') + call add_default (cnst_name(1), 1, ' ') call add_default ('Z3 ' , 1, ' ') call add_default ('OMEGA ' , 1, ' ') call add_default ('VT ', 1, ' ') call add_default ('VU ', 1, ' ') call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') + + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('Vbc_a1 ', 1, ' ') + call add_default ('Vdst_a1 ', 1, ' ') + call add_default ('Vdst_a3 ', 1, ' ') + call add_default ('Vncl_a1 ', 1, ' ') + call add_default ('Vncl_a2 ', 1, ' ') + call add_default ('Vncl_a3 ', 1, ' ') + call add_default ('Vso4_a1 ', 1, ' ') + call add_default ('Vso4_a2 ', 1, ' ') + call add_default ('Vso4_a3 ', 1, ' ') + call add_default ('Vsoa_a1 ', 1, ' ') + call add_default ('Vsoa_a2 ', 1, ' ') + call add_default ('Vpom_a1 ', 1, ' ') + endif call add_default ('UU ', 1, ' ') call add_default ('OMEGAT ', 1, ' ') + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('bc_a1_2 ', 1, ' ') + call add_default ('dst_a1_2', 1, ' ') + call add_default ('dst_a3_2', 1, ' ') + call add_default ('ncl_a1_2', 1, ' ') + call add_default ('ncl_a2_2', 1, ' ') + call add_default ('ncl_a3_2', 1, ' ') + call add_default ('so4_a1_2', 1, ' ') + call add_default ('so4_a2_2', 1, ' ') + call add_default ('so4_a3_2', 1, ' ') + call add_default ('soa_a1_2', 1, ' ') + call add_default ('soa_a2_2', 1, ' ') + call add_default ('pom_a1_2', 1, ' ') + endif + call add_default ('TMQ ', 1, ' ') call add_default ('PSL ', 1, ' ') - end if + if (moist_physics) then + call add_default ('RELHUM ', 1, ' ') + end if - if (history_vdiag) then - call add_default ('U200', 2, ' ') - call add_default ('V200', 2, ' ') - call add_default ('U850', 2, ' ') - call add_default ('U200', 3, ' ') - call add_default ('U850', 3, ' ') - call add_default ('OMEGA500', 3, ' ') - end if + ! For Tier 1b global water cycle diagostics + call add_default ('TUQ ', 1, ' ') + call add_default ('TVQ ', 1, ' ') + call add_default ('TUH ', 1, ' ') + call add_default ('TVH ', 1, ' ') + call add_default ('DTENDTQ', 1, ' ') + call add_default ('DTENDTH', 1, ' ') + end if + + if (history_vdiag) then + call add_default ('U200', 2, ' ') + call add_default ('V200', 2, ' ') + call add_default ('U850', 2, ' ') + call add_default ('U200', 3, ' ') + call add_default ('U850', 3, ' ') + call add_default ('OMEGA500', 3, ' ') + end if - if (history_eddy) then + if (history_eddy) then call add_default ('VT ', 1, ' ') call add_default ('VU ', 1, ' ') call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') call add_default ('UU ', 1, ' ') call add_default ('OMEGAT ', 1, ' ') + call add_default ('OMEGAQ ', 1, ' ') call add_default ('OMEGAU ', 1, ' ') call add_default ('OMEGAV ', 1, ' ') - endif + endif - if ( history_budget ) then + if ( history_budget ) then call add_default ('PHIS ' , history_budget_histfile_num, ' ') call add_default ('PS ' , history_budget_histfile_num, ' ') call add_default ('T ' , history_budget_histfile_num, ' ') call add_default ('U ' , history_budget_histfile_num, ' ') call add_default ('V ' , history_budget_histfile_num, ' ') + call add_default (cnst_name(1), history_budget_histfile_num, ' ') call add_default ('TTEND_TOT' , history_budget_histfile_num, ' ') ! State before physics (FV) @@ -331,612 +512,459 @@ subroutine diag_init_dry(pbuf2d) ! State after physics (FV) call add_default ('TAP ' , history_budget_histfile_num, ' ') call add_default ('UAP ' , history_budget_histfile_num, ' ') - call add_default ('VAP ' , history_budget_histfile_num, ' ') + call add_default ('VAP ' , history_budget_histfile_num, ' ') call add_default (apcnst(1) , history_budget_histfile_num, ' ') if ( dycore_is('LR') .or. dycore_is('SE') ) then - call add_default ('TFIX ' , history_budget_histfile_num, ' ') + call add_default ('TFIX ' , history_budget_histfile_num, ' ') + call add_default ('PTTEND_RESID', history_budget_histfile_num, ' ') end if - end if - - if (history_waccm) then - call add_default ('PHIS', 7, ' ') - call add_default ('PS', 7, ' ') - call add_default ('PSL', 7, ' ') - end if + end if - ! outfld calls in diag_phys_tend_writeout - call addfld ('PTTEND', (/ 'lev' /), 'A', 'K/s','T total physics tendency' ) - if ( history_budget ) then - call add_default ('PTTEND' , history_budget_histfile_num, ' ') - end if + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call add_default('QRS ', 1, ' ') + end if - ! create history variables for fourier coefficients of the diurnal - ! and semidiurnal tide in T, U, V, and Z3 - call tidal_diag_init() - - ! - ! energy diagnostics - ! - call addfld ('SE_pBF', horiz_only, 'A', 'J/m2','Dry Static Energy before energy fixer') - call addfld ('SE_pBP', horiz_only, 'A', 'J/m2','Dry Static Energy before parameterizations') - call addfld ('SE_pAP', horiz_only, 'A', 'J/m2','Dry Static Energy after parameterizations') - call addfld ('SE_pAM', horiz_only, 'A', 'J/m2','Dry Static Energy after dry mass correction') - - call addfld ('KE_pBF', horiz_only, 'A', 'J/m2','Kinetic Energy before energy fixer') - call addfld ('KE_pBP', horiz_only, 'A', 'J/m2','Kinetic Energy before parameterizations') - call addfld ('KE_pAP', horiz_only, 'A', 'J/m2','Kinetic Energy after parameterizations') - call addfld ('KE_pAM', horiz_only, 'A', 'J/m2','Kinetic Energy after dry mass correction') - - call addfld ('TT_pBF', horiz_only, 'A', 'kg/m2','Total column test tracer before energy fixer') - call addfld ('TT_pBP', horiz_only, 'A', 'kg/m2','Total column test tracer before parameterizations') - call addfld ('TT_pAP', horiz_only, 'A', 'kg/m2','Total column test tracer after parameterizations') - call addfld ('TT_pAM', horiz_only, 'A', 'kg/m2','Total column test tracer after dry mass correction') - - call addfld ('WV_pBF', horiz_only, 'A', 'kg/m2','Total column water vapor before energy fixer') - call addfld ('WV_pBP', horiz_only, 'A', 'kg/m2','Total column water vapor before parameterizations') - call addfld ('WV_pAP', horiz_only, 'A', 'kg/m2','Total column water vapor after parameterizations') - call addfld ('WV_pAM', horiz_only, 'A', 'kg/m2','Total column water vapor after dry mass correction') - - call addfld ('WL_pBF', horiz_only, 'A', 'kg/m2','Total column cloud water before energy fixer') - call addfld ('WL_pBP', horiz_only, 'A', 'kg/m2','Total column cloud water before parameterizations') - call addfld ('WL_pAP', horiz_only, 'A', 'kg/m2','Total column cloud water after parameterizations') - call addfld ('WL_pAM', horiz_only, 'A', 'kg/m2','Total column cloud water after dry mass correction') - - call addfld ('WI_pBF', horiz_only, 'A', 'kg/m2','Total column cloud ice before energy fixer') - call addfld ('WI_pBP', horiz_only, 'A', 'kg/m2','Total column cloud ice before parameterizations') - call addfld ('WI_pAP', horiz_only, 'A', 'kg/m2','Total column cloud ice after parameterizations') - call addfld ('WI_pAM', horiz_only, 'A', 'kg/m2','Total column cloud ice after dry mass correction') - ! - ! Axial Angular Momentum diagnostics - ! - call addfld ('MR_pBF', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column wind axial angular momentum before energy fixer') - call addfld ('MR_pBP', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column wind axial angular momentum before parameterizations') - call addfld ('MR_pAP', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column wind axial angular momentum after parameterizations') - call addfld ('MR_pAM', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column wind axial angular momentum after dry mass correction') - - call addfld ('MO_pBF', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column mass axial angular momentum before energy fixer') - call addfld ('MO_pBP', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column mass axial angular momentum before parameterizations') - call addfld ('MO_pAP', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column mass axial angular momentum after parameterizations') - call addfld ('MO_pAM', horiz_only, 'A', 'kg*m2/s*rad2',& - 'Total column mass axial angular momentum after dry mass correction') - - end subroutine diag_init_dry - - subroutine diag_init_moist(pbuf2d) - - ! Declare the history fields for which this module contains outfld calls. - - use cam_history, only: addfld, add_default, horiz_only - use cam_history, only: register_vector_field - use constituent_burden, only: constituent_burden_init - use physics_buffer, only: pbuf_set_field - - type(physics_buffer_desc), pointer, intent(in) :: pbuf2d(:,:) - - integer :: k, m - integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. - integer :: ierr - ! column burdens for all constituents except water vapor - call constituent_burden_init - - call cnst_get_ind('CLDLIQ', ixcldliq, abort=.false.) - call cnst_get_ind('CLDICE', ixcldice, abort=.false.) - - ! outfld calls in diag_phys_writeout - call addfld ('OMEGAQ', (/ 'lev' /), 'A', 'kgPa/kgs', 'Vertical water transport' ) - call addfld ('VQ', (/ 'lev' /), 'A', 'm/skg/kg', 'Meridional water transport') - call addfld ('QQ', (/ 'lev' /), 'A', 'kg2/kg2', 'Eddy moisture variance') - - call addfld ('MQ', (/ 'lev' /), 'A', 'kg/m2','Water vapor mass in layer') - call addfld ('TMQ', horiz_only, 'A', 'kg/m2','Total (vertically integrated) precipitable water') -!CAS - call addfld ('IVT', horiz_only, 'A', 'kg/m/s','Total (vertically integrated) vapor transport') - call addfld ('uIVT', horiz_only, 'A', 'kg/m/s','u-component (vertically integrated) vapor transport') - call addfld ('vIVT', horiz_only, 'A', 'kg/m/s','v-component (vertically integrated) vapor transport') -!CAS - call addfld ('RELHUM', (/ 'lev' /), 'A', 'percent','Relative humidity') - call addfld ('RHW', (/ 'lev' /), 'A', 'percent','Relative humidity with respect to liquid') - call addfld ('RHI', (/ 'lev' /), 'A', 'percent','Relative humidity with respect to ice') - call addfld ('RHCFMIP', (/ 'lev' /), 'A', 'percent','Relative humidity with respect to water above 273 K, ice below 273 K') - - call addfld ('THE8501000', horiz_only, 'A', 'K','ThetaE difference 850 mb - 1000 mb') - call addfld ('THE9251000', horiz_only, 'A', 'K','ThetaE difference 925 mb - 1000 mb') - - call addfld ('Q1000', horiz_only, 'A', 'kg/kg','Specific Humidity at 1000 mbar pressure surface') - call addfld ('Q925', horiz_only, 'A', 'kg/kg','Specific Humidity at 925 mbar pressure surface') - call addfld ('Q850', horiz_only, 'A', 'kg/kg','Specific Humidity at 850 mbar pressure surface') - call addfld ('Q200', horiz_only, 'A', 'kg/kg','Specific Humidity at 700 mbar pressure surface') - call addfld ('QBOT', horiz_only, 'A', 'kg/kg','Lowest model level water vapor mixing ratio') - - call addfld ('PSDRY', horiz_only, 'A', 'Pa', 'Dry surface pressure') - call addfld ('PMID', (/ 'lev' /), 'A', 'Pa', 'Pressure at layer midpoints') - call addfld ('PDELDRY', (/ 'lev' /), 'A', 'Pa', 'Dry pressure difference between levels') - - ! outfld calls in diag_conv - - call addfld ('DTCOND', (/ 'lev' /), 'A','K/s','T tendency - moist processes') - call addfld ('DTCOND_24_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. cos coeff.') - call addfld ('DTCOND_24_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. sin coeff.') - call addfld ('DTCOND_12_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. cos coeff.') - call addfld ('DTCOND_12_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. sin coeff.') - call addfld ('DTCOND_08_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 8hr. cos coeff.') - call addfld ('DTCOND_08_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 8hr. sin coeff.') - - call addfld ('PRECL', horiz_only, 'A', 'm/s','Large-scale (stable) precipitation rate (liq + ice)' ) - call addfld ('PRECC', horiz_only, 'A', 'm/s','Convective precipitation rate (liq + ice)' ) - call addfld ('PRECT', horiz_only, 'A', 'm/s','Total (convective and large-scale) precipitation rate (liq + ice)' ) - call addfld ('PREC_PCW', horiz_only, 'A', 'm/s','LS_pcw precipitation rate') - call addfld ('PREC_zmc', horiz_only, 'A', 'm/s','CV_zmc precipitation rate') - call addfld ('PRECTMX', horiz_only, 'X','m/s','Maximum (convective and large-scale) precipitation rate (liq+ice)' ) - call addfld ('PRECSL', horiz_only, 'A', 'm/s','Large-scale (stable) snow rate (water equivalent)' ) - call addfld ('PRECSC', horiz_only, 'A', 'm/s','Convective snow rate (water equivalent)' ) - call addfld ('PRECCav', horiz_only, 'A', 'm/s','Average large-scale precipitation (liq + ice)' ) - call addfld ('PRECLav', horiz_only, 'A', 'm/s','Average convective precipitation (liq + ice)' ) - - ! outfld calls in diag_surf - - call addfld ('SHFLX', horiz_only, 'A', 'W/m2','Surface sensible heat flux') - call addfld ('LHFLX', horiz_only, 'A', 'W/m2','Surface latent heat flux') - call addfld ('QFLX', horiz_only, 'A', 'kg/m2/s','Surface water flux') - - call addfld ('TAUX', horiz_only, 'A', 'N/m2','Zonal surface stress') - call addfld ('TAUY', horiz_only, 'A', 'N/m2','Meridional surface stress') - call addfld ('TREFHT', horiz_only, 'A', 'K','Reference height temperature') - call addfld ('TREFHTMN', horiz_only, 'M','K','Minimum reference height temperature over output period') - call addfld ('TREFHTMX', horiz_only, 'X','K','Maximum reference height temperature over output period') - call addfld ('QREFHT', horiz_only, 'A', 'kg/kg','Reference height humidity') - call addfld ('U10', horiz_only, 'A', 'm/s','10m wind speed') - call addfld ('RHREFHT', horiz_only, 'A', 'fraction','Reference height relative humidity') - - call addfld ('LANDFRAC', horiz_only, 'A', 'fraction','Fraction of sfc area covered by land') - call addfld ('ICEFRAC', horiz_only, 'A', 'fraction','Fraction of sfc area covered by sea-ice') - call addfld ('OCNFRAC', horiz_only, 'A', 'fraction','Fraction of sfc area covered by ocean') - - call addfld ('TREFMNAV', horiz_only, 'A', 'K','Average of TREFHT daily minimum') - call addfld ('TREFMXAV', horiz_only, 'A', 'K','Average of TREFHT daily maximum') - - call addfld ('TS', horiz_only, 'A', 'K','Surface temperature (radiative)') - call addfld ('TSMN', horiz_only, 'M','K','Minimum surface temperature over output period') - call addfld ('TSMX', horiz_only, 'X','K','Maximum surface temperature over output period') - call addfld ('SNOWHLND', horiz_only, 'A', 'm','Water equivalent snow depth') - call addfld ('SNOWHICE', horiz_only, 'A', 'm','Snow depth over ice', fill_value = 1.e30_r8) - call addfld ('TBOT', horiz_only, 'A', 'K','Lowest model level temperature') - - call addfld ('ASDIR', horiz_only, 'A', '1','albedo: shortwave, direct') - call addfld ('ASDIF', horiz_only, 'A', '1','albedo: shortwave, diffuse') - call addfld ('ALDIR', horiz_only, 'A', '1','albedo: longwave, direct') - call addfld ('ALDIF', horiz_only, 'A', '1','albedo: longwave, diffuse') - call addfld ('SST', horiz_only, 'A', 'K','sea surface temperature') - - - ! outfld calls in diag_phys_tend_writeout - - call addfld (ptendnam( 1),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name( 1))//' total physics tendency ' ) - - if (ixcldliq > 0) then - call addfld (ptendnam(ixcldliq),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldliq))//' total physics tendency ' ) - end if - if (ixcldice > 0) then - call addfld (ptendnam(ixcldice),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldice))//' total physics tendency ') - end if - if ( dycore_is('LR') )then - call addfld (dmetendnam( 1),(/ 'lev' /), 'A','kg/kg/s', & - trim(cnst_name( 1))//' dme adjustment tendency (FV) ') - if (ixcldliq > 0) then - call addfld (dmetendnam(ixcldliq),(/ 'lev' /), 'A','kg/kg/s', & - trim(cnst_name(ixcldliq))//' dme adjustment tendency (FV) ') - end if - if (ixcldice > 0) then - call addfld (dmetendnam(ixcldice),(/ 'lev' /), 'A','kg/kg/s', & - trim(cnst_name(ixcldice))//' dme adjustment tendency (FV) ') - end if - end if + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + ! Exit here for adiabatic/ideal physics cases ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + if (.not. moist_physics) return - ! outfld calls in diag_physvar_ic - - call addfld ('QCWAT&IC', (/ 'lev' /), 'I','kg/kg','q associated with cloud water' ) - call addfld ('TCWAT&IC', (/ 'lev' /), 'I','kg/kg','T associated with cloud water' ) - call addfld ('LCWAT&IC', (/ 'lev' /), 'I','kg/kg','Cloud water (ice + liq' ) - call addfld ('CLOUD&IC', (/ 'lev' /), 'I','fraction','Cloud fraction' ) - call addfld ('CONCLD&IC', (/ 'lev' /), 'I','fraction','Convective cloud fraction' ) - call addfld ('TKE&IC', (/ 'ilev' /), 'I','m2/s2','Turbulent Kinetic Energy' ) - call addfld ('CUSH&IC', horiz_only, 'I','m','Convective Scale Height' ) - call addfld ('KVH&IC', (/ 'ilev' /), 'I','m2/s','Vertical diffusion diffusivities (heat/moisture)' ) - call addfld ('KVM&IC', (/ 'ilev' /), 'I','m2/s','Vertical diffusion diffusivities (momentum)' ) - call addfld ('PBLH&IC', horiz_only, 'I','m','PBL height' ) - call addfld ('TPERT&IC', horiz_only, 'I','K','Perturbation temperature (eddies in PBL)' ) - call addfld ('QPERT&IC', horiz_only, 'I','kg/kg','Perturbation specific humidity (eddies in PBL)' ) - - ! CAM export state - call addfld('a2x_BCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic black carbon') - call addfld('a2x_BCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic black carbon') - call addfld('a2x_BCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic black carbon') - call addfld('a2x_OCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic organic carbon') - call addfld('a2x_OCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic organic carbon') - call addfld('a2x_OCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic organic carbon') - call addfld('a2x_DSTWET1', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin1)') - call addfld('a2x_DSTDRY1', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin1)') - call addfld('a2x_DSTWET2', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin2)') - call addfld('a2x_DSTDRY2', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin2)') - call addfld('a2x_DSTWET3', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin3)') - call addfld('a2x_DSTDRY3', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin3)') - call addfld('a2x_DSTWET4', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin4)') - call addfld('a2x_DSTDRY4', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin4)') - - ! defaults - if (history_amwg) then - call add_default (cnst_name(1), 1, ' ') - call add_default ('VQ ', 1, ' ') - call add_default ('TMQ ', 1, ' ') -!CAS - call add_default ('IVT ', 1, ' ') - call add_default ('uIVT ', 1, ' ') - call add_default ('vIVT ', 1, ' ') -!CAS - call add_default ('PSL ', 1, ' ') - call add_default ('RELHUM ', 1, ' ') - - call add_default ('DTCOND ', 1, ' ') - call add_default ('PRECL ', 1, ' ') - call add_default ('PRECC ', 1, ' ') - call add_default ('PRECSL ', 1, ' ') - call add_default ('PRECSC ', 1, ' ') - call add_default ('SHFLX ', 1, ' ') - call add_default ('LHFLX ', 1, ' ') - call add_default ('QFLX ', 1, ' ') - call add_default ('TAUX ', 1, ' ') - call add_default ('TAUY ', 1, ' ') - call add_default ('TREFHT ', 1, ' ') - call add_default ('LANDFRAC', 1, ' ') - call add_default ('OCNFRAC ', 1, ' ') - call add_default ('QREFHT ', 1, ' ') - call add_default ('U10 ', 1, ' ') - call add_default ('ICEFRAC ', 1, ' ') - call add_default ('TS ', 1, ' ') - call add_default ('TSMN ', 1, ' ') - call add_default ('TSMX ', 1, ' ') - call add_default ('SNOWHLND', 1, ' ') - call add_default ('SNOWHICE', 1, ' ') - end if - - if (dycore_is('SE')) then - call add_default ('PSDRY', 1, ' ') - call add_default ('PMID', 1, ' ') - end if - if (history_eddy) then - call add_default ('VQ ', 1, ' ') - endif + call addfld ('PDELDRY',(/ 'lev' /), 'A','Pa','Dry pressure difference between levels') + call addfld ('PSDRY',horiz_only, 'A','Pa','Surface pressure') - if ( history_budget ) then - call add_default (cnst_name(1), history_budget_histfile_num, ' ') - call add_default ('PTTEND' , history_budget_histfile_num, ' ') - call add_default (ptendnam( 1), history_budget_histfile_num, ' ') - if (ixcldliq > 0) then - call add_default (ptendnam(ixcldliq), history_budget_histfile_num, ' ') - end if - if (ixcldice > 0) then - call add_default (ptendnam(ixcldice), history_budget_histfile_num, ' ') - end if - if ( dycore_is('LR') )then - call add_default(dmetendnam(1) , history_budget_histfile_num, ' ') - if (ixcldliq > 0) then - call add_default(dmetendnam(ixcldliq), history_budget_histfile_num, ' ') - end if - if (ixcldice > 0) then - call add_default(dmetendnam(ixcldice), history_budget_histfile_num, ' ') - end if - end if - if( history_budget_histfile_num > 1 ) then - call add_default ('DTCOND ' , history_budget_histfile_num, ' ') - end if - end if + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + call add_default ('PS ', 2, ' ') + call add_default ('T ', 2, ' ') + end if - if (history_vdiag) then - call add_default ('PRECT ', 2, ' ') - call add_default ('PRECT ', 3, ' ') - call add_default ('PRECT ', 4, ' ') - end if + ! outfld calls in diag_conv - ! Initial file - Optional fields - if (inithist_all.or.single_column) then - call add_default ('CONCLD&IC ',0, 'I') - call add_default ('QCWAT&IC ',0, 'I') - call add_default ('TCWAT&IC ',0, 'I') - call add_default ('LCWAT&IC ',0, 'I') - call add_default ('PBLH&IC ',0, 'I') - call add_default ('TPERT&IC ',0, 'I') - call add_default ('QPERT&IC ',0, 'I') - call add_default ('CLOUD&IC ',0, 'I') - call add_default ('TKE&IC ',0, 'I') - call add_default ('CUSH&IC ',0, 'I') - call add_default ('KVH&IC ',0, 'I') - call add_default ('KVM&IC ',0, 'I') - end if + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call addfld ('DTCOND',(/ 'lev' /), 'A','K/s','T tendency - moist processes') + call addfld ('DTCOND_24_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. cos coeff.') + call addfld ('DTCOND_24_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. sin coeff.') + call addfld ('DTCOND_12_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. cos coeff.') + call addfld ('DTCOND_12_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. sin coeff.') - ! determine number of constituents for which convective tendencies must be computed - if (history_budget) then + ! determine number of constituents for which convective tendencies must be computed + if (history_budget) then dqcond_num = pcnst - else + else if (diag_cnst_conv_tend == 'none') dqcond_num = 0 if (diag_cnst_conv_tend == 'q_only') dqcond_num = 1 if (diag_cnst_conv_tend == 'all') dqcond_num = pcnst - end if + end if - do m = 1, dqcond_num + do m = 1, dqcond_num dcconnam(m) = 'DC'//cnst_name(m) - end do + end do - if ((diag_cnst_conv_tend == 'q_only') .or. (diag_cnst_conv_tend == 'all') .or. history_budget) then + if (diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' .or. history_budget) then call addfld (dcconnam(1),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(1))//' tendency due to moist processes') if ( diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' ) then - call add_default (dcconnam(1), 1, ' ') + call add_default (dcconnam(1), 1, ' ') end if if( history_budget ) then - call add_default (dcconnam(1), history_budget_histfile_num, ' ') + call add_default (dcconnam(1), history_budget_histfile_num, ' ') end if if (diag_cnst_conv_tend == 'all' .or. history_budget) then - do m = 2, pcnst - call addfld (dcconnam(m),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(m))//' tendency due to moist processes') - if( diag_cnst_conv_tend == 'all' ) then - call add_default (dcconnam(m), 1, ' ') - end if - if( history_budget .and. (m == ixcldliq .or. m == ixcldice) ) then - call add_default (dcconnam(m), history_budget_histfile_num, ' ') - end if - end do + do m = 2, pcnst + call addfld (dcconnam(m),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(m))//' tendency due to moist processes') + if( diag_cnst_conv_tend == 'all' ) then + call add_default (dcconnam(m), 1, ' ') + end if + if( history_budget .and. (m == ixcldliq .or. m == ixcldice) ) then + call add_default (dcconnam(m), history_budget_histfile_num, ' ') + end if + end do end if + end if + + call addfld ('PRECL',horiz_only, 'A','m/s','Large-scale (stable) precipitation rate (liq + ice)' ) + call addfld ('PRECC',horiz_only, 'A','m/s','Convective precipitation rate (liq + ice)' ) + call addfld ('PRECT',horiz_only, 'A','m/s','Total (convective and large-scale) precipitation rate (liq + ice)' ) + call addfld ('PREC_PCW',horiz_only, 'A','m/s','LS_pcw precipitation rate') + call addfld ('PREC_zmc',horiz_only, 'A','m/s','CV_zmc precipitation rate') + call addfld ('PRECTMX',horiz_only, 'X','m/s','Maximum (convective and large-scale) precipitation rate (liq+ice)' ) + call addfld ('PRECSL',horiz_only, 'A','m/s','Large-scale (stable) snow rate (water equivalent)' ) + call addfld ('PRECSC',horiz_only, 'A','m/s','Convective snow rate (water equivalent)' ) + call addfld ('PRECCav',horiz_only, 'A','m/s','Average large-scale precipitation (liq + ice)' ) + call addfld ('PRECLav',horiz_only, 'A','m/s','Average convective precipitation (liq + ice)' ) + + ! outfld calls in diag_surf + + call addfld ('SHFLX',horiz_only, 'A','W/m2','Surface sensible heat flux') + call addfld ('LHFLX',horiz_only, 'A','W/m2','Surface latent heat flux') + call addfld ('QFLX',horiz_only, 'A','kg/m2/s','Surface water flux') + + call addfld ('TAUX',horiz_only, 'A','N/m2','Zonal surface stress') + call addfld ('TAUY',horiz_only, 'A','N/m2','Meridional surface stress') + call addfld ('TREFHT',horiz_only, 'A','K','Reference height temperature') + call addfld ('TREFHTMN',horiz_only, 'M','K','Minimum reference height temperature over output period') + call addfld ('TREFHTMX',horiz_only, 'X','K','Maximum reference height temperature over output period') + call addfld ('QREFHT', horiz_only, 'A','kg/kg','Reference height humidity') + call addfld ('U10', horiz_only, 'A','m/s','10m wind speed') + call addfld ('RHREFHT',horiz_only, 'A','fraction','Reference height relative humidity') + + call addfld ('LANDFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by land') + call addfld ('ICEFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('OCNFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by ocean') + + call addfld ('TREFMNAV',horiz_only, 'A','K','Average of TREFHT daily minimum') + call addfld ('TREFMXAV',horiz_only, 'A','K','Average of TREFHT daily maximum') + + call addfld ('TS',horiz_only, 'A','K','Surface temperature (radiative)') + call addfld ('TSMN',horiz_only, 'M','K','Minimum surface temperature over output period') + call addfld ('TSMX',horiz_only, 'X','K','Maximum surface temperature over output period') + call addfld ('SNOWHLND',horiz_only, 'A','m','Water equivalent snow depth') + call addfld ('SNOWHICE',horiz_only, 'A','m','Snow depth over ice', fill_value = 1.e30_r8) + call addfld ('TBOT',horiz_only, 'A','K','Lowest model level temperature') + + call addfld ('ASDIR', horiz_only, 'A', '1','albedo: shortwave, direct') + call addfld ('ASDIF', horiz_only, 'A', '1','albedo: shortwave, diffuse') + call addfld ('ALDIR', horiz_only, 'A', '1','albedo: longwave, direct') + call addfld ('ALDIF', horiz_only, 'A', '1','albedo: longwave, diffuse') + call addfld ('SST', horiz_only, 'A', 'K','sea surface temperature') + + ! defaults + if (history_amwg) then + call add_default ('DTCOND ', 1, ' ') + call add_default ('PRECL ', 1, ' ') + call add_default ('PRECC ', 1, ' ') + call add_default ('PRECSL ', 1, ' ') + call add_default ('PRECSC ', 1, ' ') + call add_default ('SHFLX ', 1, ' ') + call add_default ('LHFLX ', 1, ' ') + call add_default ('QFLX ', 1, ' ') + call add_default ('TAUX ', 1, ' ') + call add_default ('TAUY ', 1, ' ') + call add_default ('TREFHT ', 1, ' ') + call add_default ('LANDFRAC', 1, ' ') + call add_default ('OCNFRAC ', 1, ' ') + call add_default ('QREFHT ', 1, ' ') + call add_default ('U10 ', 1, ' ') + call add_default ('ICEFRAC ', 1, ' ') + call add_default ('TS ', 1, ' ') + call add_default ('TSMN ', 1, ' ') + call add_default ('TSMX ', 1, ' ') + call add_default ('SNOWHLND', 1, ' ') + call add_default ('SNOWHICE', 1, ' ') + endif + + call add_default ('OMEGA500', 1, ' ') + call add_default ('TH7001000', 1, ' ') + + if (history_vdiag) then + call add_default ('PRECT ', 2, ' ') + call add_default ('PRECT ', 3, ' ') + call add_default ('PRECT ', 4, ' ') end if - ! Pbuf field indices for collecting output data - relhum_idx = pbuf_get_index('RELHUM', errcode=ierr) - qcwat_idx = pbuf_get_index('QCWAT', errcode=ierr) - tcwat_idx = pbuf_get_index('TCWAT', errcode=ierr) - lcwat_idx = pbuf_get_index('LCWAT', errcode=ierr) - cld_idx = pbuf_get_index('CLD', errcode=ierr) - concld_idx = pbuf_get_index('CONCLD', errcode=ierr) + ! outfld calls in diag_phys_tend_writeout - tke_idx = pbuf_get_index('tke', errcode=ierr) - kvm_idx = pbuf_get_index('kvm', errcode=ierr) - kvh_idx = pbuf_get_index('kvh', errcode=ierr) - cush_idx = pbuf_get_index('cush', errcode=ierr) + call addfld ('PTTEND' ,(/ 'lev' /), 'A','K/s','T total physics tendency' ) + call addfld (ptendnam( 1),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name( 1))//' total physics tendency ' ) + call addfld (ptendnam(ixcldliq),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldliq))//' total physics tendency ' ) + call addfld (ptendnam(ixcldice),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldice))//' total physics tendency ' ) + if ( dycore_is('LR') )then + call addfld (dmetendnam( 1),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name( 1))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldliq),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldliq))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldice),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldice))//' dme adjustment tendency (FV) ') + end if - pblh_idx = pbuf_get_index('pblh', errcode=ierr) - tpert_idx = pbuf_get_index('tpert', errcode=ierr) - qpert_idx = pbuf_get_index('qpert', errcode=ierr) + if ( history_budget ) then + call add_default ('PTTEND' , history_budget_histfile_num, ' ') + call add_default (ptendnam( 1), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldice), history_budget_histfile_num, ' ') + if ( dycore_is('LR') )then + call add_default(dmetendnam(1) , history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldice), history_budget_histfile_num, ' ') + end if + if( history_budget_histfile_num > 1 ) then + call add_default ('DTCOND ' , history_budget_histfile_num, ' ') + end if + end if - prec_dp_idx = pbuf_get_index('PREC_DP', errcode=ierr) - snow_dp_idx = pbuf_get_index('SNOW_DP', errcode=ierr) - prec_sh_idx = pbuf_get_index('PREC_SH', errcode=ierr) - snow_sh_idx = pbuf_get_index('SNOW_SH', errcode=ierr) - prec_sed_idx = pbuf_get_index('PREC_SED', errcode=ierr) - snow_sed_idx = pbuf_get_index('SNOW_SED', errcode=ierr) - prec_pcw_idx = pbuf_get_index('PREC_PCW', errcode=ierr) - snow_pcw_idx = pbuf_get_index('SNOW_PCW', errcode=ierr) + ! outfld calls in diag_physvar_ic - if (is_first_step()) then - call pbuf_set_field(pbuf2d, trefmxav_idx, -1.0e36_r8) - call pbuf_set_field(pbuf2d, trefmnav_idx, 1.0e36_r8) - end if + call addfld ('QCWAT&IC',(/ 'lev' /), 'I','kg/kg','q associated with cloud water' ) + call addfld ('TCWAT&IC',(/ 'lev' /), 'I','kg/kg','T associated with cloud water' ) + call addfld ('LCWAT&IC',(/ 'lev' /), 'I','kg/kg','Cloud water (ice + liq' ) + call addfld ('CLOUD&IC',(/ 'lev' /), 'I','fraction','Cloud fraction' ) + call addfld ('CONCLD&IC',(/ 'lev' /), 'I','fraction','Convective cloud fraction' ) + call addfld ('TKE&IC',(/ 'ilev' /),'I','m2/s2','Turbulent Kinetic Energy' ) + call addfld ('CUSH&IC',horiz_only, 'I','m','Convective Scale Height' ) + call addfld ('KVH&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (heat/moisture)') + call addfld ('KVM&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (momentum)' ) + call addfld ('PBLH&IC',horiz_only, 'I','m','PBL height' ) + call addfld ('TPERT&IC',horiz_only, 'I','K','Perturbation temperature (eddies in PBL)' ) + call addfld ('QPERT&IC',horiz_only, 'I','kg/kg','Perturbation specific humidity (eddies in PBL)' ) + call addfld ('TBOT&IC',horiz_only, 'I','K','Lowest model level temperature' ) - end subroutine diag_init_moist - subroutine diag_init(pbuf2d) - use cam_history, only: addfld + ! Initial file - Optional fields - ! Declare the history fields for which this module contains outfld calls. + if (inithist_all) then + call add_default ('CONCLD&IC ',0, 'I') + call add_default ('QCWAT&IC ',0, 'I') + call add_default ('TCWAT&IC ',0, 'I') + call add_default ('LCWAT&IC ',0, 'I') + call add_default ('PBLH&IC ',0, 'I') + call add_default ('TPERT&IC ',0, 'I') + call add_default ('QPERT&IC ',0, 'I') + call add_default ('CLOUD&IC ',0, 'I') + call add_default ('TKE&IC ',0, 'I') + call add_default ('CUSH&IC ',0, 'I') + call add_default ('KVH&IC ',0, 'I') + call add_default ('KVM&IC ',0, 'I') + call add_default ('TBOT&IC ',0, 'I') + end if - type(physics_buffer_desc), pointer, intent(in) :: pbuf2d(:,:) + ! CAM export state + call addfld('a2x_BCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic black carbon') + call addfld('a2x_BCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic black carbon') + call addfld('a2x_BCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic black carbon') + call addfld('a2x_OCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic organic carbon') + call addfld('a2x_OCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic organic carbon') + call addfld('a2x_OCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic organic carbon') + call addfld('a2x_DSTWET1', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin1)') + call addfld('a2x_DSTDRY1', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin1)') + call addfld('a2x_DSTWET2', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin2)') + call addfld('a2x_DSTDRY2', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin2)') + call addfld('a2x_DSTWET3', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin3)') + call addfld('a2x_DSTDRY3', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin3)') + call addfld('a2x_DSTWET4', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin4)') + call addfld('a2x_DSTDRY4', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin4)') + + !--------------------------------------------------------- + ! CAM history fields for CAM-DOM/CAM-CSIM + !--------------------------------------------------------- + + ! CAM-DOM history fields +#ifdef COUP_DOM + call addfld ('TSOCN&IC',horiz_only, 'I','m','Ocean tempertare') + call add_default ('TSOCN&IC ',0, 'I') +#endif - ! ---------------------------- - ! determine default variables - ! ---------------------------- - call phys_getopts(history_amwg_out = history_amwg , & - history_vdiag_out = history_vdiag , & - history_eddy_out = history_eddy , & - history_budget_out = history_budget , & - history_budget_histfile_num_out = history_budget_histfile_num, & - history_waccm_out = history_waccm) + ! CAM-CSIM history fields + + do k=1,plevmx + call addfld (tsnam(k),horiz_only,'A','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK' ,horiz_only,'A','m','Sea ice thickness') + call addfld ('TSICE' ,horiz_only,'A','K','Ice temperature') + do k = 1,plevmx + call addfld (trim(tsnam(k))//'&IC',horiz_only,'I','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK&IC',horiz_only,'I','m','Sea ice thickness' ) + call addfld ('TSICE&IC',horiz_only,'I','K','Ice temperature' ) + call addfld ('SNOWHICE&IC',horiz_only,'I','m','Water equivalent snow depth' ) + call addfld ('ICEFRAC&IC',horiz_only,'I','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('TSICERAD&IC',horiz_only,'I','K','Radiatively equivalent ice temperature' ) + do k = 1,plevmx + call add_default(trim(tsnam(k))//'&IC',0, 'I') + end do + call add_default ('SICTHK&IC ',0, 'I') + call add_default ('TSICE&IC ',0, 'I') + call add_default ('SNOWHICE&IC',0, 'I') + call add_default ('ICEFRAC&IC ',0, 'I') + if (inithist_all) then + call add_default ('TSICERAD&IC',0, 'I') + end if + + !--------------------------------------------------------- + ! WACCM diagnostic history fields + !--------------------------------------------------------- + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + + ! create history variables for fourier coefficients of the diurnal + ! and semidiurnal tide in T, U, V, and Z3 - call diag_init_dry(pbuf2d) - if (moist_physics) then - call diag_init_moist(pbuf2d) - end if + call tidal_diag_init() - end subroutine diag_init + endif -!=============================================================================== + qcwat_idx = pbuf_get_index('QCWAT',ierr) + tcwat_idx = pbuf_get_index('TCWAT',ierr) + lcwat_idx = pbuf_get_index('LCWAT',ierr) + cld_idx = pbuf_get_index('CLD') + concld_idx = pbuf_get_index('CONCLD') + + tke_idx = pbuf_get_index('tke') + kvm_idx = pbuf_get_index('kvm') + kvh_idx = pbuf_get_index('kvh') + cush_idx = pbuf_get_index('cush') + + pblh_idx = pbuf_get_index('pblh') + tpert_idx = pbuf_get_index('tpert') + qpert_idx = pbuf_get_index('qpert',ierr) - subroutine diag_allocate_dry() - use infnan, only: nan, assignment(=) + prec_dp_idx = pbuf_get_index('PREC_DP') + snow_dp_idx = pbuf_get_index('SNOW_DP') + prec_sh_idx = pbuf_get_index('PREC_SH') + snow_sh_idx = pbuf_get_index('SNOW_SH') + prec_sed_idx = pbuf_get_index('PREC_SED') + snow_sed_idx = pbuf_get_index('SNOW_SED') + prec_pcw_idx = pbuf_get_index('PREC_PCW') + snow_pcw_idx = pbuf_get_index('SNOW_PCW') - ! Allocate memory for module variables. - ! Done at the begining of a physics step at same point as the pbuf allocate - ! for variables with "physpkg" scope. +end subroutine diag_init - ! Local variables - character(len=*), parameter :: sub = 'diag_allocate_dry' - character(len=128) :: errmsg - integer :: istat +!=============================================================================== - allocate(dtcond(pcols,pver,begchunk:endchunk), stat=istat) - if ( istat /= 0 ) then - write(errmsg, '(2a,i0)') sub, ': allocate failed, stat = ',istat - call endrun (errmsg) - end if - dtcond = nan - end subroutine diag_allocate_dry +subroutine diag_allocate() + use infnan, only: nan, assignment(=) - subroutine diag_allocate_moist() - use infnan, only: nan, assignment(=) + ! Allocate memory for module variables. + ! Done at the begining of a physics step at same point as the pbuf allocate for + ! variables with "physpkg" scope. - ! Allocate memory for module variables. - ! Done at the begining of a physics step at same point as the pbuf allocate - ! for variables with "physpkg" scope. + ! Local variables + character(len=*), parameter :: sub = 'diag_allocate' + integer :: i, istat - ! Local variables - character(len=*), parameter :: sub = 'diag_allocate_moist' - character(len=128) :: errmsg - integer :: i, istat + allocate(dtcond(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dtcond = nan - if (dqcond_num > 0) then + if (dqcond_num > 0) then allocate(dqcond(dqcond_num)) do i = 1, dqcond_num - allocate(dqcond(i)%cnst(pcols,pver,begchunk:endchunk), stat=istat) - if ( istat /= 0 ) then - write(errmsg, '(2a,i0)') sub, ': allocate failed, stat = ',istat - call endrun (errmsg) - end if - dqcond(i)%cnst = nan + allocate(dqcond(i)%cnst(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dqcond(i)%cnst = nan end do - end if - - end subroutine diag_allocate_moist - - subroutine diag_allocate() - - call diag_allocate_dry() - if (moist_physics) then - call diag_allocate_moist() - end if + end if - end subroutine diag_allocate +end subroutine diag_allocate !=============================================================================== - subroutine diag_deallocate_dry() - ! Deallocate memory for module variables. - ! Done at the end of a physics step at same point as the pbuf deallocate for - ! variables with "physpkg" scope. - - ! Local variables - character(len=*), parameter :: sub = 'diag_deallocate_dry' - integer :: istat +subroutine diag_deallocate() - deallocate(dtcond, stat=istat) - if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') - end subroutine diag_deallocate_dry +! Deallocate memory for module variables. +! Done at the end of a physics step at same point as the pbuf deallocate for +! variables with "physpkg" scope. - subroutine diag_deallocate_moist() +! Local variables + character(len=*), parameter :: sub = 'diag_deallocate' + integer :: i, istat - ! Deallocate memory for module variables. - ! Done at the end of a physics step at same point as the pbuf deallocate for - ! variables with "physpkg" scope. + deallocate(dtcond, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') - ! Local variables - character(len=*), parameter :: sub = 'diag_deallocate_moist' - integer :: i, istat - - if (dqcond_num > 0) then + if (dqcond_num > 0) then do i = 1, dqcond_num - deallocate(dqcond(i)%cnst, stat=istat) - if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + deallocate(dqcond(i)%cnst, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') end do deallocate(dqcond, stat=istat) if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') - end if - end subroutine diag_deallocate_moist - - subroutine diag_deallocate() - - call diag_deallocate_dry() - if (moist_physics) then - call diag_deallocate_moist() - end if - - end subroutine diag_deallocate + end if +end subroutine diag_deallocate !=============================================================================== - subroutine diag_conv_tend_ini(state,pbuf) +subroutine diag_conv_tend_ini(state,pbuf) - ! Initialize convective tendency calcs. +! Initialize convective tendency calcs. - ! Arguments: - type(physics_state), intent(in) :: state - type(physics_buffer_desc), pointer :: pbuf(:) + +! Argument: - ! Local variables: + type(physics_state), intent(in) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) - integer :: i, k, m, lchnk, ncol - real(r8), pointer, dimension(:,:) :: t_ttend +! Local variables: - lchnk = state%lchnk - ncol = state%ncol + integer :: i, k, m, lchnk, ncol + real(r8), pointer, dimension(:,:) :: t_ttend - do k = 1, pver + lchnk = state%lchnk + ncol = state%ncol + + do k = 1, pver do i = 1, ncol - dtcond(i,k,lchnk) = state%t(i,k) + dtcond(i,k,lchnk) = state%s(i,k) end do - end do + end do - do m = 1, dqcond_num + do m = 1, dqcond_num do k = 1, pver - do i = 1, ncol - dqcond(m)%cnst(i,k,lchnk) = state%q(i,k,m) - end do + do i = 1, ncol + dqcond(m)%cnst(i,k,lchnk) = state%q(i,k,m) + end do end do - end do + end do - !! initialize to pbuf T_TTEND to temperature at first timestep - if (is_first_step()) then + !! initialize to pbuf T_TTEND to temperature at first timestep + if (is_first_step()) then do m = 1, dyn_time_lvls - call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,m/), kount=(/pcols,pver,1/)) - t_ttend(:ncol,:) = state%t(:ncol,:) + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,m/), kount=(/pcols,pver,1/)) + t_ttend(:ncol,:) = state%t(:ncol,:) end do - end if - - end subroutine diag_conv_tend_ini + end if +end subroutine diag_conv_tend_ini !=============================================================================== - subroutine diag_phys_writeout_dry(state, pbuf, p_surf_t) + subroutine diag_phys_writeout(state, psl) - !----------------------------------------------------------------------- - ! - ! Purpose: output dry physics diagnostics - ! - !----------------------------------------------------------------------- +!----------------------------------------------------------------------- +! +! Purpose: record dynamics variables on physics grid +! +!----------------------------------------------------------------------- use physconst, only: gravit, rga, rair, cpair, latvap, rearth, pi, cappa use time_manager, only: get_nstep use interpolate_data, only: vertinterp use constituent_burden, only: constituent_burden_comp + use cam_control_mod, only: moist_physics + use co2_cycle, only: c_i, co2_transport use tidal_diag, only: tidal_diag_write - !----------------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(inout) :: state - type(physics_buffer_desc), pointer :: pbuf(:) - real(r8), intent(out) :: p_surf_t(pcols, nsurf) ! data interpolated to a pressure surface - ! - !---------------------------Local workspace----------------------------- - ! - real(r8) :: ftem(pcols,pver) ! temporary workspace - real(r8) :: ftem1(pcols,pver) ! another temporary workspace - real(r8) :: ftem2(pcols,pver) ! another temporary workspace - real(r8) :: z3(pcols,pver) ! geo-potential height - real(r8) :: p_surf(pcols) ! data interpolated to a pressure surface - real(r8) :: tem2(pcols,pver) ! temporary workspace - real(r8) :: timestep(pcols) ! used for outfld call - real(r8) :: esl(pcols,pver) ! saturation vapor pressures - real(r8) :: esi(pcols,pver) ! - real(r8) :: dlon(pcols) ! width of grid cell (meters) - - real(r8), pointer :: psl(:) ! Sea Level Pressure - - integer :: i, k, m, lchnk, ncol, nstep - ! - !----------------------------------------------------------------------- - ! +!----------------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(inout) :: state + real(r8), optional , intent(out) :: psl(pcols) +! +!---------------------------Local workspace----------------------------- +! + real(r8) ftem(pcols,pver) ! temporary workspace + real(r8) ftem1(pcols,pver) ! another temporary workspace + real(r8) ftem2(pcols,pver) ! another temporary workspace + real(r8) ftem4(pcols,pver) ! another temporary workspace + real(r8) ftem5(pcols,pver) ! another temporary workspace + real(r8) psl_tmp(pcols) ! Sea Level Pressure + real(r8) z3(pcols,pver) ! geo-potential height + real(r8) p_surf(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t2(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q2(pcols) ! data interpolated to a pressure surface + real(r8) tem2(pcols,pver) ! temporary workspace + real(r8) timestep(pcols) ! used for outfld call + real(r8) esl(pcols,pver) ! saturation vapor pressures + real(r8) esi(pcols,pver) ! + real(r8) dlon(pcols) ! width of grid cell (meters) + integer plon ! number of longitudes + + integer i, k, m, lchnk, ncol, nstep +! +!----------------------------------------------------------------------- +! lchnk = state%lchnk ncol = state%ncol @@ -949,97 +977,226 @@ subroutine diag_phys_writeout_dry(state, pbuf, p_surf_t) call outfld('PS ',state%ps, pcols ,lchnk ) call outfld('U ',state%u , pcols ,lchnk ) call outfld('V ',state%v , pcols ,lchnk ) + do m=1,pcnst + if ( cnst_cam_outfld(m) ) then + call outfld(cnst_name(m),state%q(1,1,m),pcols ,lchnk ) + end if + end do + + if (co2_transport()) then + do m = 1,4 + call outfld(trim(cnst_name(c_i(m)))//'_BOT', state%q(1,pver,c_i(m)), pcols, lchnk) + end do + end if + + ! column burdens of all constituents except water vapor + call constituent_burden_comp(state) + + if ( moist_physics) then + call outfld('PDELDRY ',state%pdeldry, pcols, lchnk) + call outfld('PSDRY', state%psdry, pcols, lchnk) + end if call outfld('PHIS ',state%phis, pcols, lchnk ) + + #if (defined BFB_CAM_SCAM_IOP ) call outfld('phis ',state%phis, pcols, lchnk ) #endif - do m = 1, pcnst - if (cnst_cam_outfld(m)) then - call outfld(cnst_name(m), state%q(1,1,m), pcols, lchnk) - end if - end do - - ! - ! Add height of surface to midpoint height above surface - ! +! +! Add height of surface to midpoint height above surface +! do k = 1, pver - z3(:ncol,k) = state%zm(:ncol,k) + state%phis(:ncol)*rga + z3(:ncol,k) = state%zm(:ncol,k) + state%phis(:ncol)*rga end do call outfld('Z3 ',z3,pcols,lchnk) - ! - ! Output Z3 on pressure surfaces - ! +! +! Output Z3 on pressure surfaces +! if (hist_fld_active('Z1000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, z3, p_surf, & - extrapolate='Z', ln_interp=.true., ps=state%ps, phis=state%phis, tbot=state%t(:,pver)) - call outfld('Z1000 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, z3, p_surf) + call outfld('Z1000 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, z3, p_surf) + call outfld('Z975 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, z3, p_surf) + call outfld('Z950 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, z3, p_surf) + call outfld('Z925 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, z3, p_surf) + call outfld('Z900 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, z3, p_surf) + call outfld('Z850 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, z3, p_surf) + call outfld('Z800 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z700')) then - call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, z3, p_surf, & - extrapolate='Z', ln_interp=.true., ps=state%ps, phis=state%phis, tbot=state%t(:,pver)) - call outfld('Z700 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, z3, p_surf) + call outfld('Z700 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, z3, p_surf) + call outfld('Z600 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z500')) then - call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, z3, p_surf, & - extrapolate='Z', ln_interp=.true., ps=state%ps, phis=state%phis, tbot=state%t(:,pver)) - call outfld('Z500 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, z3, p_surf) + call outfld('Z500 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, z3, p_surf) + call outfld('Z400 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z300')) then - call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, z3, p_surf, ln_interp=.true.) - call outfld('Z300 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, z3, p_surf) + call outfld('Z300 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z200')) then - call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, z3, p_surf, ln_interp=.true.) - call outfld('Z200 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, z3, p_surf) + call outfld('Z200 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z100')) then - call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, z3, p_surf, ln_interp=.true.) - call outfld('Z100 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, z3, p_surf) + call outfld('Z100 ', p_surf, pcols, lchnk) end if if (hist_fld_active('Z050')) then - call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, z3, p_surf, ln_interp=.true.) - call outfld('Z050 ', p_surf, pcols, lchnk) + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, z3, p_surf) + call outfld('Z050 ', p_surf, pcols, lchnk) end if - ! - ! Quadratic height fiels Z3*Z3 - ! + if (hist_fld_active('Z010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, z3, p_surf) + call outfld('Z010 ', p_surf, pcols, lchnk) + end if +! +! Quadratic height fiels Z3*Z3 +! ftem(:ncol,:) = z3(:ncol,:)*z3(:ncol,:) call outfld('ZZ ',ftem,pcols,lchnk) - ftem(:ncol,:) = z3(:ncol,:)*state%v(:ncol,:) + ftem(:ncol,:) = z3(:ncol,:)*state%v(:ncol,:)*gravit call outfld('VZ ',ftem, pcols,lchnk) - ! - ! Meridional advection fields - ! +! +! Meridional advection fields +! ftem(:ncol,:) = state%v(:ncol,:)*state%t(:ncol,:) call outfld ('VT ',ftem ,pcols ,lchnk ) + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1) + call outfld ('VQ ',ftem ,pcols ,lchnk ) + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,14) + call outfld ('Vbc_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,15) + call outfld ('Vdst_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,22) + call outfld ('Vdst_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,16) + call outfld ('Vncl_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,20) + call outfld ('Vncl_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,23) + call outfld ('Vncl_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,11) + call outfld ('Vso4_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,18) + call outfld ('Vso4_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,24) + call outfld ('Vso4_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,13) + call outfld ('Vsoa_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,19) + call outfld ('Vsoa_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,12) + call outfld ('Vpom_a1 ',ftem ,pcols ,lchnk ) + endif + + ftem(:ncol,:) = state%q(:ncol,:,1)*state%q(:ncol,:,1) + call outfld ('QQ ',ftem ,pcols ,lchnk ) + + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%q(:ncol,:,14)*state%q(:ncol,:,14) + call outfld ('bc_a1_2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,15)*state%q(:ncol,:,15) + call outfld ('dst_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,22)*state%q(:ncol,:,22) + call outfld ('dst_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,16)*state%q(:ncol,:,16) + call outfld ('ncl_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,20)*state%q(:ncol,:,20) + call outfld ('ncl_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,23)*state%q(:ncol,:,23) + call outfld ('ncl_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,11)*state%q(:ncol,:,11) + call outfld ('so4_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,18)*state%q(:ncol,:,18) + call outfld ('so4_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,24)*state%q(:ncol,:,24) + call outfld ('so4_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,13)*state%q(:ncol,:,13) + call outfld ('soa_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,19)*state%q(:ncol,:,19) + call outfld ('soa_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,12)*state%q(:ncol,:,12) + call outfld ('pom_a1_2',ftem ,pcols ,lchnk ) + endif + ftem(:ncol,:) = state%v(:ncol,:)**2 call outfld ('VV ',ftem ,pcols ,lchnk ) ftem(:ncol,:) = state%v(:ncol,:) * state%u(:ncol,:) call outfld ('VU ',ftem ,pcols ,lchnk ) - ! - ! zonal advection - ! + +! zonal advection + ftem(:ncol,:) = state%u(:ncol,:)**2 call outfld ('UU ',ftem ,pcols ,lchnk ) - ! Wind speed +! Wind speed ftem(:ncol,:) = sqrt( state%u(:ncol,:)**2 + state%v(:ncol,:)**2) call outfld ('WSPEED ',ftem ,pcols ,lchnk ) call outfld ('WSPDSRFMX',ftem(:,pver) ,pcols ,lchnk ) - call outfld ('WSPDSRFAV',ftem(:,pver) ,pcols ,lchnk ) + call outfld ('WSPDSRFAV',ftem(:,pver) ,pcols ,lchnk ) - ! Vertical velocity and advection +! Vertical velocity and advection if (single_column) then - call outfld('OMEGA ',wfld, pcols, lchnk ) + call outfld('OMEGA ',wfld, pcols, lchnk ) else - call outfld('OMEGA ',state%omega, pcols, lchnk ) + call outfld('OMEGA ',state%omega, pcols, lchnk ) endif #if (defined BFB_CAM_SCAM_IOP ) @@ -1052,363 +1209,508 @@ subroutine diag_phys_writeout_dry(state, pbuf, p_surf_t) call outfld('OMEGAU ',ftem, pcols, lchnk ) ftem(:ncol,:) = state%omega(:ncol,:)*state%v(:ncol,:) call outfld('OMEGAV ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%q(:ncol,:,1) + call outfld('OMEGAQ ',ftem, pcols, lchnk ) ftem(:ncol,:) = state%omega(:ncol,:)*state%omega(:ncol,:) call outfld('OMGAOMGA',ftem, pcols, lchnk ) - ! - ! Output omega at 850 and 500 mb pressure levels - ! - if (hist_fld_active('OMEGA850')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%omega, p_surf) - call outfld('OMEGA850', p_surf, pcols, lchnk) - end if - if (hist_fld_active('OMEGA500')) then - call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%omega, p_surf) - call outfld('OMEGA500', p_surf, pcols, lchnk) +! +! Output omega at 850 and 500 mb pressure levels +! + if (hist_fld_active('OMEGA1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%omega, p_surf) + call outfld('OMEGA1000', p_surf, pcols, lchnk) end if - - ! Sea level pressure - call pbuf_get_field(pbuf, psl_idx, psl) - call cpslec(ncol, state%pmid, state%phis, state%ps, state%t, psl, gravit, rair) - call outfld('PSL', psl, pcols, lchnk) - - ! Output T,u,v fields on pressure surfaces - ! - if (hist_fld_active('T850')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf, & - extrapolate='T', ps=state%ps, phis=state%phis) - call outfld('T850 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%omega, p_surf) + call outfld('OMEGA975', p_surf, pcols, lchnk) end if - if (hist_fld_active('T500')) then - call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%t, p_surf, & - extrapolate='T', ps=state%ps, phis=state%phis) - call outfld('T500 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%omega, p_surf) + call outfld('OMEGA950', p_surf, pcols, lchnk) end if - if (hist_fld_active('T400')) then - call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%t, p_surf, & - extrapolate='T', ps=state%ps, phis=state%phis) - call outfld('T400 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%omega, p_surf) + call outfld('OMEGA925', p_surf, pcols, lchnk) end if - if (hist_fld_active('T300')) then - call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%t, p_surf) - call outfld('T300 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%omega, p_surf) + call outfld('OMEGA900', p_surf, pcols, lchnk) end if - if (hist_fld_active('T200')) then - call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%t, p_surf) - call outfld('T200 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%omega, p_surf) + call outfld('OMEGA850', p_surf, pcols, lchnk) end if - if (hist_fld_active('U850')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%u, p_surf) - call outfld('U850 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%omega, p_surf) + call outfld('OMEGA800', p_surf, pcols, lchnk) end if - if (hist_fld_active('U500')) then - call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%u, p_surf) - call outfld('U500 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%omega, p_surf) + call outfld('OMEGA700', p_surf, pcols, lchnk) end if - if (hist_fld_active('U250')) then - call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%u, p_surf) - call outfld('U250 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%omega, p_surf) + call outfld('OMEGA600', p_surf, pcols, lchnk) end if - if (hist_fld_active('U200')) then - call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%u, p_surf) - call outfld('U200 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%omega, p_surf) + call outfld('OMEGA500', p_surf, pcols, lchnk) end if - if (hist_fld_active('U010')) then - call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%u, p_surf) - call outfld('U010 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%omega, p_surf) + call outfld('OMEGA400', p_surf, pcols, lchnk) end if - if (hist_fld_active('V850')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%v, p_surf) - call outfld('V850 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%omega, p_surf) + call outfld('OMEGA300', p_surf, pcols, lchnk) end if - if (hist_fld_active('V500')) then - call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%v, p_surf) - call outfld('V500 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%omega, p_surf) + call outfld('OMEGA200', p_surf, pcols, lchnk) end if - if (hist_fld_active('V250')) then - call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%v, p_surf) - call outfld('V250 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGA100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%omega, p_surf) + call outfld('OMEGA100', p_surf, pcols, lchnk) end if - if (hist_fld_active('V200')) then - call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%v, p_surf) - call outfld('V200 ', p_surf, pcols, lchnk ) + if (hist_fld_active('OMEGABOT')) then + call outfld('OMEGABOT', state%omega(:,pver), pcols, lchnk) end if +! +! Mass of q, by layer and vertically integrated +! + ftem(:ncol,:) = state%q(:ncol,:,1) * state%pdel(:ncol,:) * rga + call outfld ('MQ ',ftem ,pcols ,lchnk ) - ftem(:ncol,:) = state%t(:ncol,:)*state%t(:ncol,:) - call outfld('TT ',ftem ,pcols ,lchnk ) - ! - ! Output U, V, T, P and Z at bottom level - ! - call outfld ('UBOT ', state%u(1,pver) , pcols, lchnk) - call outfld ('VBOT ', state%v(1,pver) , pcols, lchnk) - call outfld ('ZBOT ', state%zm(1,pver) , pcols, lchnk) + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TMQ ',ftem, pcols ,lchnk ) - !! Boundary layer atmospheric stability, temperature, water vapor diagnostics +!CAS integrated vapor transport calculation - p_surf_t = -99.0_r8 ! Uninitialized to impossible value - if (hist_fld_active('T1000') .or. & - hist_fld_active('T9251000') .or. & - hist_fld_active('TH9251000') .or. & - hist_fld_active('T8501000') .or. & - hist_fld_active('TH8501000') .or. & - hist_fld_active('T7001000') .or. & - hist_fld_active('TH7001000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%t, p_surf_t(:,surf_100000)) - end if + !compute uq*dp/g and vq*dp/g + ftem4(:ncol,:) = state%q(:ncol,:,1) * state%u(:ncol,:) *state%pdel(:ncol,:) * rga + ftem5(:ncol,:) = state%q(:ncol,:,1) * state%v(:ncol,:) *state%pdel(:ncol,:) * rga - if ( hist_fld_active('T925') .or. & - hist_fld_active('T9251000') .or. & - hist_fld_active('TH9251000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%t, p_surf_t(:,surf_092500)) - end if + !integrate each component + do k=2,pver + ftem4(:ncol,1) = ftem4(:ncol,1) + ftem4(:ncol,k) + ftem5(:ncol,1) = ftem5(:ncol,1) + ftem5(:ncol,k) + end do + !compute ivt + ftem(:ncol,1) = sqrt( ftem4(:ncol,1)**2 + ftem5(:ncol,1)**2) -!!! at 1000 mb and 925 mb - if (hist_fld_active('T1000')) then - call outfld('T1000 ', p_surf_t(:,surf_100000), pcols, lchnk ) - end if + call outfld ('IVT ',ftem, pcols ,lchnk ) - if (hist_fld_active('T925')) then - call outfld('T925 ', p_surf_t(:,surf_092500), pcols, lchnk ) - end if + !just output uq*dp/g + ftem(:ncol,1) = ftem4(:ncol,1) + call outfld ('uIVT ',ftem, pcols ,lchnk ) - if (hist_fld_active('T9251000')) then - p_surf = p_surf_t(:,surf_092500) - p_surf_t(:,surf_100000) - call outfld('T9251000 ', p_surf, pcols, lchnk ) - end if + !just output vq*dp/g + ftem(:ncol,1) = ftem5(:ncol,1) + call outfld ('vIVT ',ftem, pcols ,lchnk ) - if (hist_fld_active('TH9251000')) then - p_surf = (p_surf_t(:,surf_092500)*(1000.0_r8/925.0_r8)**cappa) - (p_surf_t(:,surf_100000)*(1.0_r8)**cappa) - call outfld('TH9251000 ', p_surf, pcols, lchnk ) - end if +!CAS - if (hist_fld_active('T8501000') .or. & - hist_fld_active('TH8501000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf_t(:,surf_085000)) - end if - -!!! at 1000 mb and 850 mb - if (hist_fld_active('T8501000')) then - p_surf = p_surf_t(:,surf_085000)-p_surf_t(:,surf_100000) - call outfld('T8501000 ', p_surf, pcols, lchnk ) - end if - - if (hist_fld_active('TH8501000')) then - p_surf = (p_surf_t(:,surf_085000)*(1000.0_r8/850.0_r8)**cappa)-(p_surf_t(:,surf_100000)*(1.0_r8)**cappa) - call outfld('TH8501000 ', p_surf, pcols, lchnk ) - end if - - if (hist_fld_active('T7001000') .or. & - hist_fld_active('TH7001000') .or. & - hist_fld_active('T700')) then - call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%t, p_surf_t(:,surf_070000)) - end if - -!!! at 700 mb - if (hist_fld_active('T700')) then - call outfld('T700 ', p_surf_t(:,surf_070000), pcols, lchnk ) - end if - -!!! at 1000 mb and 700 mb - if (hist_fld_active('T7001000')) then - p_surf = p_surf_t(:,surf_070000)-p_surf_t(:,surf_100000) - call outfld('T7001000 ', p_surf, pcols, lchnk ) - end if - - if (hist_fld_active('TH7001000')) then - p_surf = (p_surf_t(:,surf_070000)*(1000.0_r8/700.0_r8)**cappa)-(p_surf_t(:,surf_100000)*(1.0_r8)**cappa) - call outfld('TH7001000 ', p_surf, pcols, lchnk ) - end if - - if (hist_fld_active('T010')) then - call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%t, p_surf) - call outfld('T010 ', p_surf, pcols, lchnk ) - end if - - !--------------------------------------------------------- - ! tidal diagnostics - !--------------------------------------------------------- - call tidal_diag_write(state) - - return - end subroutine diag_phys_writeout_dry - -!=============================================================================== - - subroutine diag_phys_writeout_moist(state, pbuf, p_surf_t) - - !----------------------------------------------------------------------- - ! - ! Purpose: record dynamics variables on physics grid - ! - !----------------------------------------------------------------------- - use physconst, only: gravit, rga, rair, cpair, latvap, rearth, pi, cappa, & - epsilo, rh2o - use interpolate_data, only: vertinterp - use constituent_burden, only: constituent_burden_comp - use co2_cycle, only: c_i, co2_transport, co2_readFlux_aircraft - !----------------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(inout) :: state - type(physics_buffer_desc), pointer :: pbuf(:) - real(r8), intent(inout) :: p_surf_t(pcols, nsurf) ! data interpolated to a pressure surface - ! - !---------------------------Local workspace----------------------------- - ! - real(r8) :: ftem(pcols,pver) ! temporary workspace - real(r8) :: ftem1(pcols,pver) ! another temporary workspace - real(r8) :: ftem2(pcols,pver) ! another temporary workspace -!CAS - real(r8) :: ftem4(pcols,pver) ! another temporary workspace - real(r8) :: ftem5(pcols,pver) ! another temporary workspace -!CAS - real(r8) :: z3(pcols,pver) ! geo-potential height - real(r8) :: p_surf(pcols) ! data interpolated to a pressure surface - real(r8) :: p_surf_q1(pcols) ! data interpolated to a pressure surface - real(r8) :: p_surf_q2(pcols) ! data interpolated to a pressure surface - real(r8) :: tem2(pcols,pver) ! temporary workspace - real(r8) :: esl(pcols,pver) ! saturation vapor pressures - real(r8) :: esi(pcols,pver) ! - - real(r8), pointer :: ftem_ptr(:,:) - - integer :: i, k, m, lchnk, ncol - ! - !----------------------------------------------------------------------- - ! - lchnk = state%lchnk - ncol = state%ncol - - if (co2_transport()) then - do m = 1,4 - call outfld(trim(cnst_name(c_i(m)))//'_BOT', state%q(1,pver,c_i(m)), pcols, lchnk) - end do - - if (co2_readFlux_aircraft) then - call pbuf_get_field(pbuf, pbuf_get_index('ac_CO2'), ftem_ptr) - call outfld('TMac_CO2', sum(ftem_ptr(:ncol,:), dim=2), pcols, lchnk) - end if - end if - - ! column burdens of all constituents except water vapor - call constituent_burden_comp(state) - - call outfld('PSDRY', state%psdry, pcols, lchnk) - call outfld('PMID', state%pmid, pcols, lchnk) - call outfld('PDELDRY', state%pdeldry, pcols, lchnk) - - ! - ! Meridional advection fields - ! - ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1) - call outfld ('VQ ',ftem ,pcols ,lchnk ) - - ftem(:ncol,:) = state%q(:ncol,:,1)*state%q(:ncol,:,1) - call outfld ('QQ ',ftem ,pcols ,lchnk ) - - ! Vertical velocity and advection - ftem(:ncol,:) = state%omega(:ncol,:)*state%q(:ncol,:,1) - call outfld('OMEGAQ ',ftem, pcols, lchnk ) - ! - ! Mass of q, by layer and vertically integrated - ! - ftem(:ncol,:) = state%q(:ncol,:,1) * state%pdel(:ncol,:) * rga - call outfld ('MQ ',ftem ,pcols ,lchnk ) +! +! Mass of vertically integrated q flux +! + ftem(:ncol,:) = state%u(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga do k=2,pver - ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) end do - call outfld ('TMQ ',ftem, pcols ,lchnk ) + call outfld ('TUQ ',ftem, pcols ,lchnk ) -!CAS integrated vapor transport calculation - - !compute uq*dp/g and vq*dp/g - ftem4(:ncol,:) = state%q(:ncol,:,1) * state%u(:ncol,:) *state%pdel(:ncol,:) * rga - ftem5(:ncol,:) = state%q(:ncol,:,1) * state%v(:ncol,:) *state%pdel(:ncol,:) * rga - - !integrate each component + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga do k=2,pver - ftem4(:ncol,1) = ftem4(:ncol,1) + ftem4(:ncol,k) - ftem5(:ncol,1) = ftem5(:ncol,1) + ftem5(:ncol,k) + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) end do - !compute ivt - ftem(:ncol,1) = sqrt( ftem4(:ncol,1)**2 + ftem5(:ncol,1)**2) - call outfld ('IVT ',ftem, pcols ,lchnk ) + call outfld ('TVQ ',ftem, pcols ,lchnk ) - !just output uq*dp/g - ftem(:ncol,1) = ftem4(:ncol,1) - call outfld ('uIVT ',ftem, pcols ,lchnk ) +! +! Mass of vertically integrated MSE flux +! + ftem(:ncol,:) = state%u(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TUH ',ftem, pcols ,lchnk ) - !just output vq*dp/g - ftem(:ncol,1) = ftem5(:ncol,1) - call outfld ('vIVT ',ftem, pcols ,lchnk ) + ftem(:ncol,:) = state%v(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TVH ',ftem, pcols ,lchnk ) -!CAS + if (moist_physics) then - ! Relative humidity - if (hist_fld_active('RELHUM')) then - if (relhum_idx > 0) then - call pbuf_get_field(pbuf, relhum_idx, ftem_ptr) - ftem(:ncol,:) = ftem_ptr(:ncol,:) - else + ! Relative humidity + if (hist_fld_active('RELHUM')) then call qsat(state%t(:ncol,:), state%pmid(:ncol,:), & - tem2(:ncol,:), ftem(:ncol,:)) + tem2(:ncol,:), ftem(:ncol,:)) ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RELHUM ',ftem ,pcols ,lchnk ) end if - call outfld ('RELHUM ',ftem ,pcols ,lchnk ) - end if - - if (hist_fld_active('RHW') .or. hist_fld_active('RHI') .or. hist_fld_active('RHCFMIP') ) then - ! RH w.r.t liquid (water) - call qsat_water (state%t(:ncol,:), state%pmid(:ncol,:), & - esl(:ncol,:), ftem(:ncol,:)) - ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 - call outfld ('RHW ',ftem ,pcols ,lchnk ) + ! ftem defined above assuming RELHUM in one of the history tape + if (hist_fld_active('RH1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, ftem, p_surf) + call outfld('RH1000', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, ftem, p_surf) + call outfld('RH975', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, ftem, p_surf) + call outfld('RH950', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, ftem, p_surf) + call outfld('RH925', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, ftem, p_surf) + call outfld('RH900', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, ftem, p_surf) + call outfld('RH850', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, ftem, p_surf) + call outfld('RH800', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, ftem, p_surf) + call outfld('RH700', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, ftem, p_surf) + call outfld('RH600', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, ftem, p_surf) + call outfld('RH500', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, ftem, p_surf) + call outfld('RH400', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, ftem, p_surf) + call outfld('RH300', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, ftem, p_surf) + call outfld('RH200', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, ftem, p_surf) + call outfld('RH100', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RHBOT')) then + call outfld('RHBOT', ftem(:,pver), pcols, lchnk) + end if - ! Convert to RHI (ice) - do i=1,ncol - do k=1,pver - esi(i,k)=svp_ice(state%t(i,k)) - ftem1(i,k)=ftem(i,k)*esl(i,k)/esi(i,k) - end do - end do - call outfld ('RHI ',ftem1 ,pcols ,lchnk ) + if (hist_fld_active('RHW') .or. hist_fld_active('RHI') .or. hist_fld_active('RHCFMIP') ) then + + ! RH w.r.t liquid (water) + call qsat_water (state%t(:ncol,:), state%pmid(:ncol,:), & + esl(:ncol,:), ftem(:ncol,:)) + ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RHW ',ftem ,pcols ,lchnk ) + + ! Convert to RHI (ice) + do i=1,ncol + do k=1,pver + esi(i,k)=svp_ice(state%t(i,k)) + ftem1(i,k)=ftem(i,k)*esl(i,k)/esi(i,k) + end do + end do + call outfld ('RHI ',ftem1 ,pcols ,lchnk ) - ! use temperature to decide if you populate with ftem (liquid, above 0 C) or ftem1 (ice, below 0 C) + ! use temperature to decide if you populate with ftem (liquid, above 0 C) or ftem1 (ice, below 0 C) - ftem2(:ncol,:)=ftem(:ncol,:) + ftem2(:ncol,:)=ftem(:ncol,:) - do i=1,ncol - do k=1,pver - if (state%t(i,k) .gt. 273) then - ftem2(i,k)=ftem(i,k) !!wrt water - else - ftem2(i,k)=ftem1(i,k) !!wrt ice - end if - end do - end do + do i=1,ncol + do k=1,pver + if (state%t(i,k) .gt. 273) then + ftem2(i,k)=ftem(i,k) !!wrt water + else + ftem2(i,k)=ftem1(i,k) !!wrt ice + end if + end do + end do + + call outfld ('RHCFMIP ',ftem2 ,pcols ,lchnk ) - call outfld ('RHCFMIP ',ftem2 ,pcols ,lchnk ) + end if end if - ! - ! Output q field on pressure surfaces - ! +! +! Sea level pressure +! + if (present(psl) .or. hist_fld_active('PSL')) then + call cpslec (ncol, state%pmid, state%phis, state%ps, state%t,psl_tmp, gravit, rair) + call outfld ('PSL ',psl_tmp ,pcols, lchnk ) + if (present(psl)) then + psl(:ncol) = psl_tmp(:ncol) + end if + end if +! +! Output T,q,u,v fields on pressure surfaces +! + if (hist_fld_active('T975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%t, p_surf) + call outfld('T975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%t, p_surf) + call outfld('T950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%t, p_surf) + call outfld('T900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf) + call outfld('T850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%t, p_surf) + call outfld('T800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%t, p_surf) + call outfld('T600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%t, p_surf) + call outfld('T500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%t, p_surf) + call outfld('T400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%t, p_surf) + call outfld('T300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%t, p_surf) + call outfld('T200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%t, p_surf) + call outfld('T100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%q(1,1,1), p_surf) + call outfld('Q975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%q(1,1,1), p_surf) + call outfld('Q950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%q(1,1,1), p_surf) + call outfld('Q900 ', p_surf, pcols, lchnk ) + end if if (hist_fld_active('Q850')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf) - call outfld('Q850 ', p_surf, pcols, lchnk ) + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf) + call outfld('Q850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%q(1,1,1), p_surf) + call outfld('Q800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf) + call outfld('Q700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%q(1,1,1), p_surf) + call outfld('Q600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%q(1,1,1), p_surf) + call outfld('Q500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%q(1,1,1), p_surf) + call outfld('Q400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%q(1,1,1), p_surf) + call outfld('Q300 ', p_surf, pcols, lchnk ) end if if (hist_fld_active('Q200')) then - call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%q(1,1,1), p_surf) - call outfld('Q200 ', p_surf, pcols, lchnk ) + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%q(1,1,1), p_surf) + call outfld('Q200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%q(1,1,1), p_surf) + call outfld('Q100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%q(1,1,1), p_surf) + call outfld('Q050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%q(1,1,1), p_surf) + call outfld('Q010 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%u, p_surf) + call outfld('U1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%u, p_surf) + call outfld('U975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%u, p_surf) + call outfld('U950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%u, p_surf) + call outfld('U925 ', p_surf, pcols, lchnk ) end if - ! - ! Output Q at bottom level - ! + if (hist_fld_active('U900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%u, p_surf) + call outfld('U900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%u, p_surf) + call outfld('U850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%u, p_surf) + call outfld('U800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%u, p_surf) + call outfld('U700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%u, p_surf) + call outfld('U600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%u, p_surf) + call outfld('U500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%u, p_surf) + call outfld('U400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%u, p_surf) + call outfld('U300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%u, p_surf) + call outfld('U250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%u, p_surf) + call outfld('U200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%u, p_surf) + call outfld('U100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%u, p_surf) + call outfld('U050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%u, p_surf) + call outfld('U010 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%v, p_surf) + call outfld('V1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%v, p_surf) + call outfld('V975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%v, p_surf) + call outfld('V950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%v, p_surf) + call outfld('V925 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%v, p_surf) + call outfld('V900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%v, p_surf) + call outfld('V850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%v, p_surf) + call outfld('V800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%v, p_surf) + call outfld('V700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%v, p_surf) + call outfld('V600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%v, p_surf) + call outfld('V500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%v, p_surf) + call outfld('V400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%v, p_surf) + call outfld('V300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%v, p_surf) + call outfld('V250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%v, p_surf) + call outfld('V200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%v, p_surf) + call outfld('V100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%v, p_surf) + call outfld('V050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%v, p_surf) + call outfld('V010 ', p_surf, pcols, lchnk ) + end if + + ftem(:ncol,:) = state%t(:ncol,:)*state%t(:ncol,:) + call outfld('TT ',ftem ,pcols ,lchnk ) +! +! Output U, V, T, Q, P and Z at bottom level +! + call outfld ('UBOT ', state%u(1,pver) , pcols, lchnk) + call outfld ('VBOT ', state%v(1,pver) , pcols, lchnk) call outfld ('QBOT ', state%q(1,pver,1), pcols, lchnk) + call outfld ('ZBOT ', state%zm(1,pver) , pcols, lchnk) - ! Total energy of the atmospheric column for atmospheric heat storage calculations +! Total energy of the atmospheric column for atmospheric heat storage calculations !! temporary variable to get surface geopotential in dimensions of (ncol,pver) do k=1,pver @@ -1420,423 +1722,390 @@ subroutine diag_phys_writeout_moist(state, pbuf, p_surf_t) ftem(:ncol,:) = (cpair*state%t(:ncol,:) + ftem1(:ncol,:) + latvap*state%q(:ncol,:,1) + & 0.5_r8*(state%u(:ncol,:)**2+state%v(:ncol,:)**2))*(state%pdel(:ncol,:)/gravit) !! vertically integrate - do k=2,pver - ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) end do call outfld ('ATMEINT ',ftem(:ncol,1) ,pcols ,lchnk ) - !! Boundary layer atmospheric stability, temperature, water vapor diagnostics +!! Boundary layer atmospheric stability, temperature, water vapor diagnostics - if ( hist_fld_active('THE9251000') .or. & - hist_fld_active('THE8501000') .or. & - hist_fld_active('THE7001000')) then - if (p_surf_t(1, surf_100000) < 0.0_r8) then - call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%t, p_surf_t(:, surf_100000)) - end if + if (hist_fld_active('T1000') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%t, p_surf_t1) end if - if ( hist_fld_active('TH9251000') .or. & - hist_fld_active('THE9251000')) then - if (p_surf_t(1, surf_092500) < 0.0_r8) then - call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%t, p_surf_t(:, surf_092500)) - end if + if (hist_fld_active('T925') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%t, p_surf_t2) end if - if ( hist_fld_active('Q1000') .or. & - hist_fld_active('THE9251000') .or. & - hist_fld_active('THE8501000') .or. & - hist_fld_active('THE7001000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%q(1,1,1), p_surf_q1) + if (hist_fld_active('Q1000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%q(1,1,1), p_surf_q1) end if - if (hist_fld_active('THE9251000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%q(1,1,1), p_surf_q2) + if (hist_fld_active('Q925') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%q(1,1,1), p_surf_q2) end if -!!! at 1000 mb and 925 mb - if (hist_fld_active('Q1000')) then - call outfld('Q1000 ', p_surf_q1, pcols, lchnk ) + !!! at 1000 mb and 925 mb + if (hist_fld_active('T1000')) then + call outfld('T1000 ', p_surf_t1, pcols, lchnk ) end if - if (hist_fld_active('Q925')) then - call outfld('Q925 ', p_surf_q2, pcols, lchnk ) + if (hist_fld_active('T925')) then + call outfld('T925 ', p_surf_t2, pcols, lchnk ) end if - if (hist_fld_active('THE9251000')) then - p_surf = ((p_surf_t(:, surf_092500)*(1000.0_r8/925.0_r8)**cappa) * & - exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t(:, surf_092500)))) - & - (p_surf_t(:,surf_100000)*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t(:,surf_100000))) - call outfld('THE9251000 ', p_surf, pcols, lchnk ) + if (hist_fld_active('Q1000')) then + call outfld('Q1000 ', p_surf_q1, pcols, lchnk ) end if - if (hist_fld_active('THE8501000')) then - if (p_surf_t(1, surf_085000) < 0.0_r8) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf_t(:, surf_085000)) - end if + if (hist_fld_active('Q925')) then + call outfld('Q925 ', p_surf_q2, pcols, lchnk ) end if -!!! at 1000 mb and 850 mb - if (hist_fld_active('THE8501000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf_q2) - p_surf = ((p_surf_t(:, surf_085000)*(1000.0_r8/850.0_r8)**cappa) * & - exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t(:, surf_085000)))) - & - (p_surf_t(:,surf_100000)*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t(:,surf_100000))) - call outfld('THE8501000 ', p_surf, pcols, lchnk ) + if (hist_fld_active('T9251000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T9251000 ', p_surf, pcols, lchnk ) end if - if (hist_fld_active('THE7001000')) then - if (p_surf_t(1, surf_070000) < 0.0_r8) then - call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%t, p_surf_t(:, surf_070000)) - end if + if (hist_fld_active('TH9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH9251000 ', p_surf, pcols, lchnk ) end if -!!! at 1000 mb and 700 mb - if (hist_fld_active('THE7001000')) then - call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf_q2) - p_surf = ((p_surf_t(:, surf_070000)*(1000.0_r8/700.0_r8)**cappa) * & - exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t(:, surf_070000)))) - & - (p_surf_t(:,surf_100000)*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t(:,surf_100000))) - call outfld('THE7001000 ', p_surf, pcols, lchnk ) + if (hist_fld_active('THE9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE9251000 ', p_surf, pcols, lchnk ) end if - return - end subroutine diag_phys_writeout_moist - -!=============================================================================== - - subroutine diag_phys_writeout(state, pbuf) - - !----------------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(inout) :: state - type(physics_buffer_desc), pointer :: pbuf(:) - - ! Local variable - real(r8) :: p_surf_t(pcols, nsurf) ! data interpolated to a pressure surface - - call diag_phys_writeout_dry(state, pbuf, p_surf_t) - - if (moist_physics) then - call diag_phys_writeout_moist(state, pbuf, p_surf_t) + if (hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf_t2) end if - end subroutine diag_phys_writeout - -!=============================================================================== + !!! at 1000 mb and 850 mb + if (hist_fld_active('T8501000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T8501000 ', p_surf, pcols, lchnk ) + end if - subroutine diag_conv(state, ztodt, pbuf) + if (hist_fld_active('TH8501000')) then + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH8501000 ', p_surf, pcols, lchnk ) + end if - !----------------------------------------------------------------------- - ! - ! Output diagnostics associated with all convective processes. - ! - !----------------------------------------------------------------------- - use physconst, only: cpair - use tidal_diag, only: get_tidal_coeffs + if (hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE8501000 ', p_surf, pcols, lchnk ) + end if - ! Arguments: + if (hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('T700') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%t, p_surf_t2) + end if - real(r8), intent(in) :: ztodt ! timestep for computing physics tendencies - type(physics_state), intent(in) :: state - type(physics_buffer_desc), pointer :: pbuf(:) + !!! at 700 mb + if (hist_fld_active('T700')) then + call outfld('T700 ', p_surf_t2, pcols, lchnk ) + end if - ! convective precipitation variables - real(r8), pointer :: prec_dp(:) ! total precipitation from ZM convection - real(r8), pointer :: snow_dp(:) ! snow from ZM convection - real(r8), pointer :: prec_sh(:) ! total precipitation from Hack convection - real(r8), pointer :: snow_sh(:) ! snow from Hack convection - real(r8), pointer :: prec_sed(:) ! total precipitation from ZM convection - real(r8), pointer :: snow_sed(:) ! snow from ZM convection - real(r8), pointer :: prec_pcw(:) ! total precipitation from Hack convection - real(r8), pointer :: snow_pcw(:) ! snow from Hack convection + !!! at 1000 mb and 700 mb + if (hist_fld_active('T7001000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T7001000 ', p_surf, pcols, lchnk ) + end if - ! Local variables: + if (hist_fld_active('TH7001000')) then + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH7001000 ', p_surf, pcols, lchnk ) + end if - integer :: i, k, m, lchnk, ncol + if (hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE7001000 ', p_surf, pcols, lchnk ) + end if - real(r8) :: rtdt + if (hist_fld_active('T050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%t, p_surf) + call outfld('T050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%t, p_surf) + call outfld('T010 ', p_surf, pcols, lchnk ) + end if - real(r8):: precc(pcols) ! convective precip rate - real(r8):: precl(pcols) ! stratiform precip rate - real(r8):: snowc(pcols) ! convective snow rate - real(r8):: snowl(pcols) ! stratiform snow rate - real(r8):: prect(pcols) ! total (conv+large scale) precip rate - real(r8) :: dcoef(6) ! for tidal component of T tend - lchnk = state%lchnk - ncol = state%ncol + !--------------------------------------------------------- + ! WACCM tidal diagnostics + !--------------------------------------------------------- - rtdt = 1._r8/ztodt + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then - if (moist_physics) then - if (prec_dp_idx > 0) then - call pbuf_get_field(pbuf, prec_dp_idx, prec_dp) - else - nullify(prec_dp) - end if - if (snow_dp_idx > 0) then - call pbuf_get_field(pbuf, snow_dp_idx, snow_dp) - else - nullify(snow_dp) - end if - if (prec_sh_idx > 0) then - call pbuf_get_field(pbuf, prec_sh_idx, prec_sh) - else - nullify(prec_sh) - end if - if (snow_sh_idx > 0) then - call pbuf_get_field(pbuf, snow_sh_idx, snow_sh) - else - nullify(snow_sh) - end if - if (prec_sed_idx > 0) then - call pbuf_get_field(pbuf, prec_sed_idx, prec_sed) - else - nullify(prec_sed) - end if - if (snow_sed_idx > 0) then - call pbuf_get_field(pbuf, snow_sed_idx, snow_sed) - else - nullify(snow_sed) - end if - if (prec_pcw_idx > 0) then - call pbuf_get_field(pbuf, prec_pcw_idx, prec_pcw) - else - nullify(prec_pcw) - end if - if (snow_pcw_idx > 0) then - call pbuf_get_field(pbuf, snow_pcw_idx, snow_pcw) - else - nullify(snow_pcw) - end if + call tidal_diag_write(state) - ! Precipitation rates (multi-process) - if (associated(prec_dp) .and. associated(prec_sh)) then - precc(:ncol) = prec_dp(:ncol) + prec_sh(:ncol) - else if (associated(prec_dp)) then - precc(:ncol) = prec_dp(:ncol) - else if (associated(prec_sh)) then - precc(:ncol) = prec_sh(:ncol) - else - precc(:ncol) = 0._r8 - end if - if (associated(prec_sed) .and. associated(prec_pcw)) then - precl(:ncol) = prec_sed(:ncol) + prec_pcw(:ncol) - else if (associated(prec_sed)) then - precl(:ncol) = prec_sed(:ncol) - else if (associated(prec_pcw)) then - precl(:ncol) = prec_pcw(:ncol) - else - precl(:ncol) = 0._r8 - end if - if (associated(snow_dp) .and. associated(snow_sh)) then - snowc(:ncol) = snow_dp(:ncol) + snow_sh(:ncol) - else if (associated(snow_dp)) then - snowc(:ncol) = snow_dp(:ncol) - else if (associated(snow_sh)) then - snowc(:ncol) = snow_sh(:ncol) - else - snowc(:ncol) = 0._r8 - end if - if (associated(snow_sed) .and. associated(snow_pcw)) then - snowl(:ncol) = snow_sed(:ncol) + snow_pcw(:ncol) - else if (associated(snow_sed)) then - snowl(:ncol) = snow_sed(:ncol) - else if (associated(snow_pcw)) then - snowl(:ncol) = snow_pcw(:ncol) - else - snowl(:ncol) = 0._r8 - end if - prect(:ncol) = precc(:ncol) + precl(:ncol) + endif - call outfld('PRECC ', precc, pcols, lchnk ) - call outfld('PRECL ', precl, pcols, lchnk ) - if (associated(prec_pcw)) then - call outfld('PREC_PCW', prec_pcw,pcols ,lchnk ) - end if - if (associated(prec_dp)) then - call outfld('PREC_zmc', prec_dp ,pcols ,lchnk ) - end if - call outfld('PRECSC ', snowc, pcols, lchnk ) - call outfld('PRECSL ', snowl, pcols, lchnk ) - call outfld('PRECT ', prect, pcols, lchnk ) - call outfld('PRECTMX ', prect, pcols, lchnk ) + return + end subroutine diag_phys_writeout +!=============================================================================== - call outfld('PRECLav ', precl, pcols, lchnk ) - call outfld('PRECCav ', precc, pcols, lchnk ) +subroutine diag_conv(state, ztodt, pbuf) + +!----------------------------------------------------------------------- +! +! Output diagnostics associated with all convective processes. +! +!----------------------------------------------------------------------- + use physconst, only: cpair + use tidal_diag, only: get_tidal_coeffs + +! Arguments: + + real(r8), intent(in) :: ztodt ! timestep for computing physics tendencies + type(physics_state), intent(in) :: state + type(physics_buffer_desc), pointer :: pbuf(:) + +! convective precipitation variables + real(r8), pointer :: prec_dp(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_dp(:) ! snow from ZM convection + real(r8), pointer :: prec_sh(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_sh(:) ! snow from Hack convection + real(r8), pointer :: prec_sed(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_sed(:) ! snow from ZM convection + real(r8), pointer :: prec_pcw(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_pcw(:) ! snow from Hack convection + +! Local variables: + + integer :: i, k, m, lchnk, ncol + + real(r8) :: rtdt + + real(r8):: precc(pcols) ! convective precip rate + real(r8):: precl(pcols) ! stratiform precip rate + real(r8):: snowc(pcols) ! convective snow rate + real(r8):: snowl(pcols) ! stratiform snow rate + real(r8):: prect(pcols) ! total (conv+large scale) precip rate + real(r8) :: dcoef(4) ! for tidal component of T tend + + lchnk = state%lchnk + ncol = state%ncol + + rtdt = 1._r8/ztodt + + call pbuf_get_field(pbuf, prec_dp_idx, prec_dp) + call pbuf_get_field(pbuf, snow_dp_idx, snow_dp) + call pbuf_get_field(pbuf, prec_sh_idx, prec_sh) + call pbuf_get_field(pbuf, snow_sh_idx, snow_sh) + call pbuf_get_field(pbuf, prec_sed_idx, prec_sed) + call pbuf_get_field(pbuf, snow_sed_idx, snow_sed) + call pbuf_get_field(pbuf, prec_pcw_idx, prec_pcw) + call pbuf_get_field(pbuf, snow_pcw_idx, snow_pcw) + +! Precipitation rates (multi-process) + precc(:ncol) = prec_dp(:ncol) + prec_sh(:ncol) + precl(:ncol) = prec_sed(:ncol) + prec_pcw(:ncol) + snowc(:ncol) = snow_dp(:ncol) + snow_sh(:ncol) + snowl(:ncol) = snow_sed(:ncol) + snow_pcw(:ncol) + prect(:ncol) = precc(:ncol) + precl(:ncol) + + call outfld('PRECC ', precc, pcols, lchnk ) + call outfld('PRECL ', precl, pcols, lchnk ) + call outfld('PREC_PCW', prec_pcw,pcols ,lchnk ) + call outfld('PREC_zmc', prec_dp ,pcols ,lchnk ) + call outfld('PRECSC ', snowc, pcols, lchnk ) + call outfld('PRECSL ', snowl, pcols, lchnk ) + call outfld('PRECT ', prect, pcols, lchnk ) + call outfld('PRECTMX ', prect, pcols, lchnk ) + + call outfld('PRECLav ', precl, pcols, lchnk ) + call outfld('PRECCav ', precc, pcols, lchnk ) #if ( defined BFB_CAM_SCAM_IOP ) - call outfld('Prec ' , prect, pcols, lchnk ) + call outfld('Prec ' , prect, pcols, lchnk ) #endif - ! Total convection tendencies. + ! Total convection tendencies. - do k = 1, pver - do i = 1, ncol - dtcond(i,k,lchnk) = (state%t(i,k) - dtcond(i,k,lchnk))*rtdt - end do + do k = 1, pver + do i = 1, ncol + dtcond(i,k,lchnk) = (state%s(i,k) - dtcond(i,k,lchnk))*rtdt / cpair end do - call outfld('DTCOND ', dtcond(:,:,lchnk), pcols, lchnk) - - ! output tidal coefficients - call get_tidal_coeffs( dcoef ) - call outfld( 'DTCOND_24_SIN', dtcond(:ncol,:,lchnk)*dcoef(1), ncol, lchnk ) - call outfld( 'DTCOND_24_COS', dtcond(:ncol,:,lchnk)*dcoef(2), ncol, lchnk ) - call outfld( 'DTCOND_12_SIN', dtcond(:ncol,:,lchnk)*dcoef(3), ncol, lchnk ) - call outfld( 'DTCOND_12_COS', dtcond(:ncol,:,lchnk)*dcoef(4), ncol, lchnk ) - call outfld( 'DTCOND_08_SIN', dtcond(:ncol,:,lchnk)*dcoef(5), ncol, lchnk ) - call outfld( 'DTCOND_08_COS', dtcond(:ncol,:,lchnk)*dcoef(6), ncol, lchnk ) - - do m = 1, dqcond_num - if ( cnst_cam_outfld(m) ) then - do k = 1, pver + end do + call outfld('DTCOND ', dtcond(:,:,lchnk), pcols, lchnk) + + ! output tidal coefficients + call get_tidal_coeffs( dcoef ) + call outfld( 'DTCOND_24_SIN', dtcond(:ncol,:,lchnk)*dcoef(1), ncol, lchnk ) + call outfld( 'DTCOND_24_COS', dtcond(:ncol,:,lchnk)*dcoef(2), ncol, lchnk ) + call outfld( 'DTCOND_12_SIN', dtcond(:ncol,:,lchnk)*dcoef(3), ncol, lchnk ) + call outfld( 'DTCOND_12_COS', dtcond(:ncol,:,lchnk)*dcoef(4), ncol, lchnk ) + + do m = 1, dqcond_num + if ( cnst_cam_outfld(m) ) then + do k = 1, pver do i = 1, ncol - dqcond(m)%cnst(i,k,lchnk) = (state%q(i,k,m) - dqcond(m)%cnst(i,k,lchnk))*rtdt + dqcond(m)%cnst(i,k,lchnk) = (state%q(i,k,m) - dqcond(m)%cnst(i,k,lchnk))*rtdt end do - end do - call outfld(dcconnam(m), dqcond(m)%cnst(:,:,lchnk), pcols, lchnk) - end if - end do + end do + call outfld(dcconnam(m), dqcond(m)%cnst(:,:,lchnk), pcols, lchnk) + end if + end do - end if - end subroutine diag_conv +end subroutine diag_conv !=============================================================================== - subroutine diag_surf (cam_in, cam_out, state, pbuf) +subroutine diag_surf (cam_in, cam_out, ps, trefmxav, trefmnav ) - !----------------------------------------------------------------------- - ! - ! Purpose: record surface diagnostics - ! - !----------------------------------------------------------------------- +!----------------------------------------------------------------------- +! +! Purpose: record surface diagnostics +! +!----------------------------------------------------------------------- - use time_manager, only: is_end_curr_day - use co2_cycle, only: c_i, co2_transport - use constituents, only: sflxnam + use time_manager, only: is_end_curr_day + use co2_cycle, only: c_i, co2_transport + use constituents, only: sflxnam - !----------------------------------------------------------------------- - ! - ! Input arguments - ! +!----------------------------------------------------------------------- +! +! Input arguments +! type(cam_in_t), intent(in) :: cam_in type(cam_out_t), intent(in) :: cam_out - type(physics_state), intent(in) :: state - type(physics_buffer_desc), pointer :: pbuf(:) - ! - !---------------------------Local workspace----------------------------- - ! + + real(r8), intent(inout) :: trefmnav(pcols) ! daily minimum tref + real(r8), intent(inout) :: trefmxav(pcols) ! daily maximum tref + + real(r8), intent(in) :: ps(pcols) ! Surface pressure. +! +!---------------------------Local workspace----------------------------- +! integer :: i, k, m ! indexes integer :: lchnk ! chunk identifier integer :: ncol ! longitude dimension real(r8) tem2(pcols) ! temporary workspace real(r8) ftem(pcols) ! temporary workspace - - real(r8), pointer :: trefmnav(:) ! daily minimum tref - real(r8), pointer :: trefmxav(:) ! daily maximum tref - - ! - !----------------------------------------------------------------------- - ! +! +!----------------------------------------------------------------------- +! lchnk = cam_in%lchnk ncol = cam_in%ncol - if (moist_physics) then - call outfld('SHFLX', cam_in%shf, pcols, lchnk) - call outfld('LHFLX', cam_in%lhf, pcols, lchnk) - call outfld('QFLX', cam_in%cflx(1,1), pcols, lchnk) - - call outfld('TAUX', cam_in%wsx, pcols, lchnk) - call outfld('TAUY', cam_in%wsy, pcols, lchnk) - call outfld('TREFHT ', cam_in%tref, pcols, lchnk) - call outfld('TREFHTMX', cam_in%tref, pcols, lchnk) - call outfld('TREFHTMN', cam_in%tref, pcols, lchnk) - call outfld('QREFHT', cam_in%qref, pcols, lchnk) - call outfld('U10', cam_in%u10, pcols, lchnk) - ! - ! Calculate and output reference height RH (RHREFHT) + call outfld('SHFLX', cam_in%shf, pcols, lchnk) + call outfld('LHFLX', cam_in%lhf, pcols, lchnk) + call outfld('QFLX', cam_in%cflx(1,1), pcols, lchnk) - call qsat(cam_in%tref(:ncol), state%ps(:ncol), tem2(:ncol), ftem(:ncol)) - ftem(:ncol) = cam_in%qref(:ncol)/ftem(:ncol)*100._r8 + call outfld('TAUX', cam_in%wsx, pcols, lchnk) + call outfld('TAUY', cam_in%wsy, pcols, lchnk) + call outfld('TREFHT ', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMX', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMN', cam_in%tref, pcols, lchnk) + call outfld('QREFHT', cam_in%qref, pcols, lchnk) + call outfld('U10', cam_in%u10, pcols, lchnk) +! +! Calculate and output reference height RH (RHREFHT) + call qsat(cam_in%tref(:ncol), ps(:ncol), tem2(:ncol), ftem(:ncol)) + ftem(:ncol) = cam_in%qref(:ncol)/ftem(:ncol)*100._r8 - call outfld('RHREFHT', ftem, pcols, lchnk) + + call outfld('RHREFHT', ftem, pcols, lchnk) #if (defined BFB_CAM_SCAM_IOP ) - call outfld('shflx ',cam_in%shf, pcols, lchnk) - call outfld('lhflx ',cam_in%lhf, pcols, lchnk) - call outfld('trefht ',cam_in%tref, pcols, lchnk) + call outfld('shflx ',cam_in%shf, pcols, lchnk) + call outfld('lhflx ',cam_in%lhf, pcols, lchnk) + call outfld('trefht ',cam_in%tref, pcols, lchnk) #endif - ! - ! Ouput ocn and ice fractions - ! - call outfld('LANDFRAC', cam_in%landfrac, pcols, lchnk) - call outfld('ICEFRAC', cam_in%icefrac, pcols, lchnk) - call outfld('OCNFRAC', cam_in%ocnfrac, pcols, lchnk) - ! - ! Compute daily minimum and maximum of TREF - ! - call pbuf_get_field(pbuf, trefmxav_idx, trefmxav) - call pbuf_get_field(pbuf, trefmnav_idx, trefmnav) - do i = 1,ncol - trefmxav(i) = max(cam_in%tref(i),trefmxav(i)) - trefmnav(i) = min(cam_in%tref(i),trefmnav(i)) - end do - if (is_end_curr_day()) then - call outfld('TREFMXAV', trefmxav,pcols, lchnk ) - call outfld('TREFMNAV', trefmnav,pcols, lchnk ) - trefmxav(:ncol) = -1.0e36_r8 - trefmnav(:ncol) = 1.0e36_r8 - endif +! +! Ouput ocn and ice fractions +! + call outfld('LANDFRAC', cam_in%landfrac, pcols, lchnk) + call outfld('ICEFRAC', cam_in%icefrac, pcols, lchnk) + call outfld('OCNFRAC', cam_in%ocnfrac, pcols, lchnk) +! +! Compute daily minimum and maximum of TREF +! + do i = 1,ncol + trefmxav(i) = max(cam_in%tref(i),trefmxav(i)) + trefmnav(i) = min(cam_in%tref(i),trefmnav(i)) + end do + if (is_end_curr_day()) then + call outfld('TREFMXAV', trefmxav,pcols, lchnk ) + call outfld('TREFMNAV', trefmnav,pcols, lchnk ) + trefmxav(:ncol) = -1.0e36_r8 + trefmnav(:ncol) = 1.0e36_r8 + endif - call outfld('TBOT', cam_out%tbot, pcols, lchnk) - call outfld('TS', cam_in%ts, pcols, lchnk) - call outfld('TSMN', cam_in%ts, pcols, lchnk) - call outfld('TSMX', cam_in%ts, pcols, lchnk) - call outfld('SNOWHLND', cam_in%snowhland, pcols, lchnk) - call outfld('SNOWHICE', cam_in%snowhice, pcols, lchnk) - call outfld('ASDIR', cam_in%asdir, pcols, lchnk) - call outfld('ASDIF', cam_in%asdif, pcols, lchnk) - call outfld('ALDIR', cam_in%aldir, pcols, lchnk) - call outfld('ALDIF', cam_in%aldif, pcols, lchnk) - call outfld('SST', cam_in%sst, pcols, lchnk) - - if (co2_transport()) then - do m = 1,4 + call outfld('TBOT', cam_out%tbot, pcols, lchnk) + call outfld('TS', cam_in%ts, pcols, lchnk) + call outfld('TSMN', cam_in%ts, pcols, lchnk) + call outfld('TSMX', cam_in%ts, pcols, lchnk) + call outfld('SNOWHLND', cam_in%snowhland, pcols, lchnk) + call outfld('SNOWHICE', cam_in%snowhice, pcols, lchnk) + call outfld('ASDIR', cam_in%asdir, pcols, lchnk) + call outfld('ASDIF', cam_in%asdif, pcols, lchnk) + call outfld('ALDIR', cam_in%aldir, pcols, lchnk) + call outfld('ALDIF', cam_in%aldif, pcols, lchnk) + call outfld('SST', cam_in%sst, pcols, lchnk) + + if (co2_transport()) then + do m = 1,4 call outfld(sflxnam(c_i(m)), cam_in%cflx(:,c_i(m)), pcols, lchnk) - end do - end if + end do end if - end subroutine diag_surf +end subroutine diag_surf !=============================================================================== - subroutine diag_export(cam_out) +subroutine diag_export(cam_out) - !----------------------------------------------------------------------- - ! - ! Purpose: Write export state to history file - ! - !----------------------------------------------------------------------- +!----------------------------------------------------------------------- +! +! Purpose: Write export state to history file +! +!----------------------------------------------------------------------- - ! arguments - type(cam_out_t), intent(inout) :: cam_out + ! arguments + type(cam_out_t), intent(inout) :: cam_out - ! Local variables: - integer :: lchnk ! chunk identifier - logical :: atm_dep_flux ! true ==> sending deposition fluxes to coupler. - ! Otherwise, set them to zero. - !----------------------------------------------------------------------- + ! Local variables: + integer :: lchnk ! chunk identifier + logical :: atm_dep_flux ! true ==> sending deposition fluxes to coupler. + ! Otherwise, set them to zero. + !----------------------------------------------------------------------- - lchnk = cam_out%lchnk + lchnk = cam_out%lchnk - call phys_getopts(atm_dep_flux_out=atm_dep_flux) + call phys_getopts(atm_dep_flux_out=atm_dep_flux) - if (.not. atm_dep_flux) then + if (.not. atm_dep_flux) then ! set the fluxes to zero before outfld and sending them to the ! coupler cam_out%bcphiwet = 0.0_r8 @@ -1853,60 +2122,58 @@ subroutine diag_export(cam_out) cam_out%dstdry3 = 0.0_r8 cam_out%dstwet4 = 0.0_r8 cam_out%dstdry4 = 0.0_r8 - end if + end if - if (moist_physics) then - call outfld('a2x_BCPHIWET', cam_out%bcphiwet, pcols, lchnk) - call outfld('a2x_BCPHIDRY', cam_out%bcphidry, pcols, lchnk) - call outfld('a2x_BCPHODRY', cam_out%bcphodry, pcols, lchnk) - call outfld('a2x_OCPHIWET', cam_out%ocphiwet, pcols, lchnk) - call outfld('a2x_OCPHIDRY', cam_out%ocphidry, pcols, lchnk) - call outfld('a2x_OCPHODRY', cam_out%ocphodry, pcols, lchnk) - call outfld('a2x_DSTWET1', cam_out%dstwet1, pcols, lchnk) - call outfld('a2x_DSTDRY1', cam_out%dstdry1, pcols, lchnk) - call outfld('a2x_DSTWET2', cam_out%dstwet2, pcols, lchnk) - call outfld('a2x_DSTDRY2', cam_out%dstdry2, pcols, lchnk) - call outfld('a2x_DSTWET3', cam_out%dstwet3, pcols, lchnk) - call outfld('a2x_DSTDRY3', cam_out%dstdry3, pcols, lchnk) - call outfld('a2x_DSTWET4', cam_out%dstwet4, pcols, lchnk) - call outfld('a2x_DSTDRY4', cam_out%dstdry4, pcols, lchnk) - end if - - end subroutine diag_export + call outfld('a2x_BCPHIWET', cam_out%bcphiwet, pcols, lchnk) + call outfld('a2x_BCPHIDRY', cam_out%bcphidry, pcols, lchnk) + call outfld('a2x_BCPHODRY', cam_out%bcphodry, pcols, lchnk) + call outfld('a2x_OCPHIWET', cam_out%ocphiwet, pcols, lchnk) + call outfld('a2x_OCPHIDRY', cam_out%ocphidry, pcols, lchnk) + call outfld('a2x_OCPHODRY', cam_out%ocphodry, pcols, lchnk) + call outfld('a2x_DSTWET1', cam_out%dstwet1, pcols, lchnk) + call outfld('a2x_DSTDRY1', cam_out%dstdry1, pcols, lchnk) + call outfld('a2x_DSTWET2', cam_out%dstwet2, pcols, lchnk) + call outfld('a2x_DSTDRY2', cam_out%dstdry2, pcols, lchnk) + call outfld('a2x_DSTWET3', cam_out%dstwet3, pcols, lchnk) + call outfld('a2x_DSTDRY3', cam_out%dstdry3, pcols, lchnk) + call outfld('a2x_DSTWET4', cam_out%dstwet4, pcols, lchnk) + call outfld('a2x_DSTDRY4', cam_out%dstdry4, pcols, lchnk) + +end subroutine diag_export !####################################################################### - subroutine diag_physvar_ic (lchnk, pbuf, cam_out, cam_in) - ! - !--------------------------------------------- - ! - ! Purpose: record physics variables on IC file - ! - !--------------------------------------------- - ! - - ! - ! Arguments - ! - integer , intent(in) :: lchnk ! chunk identifier - type(physics_buffer_desc), pointer :: pbuf(:) - - type(cam_out_t), intent(inout) :: cam_out - type(cam_in_t), intent(inout) :: cam_in - ! - !---------------------------Local workspace----------------------------- - ! - integer :: k ! indices - integer :: itim_old ! indices - - real(r8), pointer, dimension(:,:) :: cwat_var - real(r8), pointer, dimension(:,:) :: conv_var_3d - real(r8), pointer, dimension(: ) :: conv_var_2d - real(r8), pointer :: tpert(:), pblh(:), qpert(:) - ! - !----------------------------------------------------------------------- - ! - if( write_inithist() .and. moist_physics ) then + subroutine diag_physvar_ic (lchnk, pbuf, cam_out, cam_in) +! +!--------------------------------------------- +! +! Purpose: record physics variables on IC file +! +!--------------------------------------------- +! + +! +! Arguments +! + integer , intent(in) :: lchnk ! chunk identifier + type(physics_buffer_desc), pointer :: pbuf(:) + + type(cam_out_t), intent(inout) :: cam_out + type(cam_in_t), intent(inout) :: cam_in +! +!---------------------------Local workspace----------------------------- +! + integer :: k ! indices + integer :: itim_old ! indices + + real(r8), pointer, dimension(:,:) :: cwat_var + real(r8), pointer, dimension(:,:) :: conv_var_3d + real(r8), pointer, dimension(: ) :: conv_var_2d + real(r8), pointer :: tpert(:), pblh(:), qpert(:) +! +!----------------------------------------------------------------------- +! + if( write_inithist() ) then ! ! Associate pointers with physics buffer fields @@ -1914,363 +2181,207 @@ subroutine diag_physvar_ic (lchnk, pbuf, cam_out, cam_in) itim_old = pbuf_old_tim_idx() if (qcwat_idx > 0) then - call pbuf_get_field(pbuf, qcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) - call outfld('QCWAT&IC ',cwat_var, pcols,lchnk) + call pbuf_get_field(pbuf, qcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('QCWAT&IC ',cwat_var, pcols,lchnk) end if if (tcwat_idx > 0) then - call pbuf_get_field(pbuf, tcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) - call outfld('TCWAT&IC ',cwat_var, pcols,lchnk) + call pbuf_get_field(pbuf, tcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('TCWAT&IC ',cwat_var, pcols,lchnk) end if if (lcwat_idx > 0) then - call pbuf_get_field(pbuf, lcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) - call outfld('LCWAT&IC ',cwat_var, pcols,lchnk) + call pbuf_get_field(pbuf, lcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('LCWAT&IC ',cwat_var, pcols,lchnk) end if - if (cld_idx > 0) then - call pbuf_get_field(pbuf, cld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) - call outfld('CLOUD&IC ',cwat_var, pcols,lchnk) - end if + call pbuf_get_field(pbuf, cld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CLOUD&IC ',cwat_var, pcols,lchnk) - if (concld_idx > 0) then - call pbuf_get_field(pbuf, concld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) - call outfld('CONCLD&IC ',cwat_var, pcols,lchnk) - end if + call pbuf_get_field(pbuf, concld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CONCLD&IC ',cwat_var, pcols,lchnk) - if (cush_idx > 0) then - call pbuf_get_field(pbuf, cush_idx, conv_var_2d ,(/1,itim_old/), (/pcols,1/)) - call outfld('CUSH&IC ',conv_var_2d, pcols,lchnk) + call pbuf_get_field(pbuf, tke_idx, conv_var_3d) + call outfld('TKE&IC ',conv_var_3d, pcols,lchnk) - end if + call pbuf_get_field(pbuf, kvm_idx, conv_var_3d) + call outfld('KVM&IC ',conv_var_3d, pcols,lchnk) - if (tke_idx > 0) then - call pbuf_get_field(pbuf, tke_idx, conv_var_3d) - call outfld('TKE&IC ',conv_var_3d, pcols,lchnk) - end if + call pbuf_get_field(pbuf, kvh_idx, conv_var_3d) + call outfld('KVH&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, cush_idx, conv_var_2d ,(/1,itim_old/), (/pcols,1/)) + call outfld('CUSH&IC ',conv_var_2d, pcols,lchnk) - if (kvm_idx > 0) then - call pbuf_get_field(pbuf, kvm_idx, conv_var_3d) - call outfld('KVM&IC ',conv_var_3d, pcols,lchnk) + if (qpert_idx > 0) then + call pbuf_get_field(pbuf, qpert_idx, qpert) + call outfld('QPERT&IC ', qpert, pcols, lchnk) end if - if (kvh_idx > 0) then - call pbuf_get_field(pbuf, kvh_idx, conv_var_3d) - call outfld('KVH&IC ',conv_var_3d, pcols,lchnk) - end if + call pbuf_get_field(pbuf, pblh_idx, pblh) + call outfld('PBLH&IC ', pblh, pcols, lchnk) - if (qpert_idx > 0) then - call pbuf_get_field(pbuf, qpert_idx, qpert) - call outfld('QPERT&IC ', qpert, pcols, lchnk) - end if + call pbuf_get_field(pbuf, tpert_idx, tpert) + call outfld('TPERT&IC ', tpert, pcols, lchnk) - if (pblh_idx > 0) then - call pbuf_get_field(pbuf, pblh_idx, pblh) - call outfld('PBLH&IC ', pblh, pcols, lchnk) - end if - - if (tpert_idx > 0) then - call pbuf_get_field(pbuf, tpert_idx, tpert) - call outfld('TPERT&IC ', tpert, pcols, lchnk) - end if - - end if + ! The following is only needed for cam-csim + call outfld('TBOT&IC ', cam_out%tbot, pcols, lchnk) + end if - end subroutine diag_physvar_ic + end subroutine diag_physvar_ic !####################################################################### - subroutine diag_phys_tend_writeout_dry(state, pbuf, tend, ztodt) - - !--------------------------------------------------------------- - ! - ! Purpose: Dump physics tendencies for temperature - ! - !--------------------------------------------------------------- - - use check_energy, only: check_energy_get_integrals - use physconst, only: cpair - - ! Arguments - - type(physics_state), intent(in) :: state - - type(physics_buffer_desc), pointer :: pbuf(:) - type(physics_tend ), intent(in) :: tend - real(r8), intent(in) :: ztodt ! physics timestep - - !---------------------------Local workspace----------------------------- - - integer :: lchnk ! chunk index - integer :: ncol ! number of columns in chunk - real(r8) :: ftem2(pcols) ! Temporary workspace for outfld variables - real(r8) :: ftem3(pcols,pver) ! Temporary workspace for outfld variables - real(r8) :: heat_glob ! global energy integral (FV only) - ! CAM pointers to get variables from the physics buffer - real(r8), pointer, dimension(:,:) :: t_ttend - integer :: itim_old,m - - !----------------------------------------------------------------------- - - lchnk = state%lchnk - ncol = state%ncol - - ! Dump out post-physics state (FV only) +subroutine diag_phys_tend_writeout(state, pbuf, tend, ztodt, tmp_q, tmp_cldliq, tmp_cldice, & + tmp_t, qini, cldliqini, cldiceini) + + !--------------------------------------------------------------- + ! + ! Purpose: Dump physics tendencies for moisture and temperature + ! + !--------------------------------------------------------------- + + use check_energy, only: check_energy_get_integrals + use physconst, only: cpair + + ! Arguments + + type(physics_state), intent(in ) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) + type(physics_tend ), intent(in ) :: tend + real(r8) , intent(in ) :: ztodt ! physics timestep + real(r8) , intent(inout) :: tmp_q (pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldliq(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldice(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_t (pcols,pver) ! holds last physics_updated T (FV) + real(r8) , intent(in ) :: qini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldliqini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldiceini (pcols,pver) ! tracer fields at beginning of physics + + !---------------------------Local workspace----------------------------- + + integer :: m ! constituent index + integer :: lchnk ! chunk index + integer :: ncol ! number of columns in chunk + real(r8) :: ftem2(pcols ) ! Temporary workspace for outfld variables + real(r8) :: ftem3(pcols,pver) ! Temporary workspace for outfld variables + real(r8) :: rtdt + real(r8) :: heat_glob ! global energy integral (FV only) + integer :: ixcldice, ixcldliq! constituent indices for cloud liquid and ice water. + ! CAM pointers to get variables from the physics buffer + real(r8), pointer, dimension(:,:) :: t_ttend + integer :: itim_old + + !----------------------------------------------------------------------- + + lchnk = state%lchnk + ncol = state%ncol + rtdt = 1._r8/ztodt + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + + ! Dump out post-physics state (FV only) + + if (dycore_is('LR') .or. dycore_is('SE')) then + tmp_t(:ncol,:pver) = (tmp_t(:ncol,:pver) - state%t(:ncol,:pver))/ztodt + call outfld('PTTEND_RESID', tmp_t, pcols, lchnk ) + end if + call outfld('TAP', state%t, pcols, lchnk ) + call outfld('UAP', state%u, pcols, lchnk ) + call outfld('VAP', state%v, pcols, lchnk ) - call outfld('TAP', state%t, pcols, lchnk ) - call outfld('UAP', state%u, pcols, lchnk ) - call outfld('VAP', state%v, pcols, lchnk ) + if ( cnst_cam_outfld( 1) ) call outfld (apcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (apcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (apcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) - ! Total physics tendency for Temperature - ! (remove global fixer tendency from total for FV and SE dycores) + ! T-tendency due to FV Energy fixer (remove from total physics tendency diagnostic) - if (dycore_is('LR') .or. dycore_is('SE')) then + if (dycore_is('LR') .or. dycore_is('SE')) then call check_energy_get_integrals( heat_glob_out=heat_glob ) ftem2(:ncol) = heat_glob/cpair call outfld('TFIX', ftem2, pcols, lchnk ) ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) - heat_glob/cpair - else + else ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) - end if - call outfld('PTTEND',ftem3, pcols, lchnk ) - - ! Total (physics+dynamics, everything!) tendency for Temperature - - !! get temperature stored in physics buffer - itim_old = pbuf_old_tim_idx() - call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,itim_old/), kount=(/pcols,pver,1/)) - - !! calculate and outfld the total temperature tendency - ftem3(:ncol,:) = (state%t(:ncol,:) - t_ttend(:ncol,:))/ztodt - call outfld('TTEND_TOT', ftem3, pcols, lchnk) - - !! update physics buffer with this time-step's temperature - t_ttend(:ncol,:) = state%t(:ncol,:) - - end subroutine diag_phys_tend_writeout_dry - -!####################################################################### - - subroutine diag_phys_tend_writeout_moist(state, pbuf, tend, ztodt, & - tmp_q, tmp_cldliq, tmp_cldice, qini, cldliqini, cldiceini) - - !--------------------------------------------------------------- - ! - ! Purpose: Dump physics tendencies for moisture - ! - !--------------------------------------------------------------- - - ! Arguments - - type(physics_state), intent(in) :: state - - type(physics_buffer_desc), pointer :: pbuf(:) - type(physics_tend ), intent(in) :: tend - real(r8), intent(in) :: ztodt ! physics timestep - real(r8), intent(inout) :: tmp_q (pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(inout) :: tmp_cldliq(pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(inout) :: tmp_cldice(pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(in) :: qini (pcols,pver) ! tracer fields at beginning of physics - real(r8), intent(in) :: cldliqini (pcols,pver) ! tracer fields at beginning of physics - real(r8), intent(in) :: cldiceini (pcols,pver) ! tracer fields at beginning of physics - - !---------------------------Local workspace----------------------------- - - integer :: lchnk ! chunk index - integer :: ncol ! number of columns in chunk - real(r8) :: ftem3(pcols,pver) ! Temporary workspace for outfld variables - real(r8) :: rtdt - integer :: ixcldice, ixcldliq! constituent indices for cloud liquid and ice water. + end if - lchnk = state%lchnk - ncol = state%ncol - rtdt = 1._r8/ztodt - call cnst_get_ind('CLDLIQ', ixcldliq, abort=.false.) - call cnst_get_ind('CLDICE', ixcldice, abort=.false.) + ! Total physics tendency for Temperature - if ( cnst_cam_outfld( 1) ) then - call outfld (apcnst( 1), state%q(1,1, 1), pcols, lchnk) - end if - if (ixcldliq > 0) then - if (cnst_cam_outfld(ixcldliq)) then - call outfld (apcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) - end if - end if - if (ixcldice > 0) then - if ( cnst_cam_outfld(ixcldice) ) then - call outfld (apcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) - end if - end if + call outfld('PTTEND',ftem3, pcols, lchnk ) - ! Tendency for dry mass adjustment of q (FV only) + ! Tendency for dry mass adjustment of q (valid for FV only) - if (dycore_is('LR')) then + if (dycore_is('LR')) then tmp_q (:ncol,:pver) = (state%q(:ncol,:pver, 1) - tmp_q (:ncol,:pver))*rtdt - if (ixcldliq > 0) then - tmp_cldliq(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - tmp_cldliq(:ncol,:pver))*rtdt - else - tmp_cldliq(:ncol,:pver) = 0.0_r8 - end if - if (ixcldice > 0) then - tmp_cldice(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - tmp_cldice(:ncol,:pver))*rtdt - else - tmp_cldice(:ncol,:pver) = 0.0_r8 - end if - if ( cnst_cam_outfld( 1) ) then - call outfld (dmetendnam( 1), tmp_q , pcols, lchnk) - end if - if (ixcldliq > 0) then - if ( cnst_cam_outfld(ixcldliq) ) then - call outfld (dmetendnam(ixcldliq), tmp_cldliq, pcols, lchnk) - end if - end if - if (ixcldice > 0) then - if ( cnst_cam_outfld(ixcldice) ) then - call outfld (dmetendnam(ixcldice), tmp_cldice, pcols, lchnk) - end if - end if - end if + tmp_cldliq(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - tmp_cldliq(:ncol,:pver))*rtdt + tmp_cldice(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - tmp_cldice(:ncol,:pver))*rtdt + if ( cnst_cam_outfld( 1) ) call outfld (dmetendnam( 1), tmp_q , pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (dmetendnam(ixcldliq), tmp_cldliq, pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (dmetendnam(ixcldice), tmp_cldice, pcols, lchnk) + end if - ! Total physics tendency for moisture and other tracers + ! Total physics tendency for moisture and other tracers - if ( cnst_cam_outfld( 1) ) then + if ( cnst_cam_outfld( 1) ) then ftem3(:ncol,:pver) = (state%q(:ncol,:pver, 1) - qini (:ncol,:pver) )*rtdt call outfld (ptendnam( 1), ftem3, pcols, lchnk) - end if - if (ixcldliq > 0) then - if (cnst_cam_outfld(ixcldliq) ) then - ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - cldliqini(:ncol,:pver) )*rtdt - call outfld (ptendnam(ixcldliq), ftem3, pcols, lchnk) - end if - end if - if (ixcldice > 0) then - if ( cnst_cam_outfld(ixcldice) ) then - ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - cldiceini(:ncol,:pver) )*rtdt - call outfld (ptendnam(ixcldice), ftem3, pcols, lchnk) - end if - end if - - end subroutine diag_phys_tend_writeout_moist - -!####################################################################### - - subroutine diag_phys_tend_writeout(state, pbuf, tend, ztodt, & - tmp_q, tmp_cldliq, tmp_cldice, qini, cldliqini, cldiceini) - - !--------------------------------------------------------------- - ! - ! Purpose: Dump physics tendencies for moisture and temperature - ! - !--------------------------------------------------------------- - - ! Arguments + end if + if ( cnst_cam_outfld(ixcldliq) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - cldliqini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldliq), ftem3, pcols, lchnk) + end if + if ( cnst_cam_outfld(ixcldice) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - cldiceini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldice), ftem3, pcols, lchnk) + end if - type(physics_state), intent(in) :: state + ! Total (physics+dynamics, everything!) tendency for Temperature - type(physics_buffer_desc), pointer :: pbuf(:) - type(physics_tend ), intent(in) :: tend - real(r8), intent(in) :: ztodt ! physics timestep - real(r8) , intent(inout) :: tmp_q (pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(inout) :: tmp_cldliq(pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(inout) :: tmp_cldice(pcols,pver) ! As input, holds pre-adjusted tracers (FV) - real(r8), intent(in) :: qini (pcols,pver) ! tracer fields at beginning of physics - real(r8), intent(in) :: cldliqini (pcols,pver) ! tracer fields at beginning of physics - real(r8), intent(in) :: cldiceini (pcols,pver) ! tracer fields at beginning of physics + !! get temperature stored in physics buffer + itim_old = pbuf_old_tim_idx() + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,itim_old/), kount=(/pcols,pver,1/)) - !----------------------------------------------------------------------- + !! calculate and outfld the total temperature tendency + ftem3(:ncol,:) = (state%t(:ncol,:) - t_ttend(:ncol,:))/ztodt + call outfld('TTEND_TOT', ftem3, pcols, lchnk) - call diag_phys_tend_writeout_dry(state, pbuf, tend, ztodt) - if (moist_physics) then - call diag_phys_tend_writeout_moist(state, pbuf, tend, ztodt, & - tmp_q, tmp_cldliq, tmp_cldice, qini, cldliqini, cldiceini) - end if + !! update physics buffer with this time-step's temperature + t_ttend(:ncol,:) = state%t(:ncol,:) - end subroutine diag_phys_tend_writeout +end subroutine diag_phys_tend_writeout !####################################################################### - subroutine diag_state_b4_phys_write_dry (state) - ! - !--------------------------------------------------------------- - ! - ! Purpose: Dump dry state just prior to executing physics - ! - !--------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(in) :: state - ! - !---------------------------Local workspace----------------------------- - ! - integer :: lchnk ! chunk index - ! - !----------------------------------------------------------------------- - ! - lchnk = state%lchnk - - call outfld('TBP', state%t, pcols, lchnk ) - - end subroutine diag_state_b4_phys_write_dry - - subroutine diag_state_b4_phys_write_moist (state) - ! - !--------------------------------------------------------------- - ! - ! Purpose: Dump moist state just prior to executing physics - ! - !--------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(in) :: state - ! - !---------------------------Local workspace----------------------------- - ! - integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. - integer :: lchnk ! chunk index - ! - !----------------------------------------------------------------------- - ! - lchnk = state%lchnk - - call cnst_get_ind('CLDLIQ', ixcldliq, abort=.false.) - call cnst_get_ind('CLDICE', ixcldice, abort=.false.) - - if ( cnst_cam_outfld( 1) ) then - call outfld (bpcnst( 1), state%q(1,1, 1), pcols, lchnk) - end if - if (ixcldliq > 0) then - if (cnst_cam_outfld(ixcldliq)) then - call outfld (bpcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) - end if - end if - if (ixcldice > 0) then - if (cnst_cam_outfld(ixcldice)) then - call outfld (bpcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) - end if - end if - - end subroutine diag_state_b4_phys_write_moist - - subroutine diag_state_b4_phys_write (state) - ! - !--------------------------------------------------------------- - ! - ! Purpose: Dump state just prior to executing physics - ! - !--------------------------------------------------------------- - ! - ! Arguments - ! - type(physics_state), intent(in) :: state - ! - - call diag_state_b4_phys_write_dry(state) - if (moist_physics) then - call diag_state_b4_phys_write_moist(state) - end if - end subroutine diag_state_b4_phys_write + subroutine diag_state_b4_phys_write (state) +! +!--------------------------------------------------------------- +! +! Purpose: Dump state just prior to executing physics +! +!--------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(in) :: state +! +!---------------------------Local workspace----------------------------- +! + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: lchnk ! chunk index +! +!----------------------------------------------------------------------- +! + lchnk = state%lchnk + + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call outfld('TBP', state%t, pcols, lchnk ) + if ( cnst_cam_outfld( 1) ) call outfld (bpcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (bpcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (bpcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) + + end subroutine diag_state_b4_phys_write end module cam_diagnostics diff --git a/SourceMods/src.cam/cam_diagnostics.F90-e3sm b/SourceMods/src.cam/cam_diagnostics.F90-e3sm new file mode 100644 index 0000000..5b4bae8 --- /dev/null +++ b/SourceMods/src.cam/cam_diagnostics.F90-e3sm @@ -0,0 +1,2315 @@ +module cam_diagnostics + +!--------------------------------------------------------------------------------- +! Module to compute a variety of diagnostics quantities for history files +!--------------------------------------------------------------------------------- + +use shr_kind_mod, only: r8 => shr_kind_r8 +use camsrfexch, only: cam_in_t, cam_out_t +use physics_types, only: physics_state, physics_tend +use ppgrid, only: pcols, pver, pverp, begchunk, endchunk +use physics_buffer, only: physics_buffer_desc, pbuf_add_field, dtype_r8, dyn_time_lvls, & + pbuf_get_field, pbuf_get_index, pbuf_old_tim_idx + + + +use cam_history, only: outfld, write_inithist, hist_fld_active +use constituents, only: pcnst, cnst_name, cnst_longname, cnst_cam_outfld, ptendnam, dmetendnam, apcnst, bpcnst, & + cnst_get_ind +use chemistry, only: chem_is +use dycore, only: dycore_is +use phys_control, only: phys_getopts +use wv_saturation, only: qsat, qsat_water, svp_ice +use time_manager, only: is_first_step + +use scamMod, only: single_column, wfld +use cam_abortutils, only: endrun + +implicit none +private +save + +! Public interfaces + +public :: & + diag_register, &! register pbuf space + diag_init, &! initialization + diag_allocate, &! allocate memory for module variables + diag_deallocate, &! deallocate memory for module variables + diag_conv_tend_ini, &! initialize convective tendency calcs + diag_phys_writeout, &! output diagnostics of the dynamics + diag_phys_tend_writeout, & ! output physics tendencies + diag_state_b4_phys_write,& ! output state before physics execution + diag_conv, &! output diagnostics of convective processes + diag_surf, &! output diagnostics of the surface + diag_export, &! output export state + diag_physvar_ic, & + diag_readnl ! read namelist options + +logical, public :: inithist_all = .false. ! Flag to indicate set of fields to be + ! included on IC file + ! .false. include only required fields + ! .true. include required *and* optional fields + +! Private data + +integer :: dqcond_num ! number of constituents to compute convective +character(len=16) :: dcconnam(pcnst) ! names of convection tendencies + ! tendencies for +real(r8), allocatable :: dtcond(:,:,:) ! temperature tendency due to convection +type dqcond_t + real(r8), allocatable :: cnst(:,:,:) ! constituent tendency due to convection +end type dqcond_t +type(dqcond_t), allocatable :: dqcond(:) + +character(len=8) :: diag_cnst_conv_tend = 'q_only' ! output constituent tendencies due to convection + ! 'none', 'q_only' or 'all' + +logical :: history_amwg ! output the variables used by the AMWG diag package +logical :: history_verbose ! produce verbose history output +logical :: history_vdiag ! output the variables used by the AMWG variability diag package +logical :: history_eddy ! output the eddy variables +logical :: history_budget ! output tendencies and state variables for CAM4 + ! temperature, water vapor, cloud ice and cloud + ! liquid budgets. +integer :: history_budget_histfile_num ! output history file number for budget fields + +!Physics buffer indices +integer :: qcwat_idx = 0 +integer :: tcwat_idx = 0 +integer :: lcwat_idx = 0 +integer :: cld_idx = 0 +integer :: concld_idx = 0 +integer :: tke_idx = 0 +integer :: kvm_idx = 0 +integer :: kvh_idx = 0 +integer :: cush_idx = 0 +integer :: t_ttend_idx = 0 + +integer :: prec_dp_idx = 0 +integer :: snow_dp_idx = 0 +integer :: prec_sh_idx = 0 +integer :: snow_sh_idx = 0 +integer :: prec_sed_idx = 0 +integer :: snow_sed_idx = 0 +integer :: prec_pcw_idx = 0 +integer :: snow_pcw_idx = 0 + + +integer :: tpert_idx=-1, qpert_idx=-1, pblh_idx=-1 +logical :: prog_modal_aero +contains + +! =============================================================================== + +subroutine diag_register + + ! Request physics buffer space for fields that persist across timesteps. + call pbuf_add_field('T_TTEND', 'global', dtype_r8, (/pcols,pver,dyn_time_lvls/), t_ttend_idx) + +end subroutine diag_register + +!=============================================================================== + +subroutine diag_readnl(nlfile) + use namelist_utils, only: find_group_name + use units, only: getunit, freeunit + use mpishorthand + use spmd_utils, only: masterproc + + character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input + + ! Local variables + integer :: unitn, ierr + character(len=*), parameter :: subname = 'diag_readnl' + + namelist /cam_diag_opts/ diag_cnst_conv_tend + !----------------------------------------------------------------------------- + + if (masterproc) then + unitn = getunit() + open( unitn, file=trim(nlfile), status='old' ) + call find_group_name(unitn, 'cam_diag_opts', status=ierr) + if (ierr == 0) then + read(unitn, cam_diag_opts, iostat=ierr) + if (ierr /= 0) then + call endrun(subname // ':: ERROR reading namelist') + end if + end if + close(unitn) + call freeunit(unitn) + end if + +#ifdef SPMD + ! Broadcast namelist variables + call mpibcast(diag_cnst_conv_tend, len(diag_cnst_conv_tend), mpichar, 0, mpicom) +#endif + +end subroutine diag_readnl + +!================================================================================================ + +subroutine diag_init() + + ! Declare the history fields for which this module contains outfld calls. + + use cam_history, only: addfld, horiz_only, add_default + use constituent_burden, only: constituent_burden_init + use cam_control_mod, only: moist_physics, ideal_phys + use tidal_diag, only: tidal_diag_init + + integer :: k, m + ! Note - this is a duplication of information in ice_constants + ! Cannot put in a use statement if want to swap ice models to cice4 + integer, parameter :: plevmx = 4 ! number of subsurface levels + character(len=8), parameter :: tsnam(plevmx) = (/ 'TS1', 'TS2', 'TS3', 'TS4' /) + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: ierr + + call phys_getopts(prog_modal_aero_out = prog_modal_aero ) + + ! outfld calls in diag_phys_writeout + + call addfld ('NSTEP',horiz_only, 'A','timestep','Model timestep') + call addfld ('PHIS',horiz_only, 'I','m2/s2','Surface geopotential') + + call addfld ('PS',horiz_only, 'A','Pa','Surface pressure') + call addfld ('T',(/ 'lev' /), 'A','K','Temperature') + call addfld ('U',(/ 'lev' /), 'A','m/s','Zonal wind') + call addfld ('V',(/ 'lev' /), 'A','m/s','Meridional wind') + call addfld (cnst_name(1),(/ 'lev' /), 'A','kg/kg',cnst_longname(1)) + + ! State before physics + call addfld ('TBP',(/ 'lev' /), 'A','K','Temperature (before physics)' ) + call addfld (bpcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (before physics)') + ! State after physics + call addfld ('TAP',(/ 'lev' /), 'A','K','Temperature (after physics)' ) + call addfld ('UAP',(/ 'lev' /), 'A','m/s','Zonal wind (after physics)' ) + call addfld ('VAP',(/ 'lev' /), 'A','m/s','Meridional wind (after physics)' ) + call addfld (apcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (after physics)') + if ( dycore_is('LR') .or. dycore_is('SE') ) then + call addfld ('TFIX',horiz_only, 'A' ,'K/s','T fixer (T equivalent of Energy correction)') + call addfld ('PTTEND_RESID',(/ 'lev' /), 'A' ,'K/s',& + 'T-tendency due to BAB kluge at end of tphysac (diagnostic not part of T-budget)' ) + end if + call addfld ('TTEND_TOT',(/ 'lev' /), 'A','K/s' ,'Total temperature tendency' ) + + ! column burdens for all constituents except water vapor + call constituent_burden_init + + call addfld ('Z3',(/ 'lev' /), 'A','m','Geopotential Height (above sea level)') + call addfld ('Z1000',horiz_only, 'A','m','Geopotential Z at 1000 mbar pressure surface') + call addfld ('Z975',horiz_only, 'A','m','Geopotential Z at 975 mbar pressure surface') + call addfld ('Z950',horiz_only, 'A','m','Geopotential Z at 950 mbar pressure surface') + call addfld ('Z925',horiz_only, 'A','m','Geopotential Z at 925 mbar pressure surface') + call addfld ('Z900',horiz_only, 'A','m','Geopotential Z at 900 mbar pressure surface') + call addfld ('Z850',horiz_only, 'A','m','Geopotential Z at 850 mbar pressure surface') + call addfld ('Z800',horiz_only, 'A','m','Geopotential Z at 800 mbar pressure surface') + call addfld ('Z700',horiz_only, 'A','m','Geopotential Z at 700 mbar pressure surface') + call addfld ('Z600',horiz_only, 'A','m','Geopotential Z at 600 mbar pressure surface') + call addfld ('Z500',horiz_only, 'A','m','Geopotential Z at 500 mbar pressure surface') + call addfld ('Z400',horiz_only, 'A','m','Geopotential Z at 400 mbar pressure surface') + call addfld ('Z300',horiz_only, 'A','m','Geopotential Z at 300 mbar pressure surface') + call addfld ('Z200',horiz_only, 'A','m','Geopotential Z at 200 mbar pressure surface') + call addfld ('Z100',horiz_only, 'A','m','Geopotential Z at 100 mbar pressure surface') + call addfld ('Z050',horiz_only, 'A','m','Geopotential Z at 50 mbar pressure surface') + + call addfld ('ZZ',(/ 'lev' /), 'A','m2','Eddy height variance' ) + call addfld ('VZ',(/ 'lev' /), 'A','m2/s','Meridional transport of geopotential energy') + call addfld ('VT',(/ 'lev' /), 'A','K m/s ','Meridional heat transport') + call addfld ('VU',(/ 'lev' /), 'A','m2/s2','Meridional flux of zonal momentum' ) + call addfld ('VV',(/ 'lev' /), 'A','m2/s2','Meridional velocity squared' ) + + if(prog_modal_aero) then !Only for prognostic aerosols + call addfld ('bc_a1_2',(/ 'lev' /), 'A','kg2/kg2','bc_a1 squared') + call addfld ('dst_a1_2',(/ 'lev' /), 'A','kg2/kg2','dst_a1 squared') + call addfld ('dst_a3_2',(/ 'lev' /), 'A','kg2/kg2','dst_a3 squared') + call addfld ('ncl_a1_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a1 squared') + call addfld ('ncl_a2_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a2 squared') + call addfld ('ncl_a3_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a3 squared') + call addfld ('so4_a1_2',(/ 'lev' /), 'A','kg2/kg2','so4_a1 squared') + call addfld ('so4_a2_2',(/ 'lev' /), 'A','kg2/kg2','so4_a2 squared') + call addfld ('so4_a3_2',(/ 'lev' /), 'A','kg2/kg2','so4_a3 squared') + call addfld ('soa_a1_2',(/ 'lev' /), 'A','kg2/kg2','soa_a1 squared') + call addfld ('soa_a2_2',(/ 'lev' /), 'A','kg2/kg2','soa_a2 squared') + call addfld ('pom_a1_2',(/ 'lev' /), 'A','kg2/kg2','pom_a1 squared') + call addfld ('Vbc_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional bc_a1 transport') + call addfld ('Vdst_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a1 transport') + call addfld ('Vdst_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a3 transport') + call addfld ('Vncl_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a1 transport') + call addfld ('Vncl_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a2 transport') + call addfld ('Vncl_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a3 transport') + call addfld ('Vso4_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a1 transport') + call addfld ('Vso4_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a2 transport') + call addfld ('Vso4_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a3 transport') + call addfld ('Vsoa_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a1 transport') + call addfld ('Vsoa_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a2 transport') + call addfld ('Vpom_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional pom_a1 transport') + endif + call addfld ('VQ',(/ 'lev' /), 'A','m/skg/kg','Meridional water transport') + call addfld ('QQ',(/ 'lev' /), 'A','kg2/kg2','Eddy moisture variance') + call addfld ('OMEGAV',(/ 'lev' /) ,'A','m Pa/s2 ','Vertical flux of meridional momentum' ) + call addfld ('OMGAOMGA',(/ 'lev' /) ,'A','Pa2/s2','Vertical flux of vertical momentum' ) + call addfld ('OMEGAQ',(/ 'lev' /) ,'A','kgPa/kgs','Vertical water transport' ) + + call addfld ('UU',(/ 'lev' /), 'A','m2/s2','Zonal velocity squared' ) + call addfld ('WSPEED',(/ 'lev' /), 'X','m/s','Horizontal total wind speed maximum' ) + call addfld ('WSPDSRFMX',horiz_only, 'X','m/s','Horizontal total wind speed maximum at the surface' ) + call addfld ('WSPDSRFAV',horiz_only, 'A','m/s','Horizontal total wind speed average at the surface' ) + + call addfld ('OMEGA',(/ 'lev' /), 'A','Pa/s','Vertical velocity (pressure)') + call addfld ('OMEGAT',(/ 'lev' /), 'A','K Pa/s ','Vertical heat flux' ) + call addfld ('OMEGAU',(/ 'lev' /), 'A','m Pa/s2 ','Vertical flux of zonal momentum' ) + call addfld ('OMEGA1000',horiz_only, 'A','Pa/s','Vertical velocity at 1000 mbar pressure surface') + call addfld ('OMEGA975',horiz_only, 'A','Pa/s','Vertical velocity at 975 mbar pressure surface') + call addfld ('OMEGA950',horiz_only, 'A','Pa/s','Vertical velocity at 950 mbar pressure surface') + call addfld ('OMEGA925',horiz_only, 'A','Pa/s','Vertical velocity at 925 mbar pressure surface') + call addfld ('OMEGA900',horiz_only, 'A','Pa/s','Vertical velocity at 900 mbar pressure surface') + call addfld ('OMEGA850',horiz_only, 'A','Pa/s','Vertical velocity at 850 mbar pressure surface') + call addfld ('OMEGA800',horiz_only, 'A','Pa/s','Vertical velocity at 800 mbar pressure surface') + call addfld ('OMEGA700',horiz_only, 'A','Pa/s','Vertical velocity at 700 mbar pressure surface') + call addfld ('OMEGA600',horiz_only, 'A','Pa/s','Vertical velocity at 600 mbar pressure surface') + call addfld ('OMEGA500',horiz_only, 'A','Pa/s','Vertical velocity at 500 mbar pressure surface') + call addfld ('OMEGA400',horiz_only, 'A','Pa/s','Vertical velocity at 400 mbar pressure surface') + call addfld ('OMEGA300',horiz_only, 'A','Pa/s','Vertical velocity at 300 mbar pressure surface') + call addfld ('OMEGA200',horiz_only, 'A','Pa/s','Vertical velocity at 200 mbar pressure surface') + call addfld ('OMEGA100',horiz_only, 'A','Pa/s','Vertical velocity at 100 mbar pressure surface') + call addfld ('OMEGABOT',horiz_only, 'A','Pa/s','Lowest model level vertical velocity') + + call addfld ('RH1000',horiz_only, 'A','%','Relative humidity at 1000 mbar pressure surface') + call addfld ('RH975',horiz_only, 'A','%','Relative humidity at 975 mbar pressure surface') + call addfld ('RH950',horiz_only, 'A','%','Relative humidity at 950 mbar pressure surface') + call addfld ('RH925',horiz_only, 'A','%','Relative humidity at 925 mbar pressure surface') + call addfld ('RH900',horiz_only, 'A','%','Relative humidity at 900 mbar pressure surface') + call addfld ('RH850',horiz_only, 'A','%','Relative humidity at 850 mbar pressure surface') + call addfld ('RH800',horiz_only, 'A','%','Relative humidity at 800 mbar pressure surface') + call addfld ('RH700',horiz_only, 'A','%','Relative humidity at 700 mbar pressure surface') + call addfld ('RH600',horiz_only, 'A','%','Relative humidity at 600 mbar pressure surface') + call addfld ('RH500',horiz_only, 'A','%','Relative humidity at 500 mbar pressure surface') + call addfld ('RH400',horiz_only, 'A','%','Relative humidity at 400 mbar pressure surface') + call addfld ('RH300',horiz_only, 'A','%','Relative humidity at 300 mbar pressure surface') + call addfld ('RH200',horiz_only, 'A','%','Relative humidity at 200 mbar pressure surface') + call addfld ('RH100',horiz_only, 'A','%','Relative humidity at 100 mbar pressure surface') + call addfld ('RHBOT',horiz_only, 'A','%','Lowest model level relative humidity') + + call addfld ('MQ',(/ 'lev' /), 'A','kg/m2','Water vapor mass in layer') + call addfld ('TMQ',horiz_only, 'A','kg/m2','Total (vertically integrated) precipitable water') + call addfld ('TUQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) zonal water flux') + call addfld ('TVQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) meridional water flux') + call addfld ('TUH',horiz_only, 'A','W/m', 'Total (vertically integrated) zonal MSE flux') + call addfld ('TVH',horiz_only, 'A','W/m', 'Total (vertically integrated) meridional MSE flux') + call addfld ('DTENDTH', horiz_only, 'A', 'W/m2', 'Dynamic Tendency of Total (vertically integrated) moist static energy') + call addfld ('DTENDTQ', horiz_only, 'A', 'kg/m2/s','Dynamic Tendency of Total (vertically integrated) specific humidity') + call addfld ('RELHUM',(/ 'lev' /), 'A','percent','Relative humidity') + call addfld ('RHW',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to liquid') + call addfld ('RHI',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to ice') + call addfld ('RHCFMIP',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to water above 273 K, ice below 273 K') + call addfld ('PSL',horiz_only, 'A','Pa','Sea level pressure') + + call addfld ('T850',horiz_only, 'A','K','Temperature at 850 mbar pressure surface') + call addfld ('T500',horiz_only, 'A','K','Temperature at 500 mbar pressure surface') + call addfld ('T300',horiz_only, 'A','K','Temperature at 300 mbar pressure surface') + call addfld ('T200',horiz_only, 'A','K','Temperature at 200 mbar pressure surface') + call addfld ('Q850',horiz_only, 'A','kg/kg','Specific Humidity at 850 mbar pressure surface') + call addfld ('Q500',horiz_only, 'A','kg/kg','Specific Humidity at 500 mbar pressure surface') + call addfld ('Q200',horiz_only, 'A','kg/kg','Specific Humidity at 200 mbar pressure surface') + call addfld ('U1000',horiz_only, 'A','m/s','Zonal wind at 1000 mbar pressure surface') + call addfld ('U975',horiz_only, 'A','m/s','Zonal wind at 975 mbar pressure surface') + call addfld ('U950',horiz_only, 'A','m/s','Zonal wind at 950 mbar pressure surface') + call addfld ('U925',horiz_only, 'A','m/s','Zonal wind at 925 mbar pressure surface') + call addfld ('U900',horiz_only, 'A','m/s','Zonal wind at 900 mbar pressure surface') + call addfld ('U850',horiz_only, 'A','m/s','Zonal wind at 850 mbar pressure surface') + call addfld ('U800',horiz_only, 'A','m/s','Zonal wind at 800 mbar pressure surface') + call addfld ('U700',horiz_only, 'A','m/s','Zonal wind at 700 mbar pressure surface') + call addfld ('U600',horiz_only, 'A','m/s','Zonal wind at 600 mbar pressure surface') + call addfld ('U500',horiz_only, 'A','m/s','Zonal wind at 500 mbar pressure surface') + call addfld ('U400',horiz_only, 'A','m/s','Zonal wind at 400 mbar pressure surface') + call addfld ('U300',horiz_only, 'A','m/s','Zonal wind at 300 mbar pressure surface') + call addfld ('U250',horiz_only, 'A','m/s','Zonal wind at 250 mbar pressure surface') + call addfld ('U200',horiz_only, 'A','m/s','Zonal wind at 200 mbar pressure surface') + call addfld ('U100',horiz_only, 'A','m/s','Zonal wind at 100 mbar pressure surface') + call addfld ('U010',horiz_only, 'A','m/s','Zonal wind at 10 mbar pressure surface') + call addfld ('V1000',horiz_only, 'A','m/s','Meridional wind at 1000 mbar pressure surface') + call addfld ('V975',horiz_only, 'A','m/s','Meridional wind at 975 mbar pressure surface') + call addfld ('V950',horiz_only, 'A','m/s','Meridional wind at 950 mbar pressure surface') + call addfld ('V925',horiz_only, 'A','m/s','Meridional wind at 925 mbar pressure surface') + call addfld ('V900',horiz_only, 'A','m/s','Meridional wind at 900 mbar pressure surface') + call addfld ('V850',horiz_only, 'A','m/s','Meridional wind at 850 mbar pressure surface') + call addfld ('V800',horiz_only, 'A','m/s','Meridional wind at 800 mbar pressure surface') + call addfld ('V700',horiz_only, 'A','m/s','Meridional wind at 700 mbar pressure surface') + call addfld ('V600',horiz_only, 'A','m/s','Meridional wind at 600 mbar pressure surface') + call addfld ('V500',horiz_only, 'A','m/s','Meridional wind at 500 mbar pressure surface') + call addfld ('V400',horiz_only, 'A','m/s','Meridional wind at 400 mbar pressure surface') + call addfld ('V300',horiz_only, 'A','m/s','Meridional wind at 300 mbar pressure surface') + call addfld ('V250',horiz_only, 'A','m/s','Meridional wind at 250 mbar pressure surface') + call addfld ('V200',horiz_only, 'A','m/s','Meridional wind at 200 mbar pressure surface') + call addfld ('V100',horiz_only, 'A','m/s','Meridional wind at 100 mbar pressure surface') + + call addfld ('TT',(/ 'lev' /), 'A','K2','Eddy temperature variance' ) + + call addfld ('UBOT',horiz_only, 'A','m/s','Lowest model level zonal wind') + call addfld ('VBOT',horiz_only, 'A','m/s','Lowest model level meridional wind') + call addfld ('QBOT',horiz_only, 'A','kg/kg','Lowest model level water vapor mixing ratio') + call addfld ('ZBOT',horiz_only, 'A','m','Lowest model level height') + + call addfld ('ATMEINT',horiz_only, 'A','J/m2','Vertically integrated total atmospheric energy ') + + call addfld ('T1000',horiz_only, 'A','K','Temperature at 1000 mbar pressure surface') + call addfld ('T975',horiz_only, 'A','K','Temperature at 975 mbar pressure surface') + call addfld ('T950',horiz_only, 'A','K','Temperature at 950 mbar pressure surface') + call addfld ('T925',horiz_only, 'A','K','Temperature at 925 mbar pressure surface') + call addfld ('T900',horiz_only, 'A','K','Temperature at 900 mbar pressure surface') + call addfld ('T800',horiz_only, 'A','K','Temperature at 800 mbar pressure surface') + call addfld ('T700',horiz_only, 'A','K','Temperature at 700 mbar pressure surface') + call addfld ('T600',horiz_only, 'A','K','Temperature at 600 mbar pressure surface') + call addfld ('T400',horiz_only, 'A','K','Temperature at 400 mbar pressure surface') + call addfld ('T100',horiz_only, 'A','K','Temperature at 100 mbar pressure surface') + call addfld ('T010',horiz_only, 'A','K','Temperature at 10 mbar pressure surface') + call addfld ('Q1000',horiz_only, 'A','kg/kg','Specific Humidity at 1000 mbar pressure surface') + call addfld ('Q975',horiz_only, 'A','kg/kg','Specific Humidity at 975 mbar pressure surface') + call addfld ('Q950',horiz_only, 'A','kg/kg','Specific Humidity at 950 mbar pressure surface') + call addfld ('Q925',horiz_only, 'A','kg/kg','Specific Humidity at 925 mbar pressure surface') + call addfld ('Q900',horiz_only, 'A','kg/kg','Specific Humidity at 900 mbar pressure surface') + call addfld ('Q800',horiz_only, 'A','kg/kg','Specific Humidity at 800 mbar pressure surface') + call addfld ('Q700',horiz_only, 'A','kg/kg','Specific Humidity at 700 mbar pressure surface') + call addfld ('Q600',horiz_only, 'A','kg/kg','Specific Humidity at 600 mbar pressure surface') + call addfld ('Q400',horiz_only, 'A','kg/kg','Specific Humidity at 400 mbar pressure surface') + call addfld ('Q300',horiz_only, 'A','kg/kg','Specific Humidity at 300 mbar pressure surface') + call addfld ('Q100',horiz_only, 'A','kg/kg','Specific Humidity at 100 mbar pressure surface') + + call addfld ('T7001000',horiz_only, 'A','K','Temperature difference 700 mb - 1000 mb') + call addfld ('TH7001000',horiz_only, 'A','K','Theta difference 700 mb - 1000 mb') + call addfld ('THE7001000',horiz_only, 'A','K','ThetaE difference 700 mb - 1000 mb') + + call addfld ('T8501000',horiz_only, 'A','K','Temperature difference 850 mb - 1000 mb') + call addfld ('TH8501000',horiz_only, 'A','K','Theta difference 850 mb - 1000 mb') + call addfld ('THE8501000',horiz_only, 'A','K','ThetaE difference 850 mb - 1000 mb') + call addfld ('T9251000',horiz_only, 'A','K','Temperature difference 925 mb - 1000 mb') + call addfld ('TH9251000',horiz_only, 'A','K','Theta difference 925 mb - 1000 mb') + call addfld ('THE9251000',horiz_only, 'A','K','ThetaE difference 925 mb - 1000 mb') + + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call addfld('QRS', (/ 'lev' /), 'A', 'K/s', 'Solar heating rate') + end if + + ! ---------------------------- + ! determine default variables + ! ---------------------------- + call phys_getopts(history_amwg_out = history_amwg , & + history_verbose_out = history_verbose , & + history_vdiag_out = history_vdiag , & + history_eddy_out = history_eddy , & + history_budget_out = history_budget , & + history_budget_histfile_num_out = history_budget_histfile_num) + + if (history_amwg) then + call add_default ('PHIS ' , 1, ' ') + call add_default ('PS ' , 1, ' ') + call add_default ('T ' , 1, ' ') + call add_default ('U ' , 1, ' ') + call add_default ('V ' , 1, ' ') + call add_default (cnst_name(1), 1, ' ') + call add_default ('Z3 ' , 1, ' ') + call add_default ('OMEGA ' , 1, ' ') + call add_default ('VT ', 1, ' ') + call add_default ('VU ', 1, ' ') + call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') + + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('Vbc_a1 ', 1, ' ') + call add_default ('Vdst_a1 ', 1, ' ') + call add_default ('Vdst_a3 ', 1, ' ') + call add_default ('Vncl_a1 ', 1, ' ') + call add_default ('Vncl_a2 ', 1, ' ') + call add_default ('Vncl_a3 ', 1, ' ') + call add_default ('Vso4_a1 ', 1, ' ') + call add_default ('Vso4_a2 ', 1, ' ') + call add_default ('Vso4_a3 ', 1, ' ') + call add_default ('Vsoa_a1 ', 1, ' ') + call add_default ('Vsoa_a2 ', 1, ' ') + call add_default ('Vpom_a1 ', 1, ' ') + endif + call add_default ('UU ', 1, ' ') + call add_default ('OMEGAT ', 1, ' ') + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('bc_a1_2 ', 1, ' ') + call add_default ('dst_a1_2', 1, ' ') + call add_default ('dst_a3_2', 1, ' ') + call add_default ('ncl_a1_2', 1, ' ') + call add_default ('ncl_a2_2', 1, ' ') + call add_default ('ncl_a3_2', 1, ' ') + call add_default ('so4_a1_2', 1, ' ') + call add_default ('so4_a2_2', 1, ' ') + call add_default ('so4_a3_2', 1, ' ') + call add_default ('soa_a1_2', 1, ' ') + call add_default ('soa_a2_2', 1, ' ') + call add_default ('pom_a1_2', 1, ' ') + endif + call add_default ('TMQ ', 1, ' ') + call add_default ('PSL ', 1, ' ') + if (moist_physics) then + call add_default ('RELHUM ', 1, ' ') + end if + + ! For Tier 1b global water cycle diagostics + call add_default ('TUQ ', 1, ' ') + call add_default ('TVQ ', 1, ' ') + call add_default ('TUH ', 1, ' ') + call add_default ('TVH ', 1, ' ') + call add_default ('DTENDTQ', 1, ' ') + call add_default ('DTENDTH', 1, ' ') + end if + + if (history_vdiag) then + call add_default ('U200', 2, ' ') + call add_default ('V200', 2, ' ') + call add_default ('U850', 2, ' ') + call add_default ('U200', 3, ' ') + call add_default ('U850', 3, ' ') + call add_default ('OMEGA500', 3, ' ') + end if + + if (history_eddy) then + call add_default ('VT ', 1, ' ') + call add_default ('VU ', 1, ' ') + call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') + call add_default ('UU ', 1, ' ') + call add_default ('OMEGAT ', 1, ' ') + call add_default ('OMEGAQ ', 1, ' ') + call add_default ('OMEGAU ', 1, ' ') + call add_default ('OMEGAV ', 1, ' ') + endif + + if ( history_budget ) then + call add_default ('PHIS ' , history_budget_histfile_num, ' ') + call add_default ('PS ' , history_budget_histfile_num, ' ') + call add_default ('T ' , history_budget_histfile_num, ' ') + call add_default ('U ' , history_budget_histfile_num, ' ') + call add_default ('V ' , history_budget_histfile_num, ' ') + call add_default (cnst_name(1), history_budget_histfile_num, ' ') + call add_default ('TTEND_TOT' , history_budget_histfile_num, ' ') + + ! State before physics (FV) + call add_default ('TBP ' , history_budget_histfile_num, ' ') + call add_default (bpcnst(1) , history_budget_histfile_num, ' ') + ! State after physics (FV) + call add_default ('TAP ' , history_budget_histfile_num, ' ') + call add_default ('UAP ' , history_budget_histfile_num, ' ') + call add_default ('VAP ' , history_budget_histfile_num, ' ') + call add_default (apcnst(1) , history_budget_histfile_num, ' ') + if ( dycore_is('LR') .or. dycore_is('SE') ) then + call add_default ('TFIX ' , history_budget_histfile_num, ' ') + call add_default ('PTTEND_RESID', history_budget_histfile_num, ' ') + end if + end if + + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call add_default('QRS ', 1, ' ') + end if + + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + ! Exit here for adiabatic/ideal physics cases ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + if (.not. moist_physics) return + + + call addfld ('PDELDRY',(/ 'lev' /), 'A','Pa','Dry pressure difference between levels') + call addfld ('PSDRY',horiz_only, 'A','Pa','Surface pressure') + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + call add_default ('PS ', 2, ' ') + call add_default ('T ', 2, ' ') + end if + + ! outfld calls in diag_conv + + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call addfld ('DTCOND',(/ 'lev' /), 'A','K/s','T tendency - moist processes') + call addfld ('DTCOND_24_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. cos coeff.') + call addfld ('DTCOND_24_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. sin coeff.') + call addfld ('DTCOND_12_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. cos coeff.') + call addfld ('DTCOND_12_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. sin coeff.') + + ! determine number of constituents for which convective tendencies must be computed + if (history_budget) then + dqcond_num = pcnst + else + if (diag_cnst_conv_tend == 'none') dqcond_num = 0 + if (diag_cnst_conv_tend == 'q_only') dqcond_num = 1 + if (diag_cnst_conv_tend == 'all') dqcond_num = pcnst + end if + + do m = 1, dqcond_num + dcconnam(m) = 'DC'//cnst_name(m) + end do + + if (diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' .or. history_budget) then + call addfld (dcconnam(1),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(1))//' tendency due to moist processes') + if ( diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' ) then + call add_default (dcconnam(1), 1, ' ') + end if + if( history_budget ) then + call add_default (dcconnam(1), history_budget_histfile_num, ' ') + end if + if (diag_cnst_conv_tend == 'all' .or. history_budget) then + do m = 2, pcnst + call addfld (dcconnam(m),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(m))//' tendency due to moist processes') + if( diag_cnst_conv_tend == 'all' ) then + call add_default (dcconnam(m), 1, ' ') + end if + if( history_budget .and. (m == ixcldliq .or. m == ixcldice) ) then + call add_default (dcconnam(m), history_budget_histfile_num, ' ') + end if + end do + end if + end if + + call addfld ('PRECL',horiz_only, 'A','m/s','Large-scale (stable) precipitation rate (liq + ice)' ) + call addfld ('PRECC',horiz_only, 'A','m/s','Convective precipitation rate (liq + ice)' ) + call addfld ('PRECT',horiz_only, 'A','m/s','Total (convective and large-scale) precipitation rate (liq + ice)' ) + call addfld ('PREC_PCW',horiz_only, 'A','m/s','LS_pcw precipitation rate') + call addfld ('PREC_zmc',horiz_only, 'A','m/s','CV_zmc precipitation rate') + call addfld ('PRECTMX',horiz_only, 'X','m/s','Maximum (convective and large-scale) precipitation rate (liq+ice)' ) + call addfld ('PRECSL',horiz_only, 'A','m/s','Large-scale (stable) snow rate (water equivalent)' ) + call addfld ('PRECSC',horiz_only, 'A','m/s','Convective snow rate (water equivalent)' ) + call addfld ('PRECCav',horiz_only, 'A','m/s','Average large-scale precipitation (liq + ice)' ) + call addfld ('PRECLav',horiz_only, 'A','m/s','Average convective precipitation (liq + ice)' ) + + ! outfld calls in diag_surf + + call addfld ('SHFLX',horiz_only, 'A','W/m2','Surface sensible heat flux') + call addfld ('LHFLX',horiz_only, 'A','W/m2','Surface latent heat flux') + call addfld ('QFLX',horiz_only, 'A','kg/m2/s','Surface water flux') + + call addfld ('TAUX',horiz_only, 'A','N/m2','Zonal surface stress') + call addfld ('TAUY',horiz_only, 'A','N/m2','Meridional surface stress') + call addfld ('TREFHT',horiz_only, 'A','K','Reference height temperature') + call addfld ('TREFHTMN',horiz_only, 'M','K','Minimum reference height temperature over output period') + call addfld ('TREFHTMX',horiz_only, 'X','K','Maximum reference height temperature over output period') + call addfld ('QREFHT',horiz_only, 'A','kg/kg','Reference height humidity') + call addfld ('U10',horiz_only, 'A','m/s','10m wind speed') + call addfld ('RHREFHT',horiz_only, 'A','fraction','Reference height relative humidity') + + call addfld ('LANDFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by land') + call addfld ('ICEFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('OCNFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by ocean') + + call addfld ('TREFMNAV',horiz_only, 'A','K','Average of TREFHT daily minimum') + call addfld ('TREFMXAV',horiz_only, 'A','K','Average of TREFHT daily maximum') + + call addfld ('TS',horiz_only, 'A','K','Surface temperature (radiative)') + call addfld ('TSMN',horiz_only, 'M','K','Minimum surface temperature over output period') + call addfld ('TSMX',horiz_only, 'X','K','Maximum surface temperature over output period') + call addfld ('SNOWHLND',horiz_only, 'A','m','Water equivalent snow depth') + call addfld ('SNOWHICE',horiz_only, 'A','m','Snow depth over ice', fill_value = 1.e30_r8) + call addfld ('TBOT',horiz_only, 'A','K','Lowest model level temperature') + + call addfld ('ASDIR', horiz_only, 'A', '1','albedo: shortwave, direct') + call addfld ('ASDIF', horiz_only, 'A', '1','albedo: shortwave, diffuse') + call addfld ('ALDIR', horiz_only, 'A', '1','albedo: longwave, direct') + call addfld ('ALDIF', horiz_only, 'A', '1','albedo: longwave, diffuse') + call addfld ('SST', horiz_only, 'A', 'K','sea surface temperature') + + ! defaults + if (history_amwg) then + call add_default ('DTCOND ', 1, ' ') + call add_default ('PRECL ', 1, ' ') + call add_default ('PRECC ', 1, ' ') + call add_default ('PRECSL ', 1, ' ') + call add_default ('PRECSC ', 1, ' ') + call add_default ('SHFLX ', 1, ' ') + call add_default ('LHFLX ', 1, ' ') + call add_default ('QFLX ', 1, ' ') + call add_default ('TAUX ', 1, ' ') + call add_default ('TAUY ', 1, ' ') + call add_default ('TREFHT ', 1, ' ') + call add_default ('LANDFRAC', 1, ' ') + call add_default ('OCNFRAC ', 1, ' ') + call add_default ('QREFHT ', 1, ' ') + call add_default ('U10 ', 1, ' ') + call add_default ('ICEFRAC ', 1, ' ') + call add_default ('TS ', 1, ' ') + call add_default ('TSMN ', 1, ' ') + call add_default ('TSMX ', 1, ' ') + call add_default ('SNOWHLND', 1, ' ') + call add_default ('SNOWHICE', 1, ' ') + endif + + call add_default ('OMEGA500', 1, ' ') + call add_default ('TH7001000', 1, ' ') + + if (history_vdiag) then + call add_default ('PRECT ', 2, ' ') + call add_default ('PRECT ', 3, ' ') + call add_default ('PRECT ', 4, ' ') + end if + + ! outfld calls in diag_phys_tend_writeout + + call addfld ('PTTEND' ,(/ 'lev' /), 'A','K/s','T total physics tendency' ) + call addfld (ptendnam( 1),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name( 1))//' total physics tendency ' ) + call addfld (ptendnam(ixcldliq),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldliq))//' total physics tendency ' ) + call addfld (ptendnam(ixcldice),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldice))//' total physics tendency ' ) + if ( dycore_is('LR') )then + call addfld (dmetendnam( 1),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name( 1))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldliq),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldliq))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldice),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldice))//' dme adjustment tendency (FV) ') + end if + + if ( history_budget ) then + call add_default ('PTTEND' , history_budget_histfile_num, ' ') + call add_default (ptendnam( 1), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldice), history_budget_histfile_num, ' ') + if ( dycore_is('LR') )then + call add_default(dmetendnam(1) , history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldice), history_budget_histfile_num, ' ') + end if + if( history_budget_histfile_num > 1 ) then + call add_default ('DTCOND ' , history_budget_histfile_num, ' ') + end if + end if + + ! outfld calls in diag_physvar_ic + + call addfld ('QCWAT&IC',(/ 'lev' /), 'I','kg/kg','q associated with cloud water' ) + call addfld ('TCWAT&IC',(/ 'lev' /), 'I','kg/kg','T associated with cloud water' ) + call addfld ('LCWAT&IC',(/ 'lev' /), 'I','kg/kg','Cloud water (ice + liq' ) + call addfld ('CLOUD&IC',(/ 'lev' /), 'I','fraction','Cloud fraction' ) + call addfld ('CONCLD&IC',(/ 'lev' /), 'I','fraction','Convective cloud fraction' ) + call addfld ('TKE&IC',(/ 'ilev' /),'I','m2/s2','Turbulent Kinetic Energy' ) + call addfld ('CUSH&IC',horiz_only, 'I','m','Convective Scale Height' ) + call addfld ('KVH&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (heat/moisture)') + call addfld ('KVM&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (momentum)' ) + call addfld ('PBLH&IC',horiz_only, 'I','m','PBL height' ) + call addfld ('TPERT&IC',horiz_only, 'I','K','Perturbation temperature (eddies in PBL)' ) + call addfld ('QPERT&IC',horiz_only, 'I','kg/kg','Perturbation specific humidity (eddies in PBL)' ) + call addfld ('TBOT&IC',horiz_only, 'I','K','Lowest model level temperature' ) + + + ! Initial file - Optional fields + + if (inithist_all) then + call add_default ('CONCLD&IC ',0, 'I') + call add_default ('QCWAT&IC ',0, 'I') + call add_default ('TCWAT&IC ',0, 'I') + call add_default ('LCWAT&IC ',0, 'I') + call add_default ('PBLH&IC ',0, 'I') + call add_default ('TPERT&IC ',0, 'I') + call add_default ('QPERT&IC ',0, 'I') + call add_default ('CLOUD&IC ',0, 'I') + call add_default ('TKE&IC ',0, 'I') + call add_default ('CUSH&IC ',0, 'I') + call add_default ('KVH&IC ',0, 'I') + call add_default ('KVM&IC ',0, 'I') + call add_default ('TBOT&IC ',0, 'I') + end if + + ! CAM export state + call addfld('a2x_BCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic black carbon') + call addfld('a2x_BCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic black carbon') + call addfld('a2x_BCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic black carbon') + call addfld('a2x_OCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic organic carbon') + call addfld('a2x_OCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic organic carbon') + call addfld('a2x_OCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic organic carbon') + call addfld('a2x_DSTWET1', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin1)') + call addfld('a2x_DSTDRY1', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin1)') + call addfld('a2x_DSTWET2', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin2)') + call addfld('a2x_DSTDRY2', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin2)') + call addfld('a2x_DSTWET3', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin3)') + call addfld('a2x_DSTDRY3', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin3)') + call addfld('a2x_DSTWET4', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin4)') + call addfld('a2x_DSTDRY4', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin4)') + + !--------------------------------------------------------- + ! CAM history fields for CAM-DOM/CAM-CSIM + !--------------------------------------------------------- + + ! CAM-DOM history fields +#ifdef COUP_DOM + call addfld ('TSOCN&IC',horiz_only, 'I','m','Ocean tempertare') + call add_default ('TSOCN&IC ',0, 'I') +#endif + + ! CAM-CSIM history fields + + do k=1,plevmx + call addfld (tsnam(k),horiz_only,'A','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK' ,horiz_only,'A','m','Sea ice thickness') + call addfld ('TSICE' ,horiz_only,'A','K','Ice temperature') + do k = 1,plevmx + call addfld (trim(tsnam(k))//'&IC',horiz_only,'I','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK&IC',horiz_only,'I','m','Sea ice thickness' ) + call addfld ('TSICE&IC',horiz_only,'I','K','Ice temperature' ) + call addfld ('SNOWHICE&IC',horiz_only,'I','m','Water equivalent snow depth' ) + call addfld ('ICEFRAC&IC',horiz_only,'I','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('TSICERAD&IC',horiz_only,'I','K','Radiatively equivalent ice temperature' ) + do k = 1,plevmx + call add_default(trim(tsnam(k))//'&IC',0, 'I') + end do + call add_default ('SICTHK&IC ',0, 'I') + call add_default ('TSICE&IC ',0, 'I') + call add_default ('SNOWHICE&IC',0, 'I') + call add_default ('ICEFRAC&IC ',0, 'I') + if (inithist_all) then + call add_default ('TSICERAD&IC',0, 'I') + end if + + !--------------------------------------------------------- + ! WACCM diagnostic history fields + !--------------------------------------------------------- + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + + ! create history variables for fourier coefficients of the diurnal + ! and semidiurnal tide in T, U, V, and Z3 + + call tidal_diag_init() + + endif + + qcwat_idx = pbuf_get_index('QCWAT',ierr) + tcwat_idx = pbuf_get_index('TCWAT',ierr) + lcwat_idx = pbuf_get_index('LCWAT',ierr) + cld_idx = pbuf_get_index('CLD') + concld_idx = pbuf_get_index('CONCLD') + + tke_idx = pbuf_get_index('tke') + kvm_idx = pbuf_get_index('kvm') + kvh_idx = pbuf_get_index('kvh') + cush_idx = pbuf_get_index('cush') + + pblh_idx = pbuf_get_index('pblh') + tpert_idx = pbuf_get_index('tpert') + qpert_idx = pbuf_get_index('qpert',ierr) + + prec_dp_idx = pbuf_get_index('PREC_DP') + snow_dp_idx = pbuf_get_index('SNOW_DP') + prec_sh_idx = pbuf_get_index('PREC_SH') + snow_sh_idx = pbuf_get_index('SNOW_SH') + prec_sed_idx = pbuf_get_index('PREC_SED') + snow_sed_idx = pbuf_get_index('SNOW_SED') + prec_pcw_idx = pbuf_get_index('PREC_PCW') + snow_pcw_idx = pbuf_get_index('SNOW_PCW') + +end subroutine diag_init + +!=============================================================================== + +subroutine diag_allocate() + use infnan, only: nan, assignment(=) + + ! Allocate memory for module variables. + ! Done at the begining of a physics step at same point as the pbuf allocate for + ! variables with "physpkg" scope. + + ! Local variables + character(len=*), parameter :: sub = 'diag_allocate' + integer :: i, istat + + allocate(dtcond(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dtcond = nan + + if (dqcond_num > 0) then + allocate(dqcond(dqcond_num)) + do i = 1, dqcond_num + allocate(dqcond(i)%cnst(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dqcond(i)%cnst = nan + end do + end if + +end subroutine diag_allocate + +!=============================================================================== + +subroutine diag_deallocate() + +! Deallocate memory for module variables. +! Done at the end of a physics step at same point as the pbuf deallocate for +! variables with "physpkg" scope. + +! Local variables + character(len=*), parameter :: sub = 'diag_deallocate' + integer :: i, istat + + deallocate(dtcond, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + + if (dqcond_num > 0) then + do i = 1, dqcond_num + deallocate(dqcond(i)%cnst, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + end do + deallocate(dqcond, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + end if + +end subroutine diag_deallocate +!=============================================================================== + +subroutine diag_conv_tend_ini(state,pbuf) + +! Initialize convective tendency calcs. + + +! Argument: + + type(physics_state), intent(in) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) + +! Local variables: + + integer :: i, k, m, lchnk, ncol + real(r8), pointer, dimension(:,:) :: t_ttend + + lchnk = state%lchnk + ncol = state%ncol + + do k = 1, pver + do i = 1, ncol + dtcond(i,k,lchnk) = state%s(i,k) + end do + end do + + do m = 1, dqcond_num + do k = 1, pver + do i = 1, ncol + dqcond(m)%cnst(i,k,lchnk) = state%q(i,k,m) + end do + end do + end do + + !! initialize to pbuf T_TTEND to temperature at first timestep + if (is_first_step()) then + do m = 1, dyn_time_lvls + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,m/), kount=(/pcols,pver,1/)) + t_ttend(:ncol,:) = state%t(:ncol,:) + end do + end if + +end subroutine diag_conv_tend_ini +!=============================================================================== + + subroutine diag_phys_writeout(state, psl) + +!----------------------------------------------------------------------- +! +! Purpose: record dynamics variables on physics grid +! +!----------------------------------------------------------------------- + use physconst, only: gravit, rga, rair, cpair, latvap, rearth, pi, cappa + use time_manager, only: get_nstep + use interpolate_data, only: vertinterp + use constituent_burden, only: constituent_burden_comp + use cam_control_mod, only: moist_physics + use co2_cycle, only: c_i, co2_transport + + use tidal_diag, only: tidal_diag_write +!----------------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(inout) :: state + real(r8), optional , intent(out) :: psl(pcols) +! +!---------------------------Local workspace----------------------------- +! + real(r8) ftem(pcols,pver) ! temporary workspace + real(r8) ftem1(pcols,pver) ! another temporary workspace + real(r8) ftem2(pcols,pver) ! another temporary workspace + real(r8) psl_tmp(pcols) ! Sea Level Pressure + real(r8) z3(pcols,pver) ! geo-potential height + real(r8) p_surf(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t2(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q2(pcols) ! data interpolated to a pressure surface + real(r8) tem2(pcols,pver) ! temporary workspace + real(r8) timestep(pcols) ! used for outfld call + real(r8) esl(pcols,pver) ! saturation vapor pressures + real(r8) esi(pcols,pver) ! + real(r8) dlon(pcols) ! width of grid cell (meters) + integer plon ! number of longitudes + + integer i, k, m, lchnk, ncol, nstep +! +!----------------------------------------------------------------------- +! + lchnk = state%lchnk + ncol = state%ncol + + ! Output NSTEP for debugging + nstep = get_nstep() + timestep(:ncol) = nstep + call outfld ('NSTEP ',timestep, pcols, lchnk) + + call outfld('T ',state%t , pcols ,lchnk ) + call outfld('PS ',state%ps, pcols ,lchnk ) + call outfld('U ',state%u , pcols ,lchnk ) + call outfld('V ',state%v , pcols ,lchnk ) + do m=1,pcnst + if ( cnst_cam_outfld(m) ) then + call outfld(cnst_name(m),state%q(1,1,m),pcols ,lchnk ) + end if + end do + + if (co2_transport()) then + do m = 1,4 + call outfld(trim(cnst_name(c_i(m)))//'_BOT', state%q(1,pver,c_i(m)), pcols, lchnk) + end do + end if + + ! column burdens of all constituents except water vapor + call constituent_burden_comp(state) + + if ( moist_physics) then + call outfld('PDELDRY ',state%pdeldry, pcols, lchnk) + call outfld('PSDRY', state%psdry, pcols, lchnk) + end if + + call outfld('PHIS ',state%phis, pcols, lchnk ) + + + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('phis ',state%phis, pcols, lchnk ) +#endif + +! +! Add height of surface to midpoint height above surface +! + do k = 1, pver + z3(:ncol,k) = state%zm(:ncol,k) + state%phis(:ncol)*rga + end do + call outfld('Z3 ',z3,pcols,lchnk) +! +! Output Z3 on pressure surfaces +! + if (hist_fld_active('Z1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, z3, p_surf) + call outfld('Z1000 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, z3, p_surf) + call outfld('Z975 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, z3, p_surf) + call outfld('Z950 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, z3, p_surf) + call outfld('Z925 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, z3, p_surf) + call outfld('Z900 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, z3, p_surf) + call outfld('Z850 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, z3, p_surf) + call outfld('Z800 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, z3, p_surf) + call outfld('Z700 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, z3, p_surf) + call outfld('Z600 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, z3, p_surf) + call outfld('Z500 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, z3, p_surf) + call outfld('Z400 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, z3, p_surf) + call outfld('Z300 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, z3, p_surf) + call outfld('Z200 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, z3, p_surf) + call outfld('Z100 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, z3, p_surf) + call outfld('Z050 ', p_surf, pcols, lchnk) + end if +! +! Quadratic height fiels Z3*Z3 +! + ftem(:ncol,:) = z3(:ncol,:)*z3(:ncol,:) + call outfld('ZZ ',ftem,pcols,lchnk) + + ftem(:ncol,:) = z3(:ncol,:)*state%v(:ncol,:)*gravit + call outfld('VZ ',ftem, pcols,lchnk) +! +! Meridional advection fields +! + ftem(:ncol,:) = state%v(:ncol,:)*state%t(:ncol,:) + call outfld ('VT ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1) + call outfld ('VQ ',ftem ,pcols ,lchnk ) + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,14) + call outfld ('Vbc_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,15) + call outfld ('Vdst_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,22) + call outfld ('Vdst_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,16) + call outfld ('Vncl_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,20) + call outfld ('Vncl_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,23) + call outfld ('Vncl_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,11) + call outfld ('Vso4_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,18) + call outfld ('Vso4_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,24) + call outfld ('Vso4_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,13) + call outfld ('Vsoa_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,19) + call outfld ('Vsoa_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,12) + call outfld ('Vpom_a1 ',ftem ,pcols ,lchnk ) + endif + + ftem(:ncol,:) = state%q(:ncol,:,1)*state%q(:ncol,:,1) + call outfld ('QQ ',ftem ,pcols ,lchnk ) + + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%q(:ncol,:,14)*state%q(:ncol,:,14) + call outfld ('bc_a1_2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,15)*state%q(:ncol,:,15) + call outfld ('dst_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,22)*state%q(:ncol,:,22) + call outfld ('dst_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,16)*state%q(:ncol,:,16) + call outfld ('ncl_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,20)*state%q(:ncol,:,20) + call outfld ('ncl_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,23)*state%q(:ncol,:,23) + call outfld ('ncl_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,11)*state%q(:ncol,:,11) + call outfld ('so4_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,18)*state%q(:ncol,:,18) + call outfld ('so4_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,24)*state%q(:ncol,:,24) + call outfld ('so4_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,13)*state%q(:ncol,:,13) + call outfld ('soa_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,19)*state%q(:ncol,:,19) + call outfld ('soa_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,12)*state%q(:ncol,:,12) + call outfld ('pom_a1_2',ftem ,pcols ,lchnk ) + endif + + ftem(:ncol,:) = state%v(:ncol,:)**2 + call outfld ('VV ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:) * state%u(:ncol,:) + call outfld ('VU ',ftem ,pcols ,lchnk ) + +! zonal advection + + ftem(:ncol,:) = state%u(:ncol,:)**2 + call outfld ('UU ',ftem ,pcols ,lchnk ) + +! Wind speed + ftem(:ncol,:) = sqrt( state%u(:ncol,:)**2 + state%v(:ncol,:)**2) + call outfld ('WSPEED ',ftem ,pcols ,lchnk ) + call outfld ('WSPDSRFMX',ftem(:,pver) ,pcols ,lchnk ) + call outfld ('WSPDSRFAV',ftem(:,pver) ,pcols ,lchnk ) + +! Vertical velocity and advection + + if (single_column) then + call outfld('OMEGA ',wfld, pcols, lchnk ) + else + call outfld('OMEGA ',state%omega, pcols, lchnk ) + endif + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('omega ',state%omega, pcols, lchnk ) +#endif + + ftem(:ncol,:) = state%omega(:ncol,:)*state%t(:ncol,:) + call outfld('OMEGAT ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%u(:ncol,:) + call outfld('OMEGAU ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%v(:ncol,:) + call outfld('OMEGAV ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%q(:ncol,:,1) + call outfld('OMEGAQ ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%omega(:ncol,:) + call outfld('OMGAOMGA',ftem, pcols, lchnk ) +! +! Output omega at 850 and 500 mb pressure levels +! + if (hist_fld_active('OMEGA1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%omega, p_surf) + call outfld('OMEGA1000', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%omega, p_surf) + call outfld('OMEGA975', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%omega, p_surf) + call outfld('OMEGA950', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%omega, p_surf) + call outfld('OMEGA925', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%omega, p_surf) + call outfld('OMEGA900', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%omega, p_surf) + call outfld('OMEGA850', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%omega, p_surf) + call outfld('OMEGA800', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%omega, p_surf) + call outfld('OMEGA700', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%omega, p_surf) + call outfld('OMEGA600', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%omega, p_surf) + call outfld('OMEGA500', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%omega, p_surf) + call outfld('OMEGA400', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%omega, p_surf) + call outfld('OMEGA300', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%omega, p_surf) + call outfld('OMEGA200', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%omega, p_surf) + call outfld('OMEGA100', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGABOT')) then + call outfld('OMEGABOT', state%omega(:,pver), pcols, lchnk) + end if +! +! Mass of q, by layer and vertically integrated +! + ftem(:ncol,:) = state%q(:ncol,:,1) * state%pdel(:ncol,:) * rga + call outfld ('MQ ',ftem ,pcols ,lchnk ) + + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TMQ ',ftem, pcols ,lchnk ) +! +! Mass of vertically integrated q flux +! + ftem(:ncol,:) = state%u(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TUQ ',ftem, pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TVQ ',ftem, pcols ,lchnk ) + +! +! Mass of vertically integrated MSE flux +! + ftem(:ncol,:) = state%u(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TUH ',ftem, pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TVH ',ftem, pcols ,lchnk ) + + if (moist_physics) then + + ! Relative humidity + if (hist_fld_active('RELHUM')) then + call qsat(state%t(:ncol,:), state%pmid(:ncol,:), & + tem2(:ncol,:), ftem(:ncol,:)) + ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RELHUM ',ftem ,pcols ,lchnk ) + end if + + ! ftem defined above assuming RELHUM in one of the history tape + if (hist_fld_active('RH1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, ftem, p_surf) + call outfld('RH1000', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, ftem, p_surf) + call outfld('RH975', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, ftem, p_surf) + call outfld('RH950', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, ftem, p_surf) + call outfld('RH925', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, ftem, p_surf) + call outfld('RH900', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, ftem, p_surf) + call outfld('RH850', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, ftem, p_surf) + call outfld('RH800', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, ftem, p_surf) + call outfld('RH700', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, ftem, p_surf) + call outfld('RH600', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, ftem, p_surf) + call outfld('RH500', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, ftem, p_surf) + call outfld('RH400', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, ftem, p_surf) + call outfld('RH300', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, ftem, p_surf) + call outfld('RH200', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, ftem, p_surf) + call outfld('RH100', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RHBOT')) then + call outfld('RHBOT', ftem(:,pver), pcols, lchnk) + end if + + if (hist_fld_active('RHW') .or. hist_fld_active('RHI') .or. hist_fld_active('RHCFMIP') ) then + + ! RH w.r.t liquid (water) + call qsat_water (state%t(:ncol,:), state%pmid(:ncol,:), & + esl(:ncol,:), ftem(:ncol,:)) + ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RHW ',ftem ,pcols ,lchnk ) + + ! Convert to RHI (ice) + do i=1,ncol + do k=1,pver + esi(i,k)=svp_ice(state%t(i,k)) + ftem1(i,k)=ftem(i,k)*esl(i,k)/esi(i,k) + end do + end do + call outfld ('RHI ',ftem1 ,pcols ,lchnk ) + + ! use temperature to decide if you populate with ftem (liquid, above 0 C) or ftem1 (ice, below 0 C) + + ftem2(:ncol,:)=ftem(:ncol,:) + + do i=1,ncol + do k=1,pver + if (state%t(i,k) .gt. 273) then + ftem2(i,k)=ftem(i,k) !!wrt water + else + ftem2(i,k)=ftem1(i,k) !!wrt ice + end if + end do + end do + + call outfld ('RHCFMIP ',ftem2 ,pcols ,lchnk ) + + end if + + end if +! +! Sea level pressure +! + if (present(psl) .or. hist_fld_active('PSL')) then + call cpslec (ncol, state%pmid, state%phis, state%ps, state%t,psl_tmp, gravit, rair) + call outfld ('PSL ',psl_tmp ,pcols, lchnk ) + if (present(psl)) then + psl(:ncol) = psl_tmp(:ncol) + end if + end if +! +! Output T,q,u,v fields on pressure surfaces +! + if (hist_fld_active('T975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%t, p_surf) + call outfld('T975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%t, p_surf) + call outfld('T950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%t, p_surf) + call outfld('T900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf) + call outfld('T850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%t, p_surf) + call outfld('T800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%t, p_surf) + call outfld('T600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%t, p_surf) + call outfld('T500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%t, p_surf) + call outfld('T400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%t, p_surf) + call outfld('T300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%t, p_surf) + call outfld('T200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%t, p_surf) + call outfld('T100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%q(1,1,1), p_surf) + call outfld('Q975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%q(1,1,1), p_surf) + call outfld('Q950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%q(1,1,1), p_surf) + call outfld('Q900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf) + call outfld('Q850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%q(1,1,1), p_surf) + call outfld('Q800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf) + call outfld('Q700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%q(1,1,1), p_surf) + call outfld('Q600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%q(1,1,1), p_surf) + call outfld('Q500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%q(1,1,1), p_surf) + call outfld('Q400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%q(1,1,1), p_surf) + call outfld('Q300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%q(1,1,1), p_surf) + call outfld('Q200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%q(1,1,1), p_surf) + call outfld('Q100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%u, p_surf) + call outfld('U1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%u, p_surf) + call outfld('U975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%u, p_surf) + call outfld('U950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%u, p_surf) + call outfld('U925 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%u, p_surf) + call outfld('U900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%u, p_surf) + call outfld('U850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%u, p_surf) + call outfld('U800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%u, p_surf) + call outfld('U700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%u, p_surf) + call outfld('U600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%u, p_surf) + call outfld('U500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%u, p_surf) + call outfld('U400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%u, p_surf) + call outfld('U300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%u, p_surf) + call outfld('U250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%u, p_surf) + call outfld('U200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%u, p_surf) + call outfld('U100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%u, p_surf) + call outfld('U010 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%v, p_surf) + call outfld('V1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%v, p_surf) + call outfld('V975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%v, p_surf) + call outfld('V950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%v, p_surf) + call outfld('V925 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%v, p_surf) + call outfld('V900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%v, p_surf) + call outfld('V850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%v, p_surf) + call outfld('V800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%v, p_surf) + call outfld('V700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%v, p_surf) + call outfld('V600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%v, p_surf) + call outfld('V500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%v, p_surf) + call outfld('V400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%v, p_surf) + call outfld('V300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%v, p_surf) + call outfld('V250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%v, p_surf) + call outfld('V200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%v, p_surf) + call outfld('V100 ', p_surf, pcols, lchnk ) + end if + + ftem(:ncol,:) = state%t(:ncol,:)*state%t(:ncol,:) + call outfld('TT ',ftem ,pcols ,lchnk ) +! +! Output U, V, T, Q, P and Z at bottom level +! + call outfld ('UBOT ', state%u(1,pver) , pcols, lchnk) + call outfld ('VBOT ', state%v(1,pver) , pcols, lchnk) + call outfld ('QBOT ', state%q(1,pver,1), pcols, lchnk) + call outfld ('ZBOT ', state%zm(1,pver) , pcols, lchnk) + +! Total energy of the atmospheric column for atmospheric heat storage calculations + + !! temporary variable to get surface geopotential in dimensions of (ncol,pver) + do k=1,pver + ftem1(:ncol,k)=state%phis(:ncol) !! surface geopotential in units (m2/s2) + end do + + !! calculate sum of sensible, kinetic, latent, and surface geopotential energy + !! E=CpT+PHIS+Lv*q+(0.5)*(u^2+v^2) + ftem(:ncol,:) = (cpair*state%t(:ncol,:) + ftem1(:ncol,:) + latvap*state%q(:ncol,:,1) + & + 0.5_r8*(state%u(:ncol,:)**2+state%v(:ncol,:)**2))*(state%pdel(:ncol,:)/gravit) + !! vertically integrate + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('ATMEINT ',ftem(:ncol,1) ,pcols ,lchnk ) + +!! Boundary layer atmospheric stability, temperature, water vapor diagnostics + + if (hist_fld_active('T1000') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%t, p_surf_t1) + end if + + if (hist_fld_active('T925') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%t, p_surf_t2) + end if + + if (hist_fld_active('Q1000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%q(1,1,1), p_surf_q1) + end if + + if (hist_fld_active('Q925') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%q(1,1,1), p_surf_q2) + end if + + !!! at 1000 mb and 925 mb + if (hist_fld_active('T1000')) then + call outfld('T1000 ', p_surf_t1, pcols, lchnk ) + end if + + if (hist_fld_active('T925')) then + call outfld('T925 ', p_surf_t2, pcols, lchnk ) + end if + + if (hist_fld_active('Q1000')) then + call outfld('Q1000 ', p_surf_q1, pcols, lchnk ) + end if + + if (hist_fld_active('Q925')) then + call outfld('Q925 ', p_surf_q2, pcols, lchnk ) + end if + + if (hist_fld_active('T9251000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf_t2) + end if + + !!! at 1000 mb and 850 mb + if (hist_fld_active('T8501000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH8501000')) then + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('T700') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%t, p_surf_t2) + end if + + !!! at 700 mb + if (hist_fld_active('T700')) then + call outfld('T700 ', p_surf_t2, pcols, lchnk ) + end if + + !!! at 1000 mb and 700 mb + if (hist_fld_active('T7001000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH7001000')) then + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%t, p_surf) + call outfld('T010 ', p_surf, pcols, lchnk ) + end if + + + !--------------------------------------------------------- + ! WACCM tidal diagnostics + !--------------------------------------------------------- + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + + call tidal_diag_write(state) + + endif + + return + end subroutine diag_phys_writeout +!=============================================================================== + +subroutine diag_conv(state, ztodt, pbuf) + +!----------------------------------------------------------------------- +! +! Output diagnostics associated with all convective processes. +! +!----------------------------------------------------------------------- + use physconst, only: cpair + use tidal_diag, only: get_tidal_coeffs + +! Arguments: + + real(r8), intent(in) :: ztodt ! timestep for computing physics tendencies + type(physics_state), intent(in) :: state + type(physics_buffer_desc), pointer :: pbuf(:) + +! convective precipitation variables + real(r8), pointer :: prec_dp(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_dp(:) ! snow from ZM convection + real(r8), pointer :: prec_sh(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_sh(:) ! snow from Hack convection + real(r8), pointer :: prec_sed(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_sed(:) ! snow from ZM convection + real(r8), pointer :: prec_pcw(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_pcw(:) ! snow from Hack convection + +! Local variables: + + integer :: i, k, m, lchnk, ncol + + real(r8) :: rtdt + + real(r8):: precc(pcols) ! convective precip rate + real(r8):: precl(pcols) ! stratiform precip rate + real(r8):: snowc(pcols) ! convective snow rate + real(r8):: snowl(pcols) ! stratiform snow rate + real(r8):: prect(pcols) ! total (conv+large scale) precip rate + real(r8) :: dcoef(4) ! for tidal component of T tend + + lchnk = state%lchnk + ncol = state%ncol + + rtdt = 1._r8/ztodt + + call pbuf_get_field(pbuf, prec_dp_idx, prec_dp) + call pbuf_get_field(pbuf, snow_dp_idx, snow_dp) + call pbuf_get_field(pbuf, prec_sh_idx, prec_sh) + call pbuf_get_field(pbuf, snow_sh_idx, snow_sh) + call pbuf_get_field(pbuf, prec_sed_idx, prec_sed) + call pbuf_get_field(pbuf, snow_sed_idx, snow_sed) + call pbuf_get_field(pbuf, prec_pcw_idx, prec_pcw) + call pbuf_get_field(pbuf, snow_pcw_idx, snow_pcw) + +! Precipitation rates (multi-process) + precc(:ncol) = prec_dp(:ncol) + prec_sh(:ncol) + precl(:ncol) = prec_sed(:ncol) + prec_pcw(:ncol) + snowc(:ncol) = snow_dp(:ncol) + snow_sh(:ncol) + snowl(:ncol) = snow_sed(:ncol) + snow_pcw(:ncol) + prect(:ncol) = precc(:ncol) + precl(:ncol) + + call outfld('PRECC ', precc, pcols, lchnk ) + call outfld('PRECL ', precl, pcols, lchnk ) + call outfld('PREC_PCW', prec_pcw,pcols ,lchnk ) + call outfld('PREC_zmc', prec_dp ,pcols ,lchnk ) + call outfld('PRECSC ', snowc, pcols, lchnk ) + call outfld('PRECSL ', snowl, pcols, lchnk ) + call outfld('PRECT ', prect, pcols, lchnk ) + call outfld('PRECTMX ', prect, pcols, lchnk ) + + call outfld('PRECLav ', precl, pcols, lchnk ) + call outfld('PRECCav ', precc, pcols, lchnk ) + +#if ( defined BFB_CAM_SCAM_IOP ) + call outfld('Prec ' , prect, pcols, lchnk ) +#endif + + ! Total convection tendencies. + + do k = 1, pver + do i = 1, ncol + dtcond(i,k,lchnk) = (state%s(i,k) - dtcond(i,k,lchnk))*rtdt / cpair + end do + end do + call outfld('DTCOND ', dtcond(:,:,lchnk), pcols, lchnk) + + ! output tidal coefficients + call get_tidal_coeffs( dcoef ) + call outfld( 'DTCOND_24_SIN', dtcond(:ncol,:,lchnk)*dcoef(1), ncol, lchnk ) + call outfld( 'DTCOND_24_COS', dtcond(:ncol,:,lchnk)*dcoef(2), ncol, lchnk ) + call outfld( 'DTCOND_12_SIN', dtcond(:ncol,:,lchnk)*dcoef(3), ncol, lchnk ) + call outfld( 'DTCOND_12_COS', dtcond(:ncol,:,lchnk)*dcoef(4), ncol, lchnk ) + + do m = 1, dqcond_num + if ( cnst_cam_outfld(m) ) then + do k = 1, pver + do i = 1, ncol + dqcond(m)%cnst(i,k,lchnk) = (state%q(i,k,m) - dqcond(m)%cnst(i,k,lchnk))*rtdt + end do + end do + call outfld(dcconnam(m), dqcond(m)%cnst(:,:,lchnk), pcols, lchnk) + end if + end do + +end subroutine diag_conv + +!=============================================================================== + +subroutine diag_surf (cam_in, cam_out, ps, trefmxav, trefmnav ) + +!----------------------------------------------------------------------- +! +! Purpose: record surface diagnostics +! +!----------------------------------------------------------------------- + + use time_manager, only: is_end_curr_day + use co2_cycle, only: c_i, co2_transport + use constituents, only: sflxnam + +!----------------------------------------------------------------------- +! +! Input arguments +! + type(cam_in_t), intent(in) :: cam_in + type(cam_out_t), intent(in) :: cam_out + + real(r8), intent(inout) :: trefmnav(pcols) ! daily minimum tref + real(r8), intent(inout) :: trefmxav(pcols) ! daily maximum tref + + real(r8), intent(in) :: ps(pcols) ! Surface pressure. +! +!---------------------------Local workspace----------------------------- +! + integer :: i, k, m ! indexes + integer :: lchnk ! chunk identifier + integer :: ncol ! longitude dimension + real(r8) tem2(pcols) ! temporary workspace + real(r8) ftem(pcols) ! temporary workspace +! +!----------------------------------------------------------------------- +! + lchnk = cam_in%lchnk + ncol = cam_in%ncol + + call outfld('SHFLX', cam_in%shf, pcols, lchnk) + call outfld('LHFLX', cam_in%lhf, pcols, lchnk) + call outfld('QFLX', cam_in%cflx(1,1), pcols, lchnk) + + call outfld('TAUX', cam_in%wsx, pcols, lchnk) + call outfld('TAUY', cam_in%wsy, pcols, lchnk) + call outfld('TREFHT ', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMX', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMN', cam_in%tref, pcols, lchnk) + call outfld('QREFHT', cam_in%qref, pcols, lchnk) + call outfld('U10', cam_in%u10, pcols, lchnk) +! +! Calculate and output reference height RH (RHREFHT) + + call qsat(cam_in%tref(:ncol), ps(:ncol), tem2(:ncol), ftem(:ncol)) + ftem(:ncol) = cam_in%qref(:ncol)/ftem(:ncol)*100._r8 + + + call outfld('RHREFHT', ftem, pcols, lchnk) + + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('shflx ',cam_in%shf, pcols, lchnk) + call outfld('lhflx ',cam_in%lhf, pcols, lchnk) + call outfld('trefht ',cam_in%tref, pcols, lchnk) +#endif +! +! Ouput ocn and ice fractions +! + call outfld('LANDFRAC', cam_in%landfrac, pcols, lchnk) + call outfld('ICEFRAC', cam_in%icefrac, pcols, lchnk) + call outfld('OCNFRAC', cam_in%ocnfrac, pcols, lchnk) +! +! Compute daily minimum and maximum of TREF +! + do i = 1,ncol + trefmxav(i) = max(cam_in%tref(i),trefmxav(i)) + trefmnav(i) = min(cam_in%tref(i),trefmnav(i)) + end do + if (is_end_curr_day()) then + call outfld('TREFMXAV', trefmxav,pcols, lchnk ) + call outfld('TREFMNAV', trefmnav,pcols, lchnk ) + trefmxav(:ncol) = -1.0e36_r8 + trefmnav(:ncol) = 1.0e36_r8 + endif + + call outfld('TBOT', cam_out%tbot, pcols, lchnk) + call outfld('TS', cam_in%ts, pcols, lchnk) + call outfld('TSMN', cam_in%ts, pcols, lchnk) + call outfld('TSMX', cam_in%ts, pcols, lchnk) + call outfld('SNOWHLND', cam_in%snowhland, pcols, lchnk) + call outfld('SNOWHICE', cam_in%snowhice, pcols, lchnk) + call outfld('ASDIR', cam_in%asdir, pcols, lchnk) + call outfld('ASDIF', cam_in%asdif, pcols, lchnk) + call outfld('ALDIR', cam_in%aldir, pcols, lchnk) + call outfld('ALDIF', cam_in%aldif, pcols, lchnk) + call outfld('SST', cam_in%sst, pcols, lchnk) + + if (co2_transport()) then + do m = 1,4 + call outfld(sflxnam(c_i(m)), cam_in%cflx(:,c_i(m)), pcols, lchnk) + end do + end if + +end subroutine diag_surf + +!=============================================================================== + +subroutine diag_export(cam_out) + +!----------------------------------------------------------------------- +! +! Purpose: Write export state to history file +! +!----------------------------------------------------------------------- + + ! arguments + type(cam_out_t), intent(inout) :: cam_out + + ! Local variables: + integer :: lchnk ! chunk identifier + logical :: atm_dep_flux ! true ==> sending deposition fluxes to coupler. + ! Otherwise, set them to zero. + !----------------------------------------------------------------------- + + lchnk = cam_out%lchnk + + call phys_getopts(atm_dep_flux_out=atm_dep_flux) + + if (.not. atm_dep_flux) then + ! set the fluxes to zero before outfld and sending them to the + ! coupler + cam_out%bcphiwet = 0.0_r8 + cam_out%bcphidry = 0.0_r8 + cam_out%bcphodry = 0.0_r8 + cam_out%ocphiwet = 0.0_r8 + cam_out%ocphidry = 0.0_r8 + cam_out%ocphodry = 0.0_r8 + cam_out%dstwet1 = 0.0_r8 + cam_out%dstdry1 = 0.0_r8 + cam_out%dstwet2 = 0.0_r8 + cam_out%dstdry2 = 0.0_r8 + cam_out%dstwet3 = 0.0_r8 + cam_out%dstdry3 = 0.0_r8 + cam_out%dstwet4 = 0.0_r8 + cam_out%dstdry4 = 0.0_r8 + end if + + call outfld('a2x_BCPHIWET', cam_out%bcphiwet, pcols, lchnk) + call outfld('a2x_BCPHIDRY', cam_out%bcphidry, pcols, lchnk) + call outfld('a2x_BCPHODRY', cam_out%bcphodry, pcols, lchnk) + call outfld('a2x_OCPHIWET', cam_out%ocphiwet, pcols, lchnk) + call outfld('a2x_OCPHIDRY', cam_out%ocphidry, pcols, lchnk) + call outfld('a2x_OCPHODRY', cam_out%ocphodry, pcols, lchnk) + call outfld('a2x_DSTWET1', cam_out%dstwet1, pcols, lchnk) + call outfld('a2x_DSTDRY1', cam_out%dstdry1, pcols, lchnk) + call outfld('a2x_DSTWET2', cam_out%dstwet2, pcols, lchnk) + call outfld('a2x_DSTDRY2', cam_out%dstdry2, pcols, lchnk) + call outfld('a2x_DSTWET3', cam_out%dstwet3, pcols, lchnk) + call outfld('a2x_DSTDRY3', cam_out%dstdry3, pcols, lchnk) + call outfld('a2x_DSTWET4', cam_out%dstwet4, pcols, lchnk) + call outfld('a2x_DSTDRY4', cam_out%dstdry4, pcols, lchnk) + +end subroutine diag_export + +!####################################################################### + + subroutine diag_physvar_ic (lchnk, pbuf, cam_out, cam_in) +! +!--------------------------------------------- +! +! Purpose: record physics variables on IC file +! +!--------------------------------------------- +! + +! +! Arguments +! + integer , intent(in) :: lchnk ! chunk identifier + type(physics_buffer_desc), pointer :: pbuf(:) + + type(cam_out_t), intent(inout) :: cam_out + type(cam_in_t), intent(inout) :: cam_in +! +!---------------------------Local workspace----------------------------- +! + integer :: k ! indices + integer :: itim_old ! indices + + real(r8), pointer, dimension(:,:) :: cwat_var + real(r8), pointer, dimension(:,:) :: conv_var_3d + real(r8), pointer, dimension(: ) :: conv_var_2d + real(r8), pointer :: tpert(:), pblh(:), qpert(:) +! +!----------------------------------------------------------------------- +! + if( write_inithist() ) then + + ! + ! Associate pointers with physics buffer fields + ! + itim_old = pbuf_old_tim_idx() + + if (qcwat_idx > 0) then + call pbuf_get_field(pbuf, qcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('QCWAT&IC ',cwat_var, pcols,lchnk) + end if + + if (tcwat_idx > 0) then + call pbuf_get_field(pbuf, tcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('TCWAT&IC ',cwat_var, pcols,lchnk) + end if + + if (lcwat_idx > 0) then + call pbuf_get_field(pbuf, lcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('LCWAT&IC ',cwat_var, pcols,lchnk) + end if + + call pbuf_get_field(pbuf, cld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CLOUD&IC ',cwat_var, pcols,lchnk) + + call pbuf_get_field(pbuf, concld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CONCLD&IC ',cwat_var, pcols,lchnk) + + call pbuf_get_field(pbuf, tke_idx, conv_var_3d) + call outfld('TKE&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, kvm_idx, conv_var_3d) + call outfld('KVM&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, kvh_idx, conv_var_3d) + call outfld('KVH&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, cush_idx, conv_var_2d ,(/1,itim_old/), (/pcols,1/)) + call outfld('CUSH&IC ',conv_var_2d, pcols,lchnk) + + if (qpert_idx > 0) then + call pbuf_get_field(pbuf, qpert_idx, qpert) + call outfld('QPERT&IC ', qpert, pcols, lchnk) + end if + + call pbuf_get_field(pbuf, pblh_idx, pblh) + call outfld('PBLH&IC ', pblh, pcols, lchnk) + + call pbuf_get_field(pbuf, tpert_idx, tpert) + call outfld('TPERT&IC ', tpert, pcols, lchnk) + + ! The following is only needed for cam-csim + call outfld('TBOT&IC ', cam_out%tbot, pcols, lchnk) + end if + + end subroutine diag_physvar_ic + + +!####################################################################### + +subroutine diag_phys_tend_writeout(state, pbuf, tend, ztodt, tmp_q, tmp_cldliq, tmp_cldice, & + tmp_t, qini, cldliqini, cldiceini) + + !--------------------------------------------------------------- + ! + ! Purpose: Dump physics tendencies for moisture and temperature + ! + !--------------------------------------------------------------- + + use check_energy, only: check_energy_get_integrals + use physconst, only: cpair + + ! Arguments + + type(physics_state), intent(in ) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) + type(physics_tend ), intent(in ) :: tend + real(r8) , intent(in ) :: ztodt ! physics timestep + real(r8) , intent(inout) :: tmp_q (pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldliq(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldice(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_t (pcols,pver) ! holds last physics_updated T (FV) + real(r8) , intent(in ) :: qini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldliqini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldiceini (pcols,pver) ! tracer fields at beginning of physics + + !---------------------------Local workspace----------------------------- + + integer :: m ! constituent index + integer :: lchnk ! chunk index + integer :: ncol ! number of columns in chunk + real(r8) :: ftem2(pcols ) ! Temporary workspace for outfld variables + real(r8) :: ftem3(pcols,pver) ! Temporary workspace for outfld variables + real(r8) :: rtdt + real(r8) :: heat_glob ! global energy integral (FV only) + integer :: ixcldice, ixcldliq! constituent indices for cloud liquid and ice water. + ! CAM pointers to get variables from the physics buffer + real(r8), pointer, dimension(:,:) :: t_ttend + integer :: itim_old + + !----------------------------------------------------------------------- + + lchnk = state%lchnk + ncol = state%ncol + rtdt = 1._r8/ztodt + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + + ! Dump out post-physics state (FV only) + + if (dycore_is('LR') .or. dycore_is('SE')) then + tmp_t(:ncol,:pver) = (tmp_t(:ncol,:pver) - state%t(:ncol,:pver))/ztodt + call outfld('PTTEND_RESID', tmp_t, pcols, lchnk ) + end if + call outfld('TAP', state%t, pcols, lchnk ) + call outfld('UAP', state%u, pcols, lchnk ) + call outfld('VAP', state%v, pcols, lchnk ) + + if ( cnst_cam_outfld( 1) ) call outfld (apcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (apcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (apcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) + + ! T-tendency due to FV Energy fixer (remove from total physics tendency diagnostic) + + if (dycore_is('LR') .or. dycore_is('SE')) then + call check_energy_get_integrals( heat_glob_out=heat_glob ) + ftem2(:ncol) = heat_glob/cpair + call outfld('TFIX', ftem2, pcols, lchnk ) + ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) - heat_glob/cpair + else + ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) + end if + + ! Total physics tendency for Temperature + + call outfld('PTTEND',ftem3, pcols, lchnk ) + + ! Tendency for dry mass adjustment of q (valid for FV only) + + if (dycore_is('LR')) then + tmp_q (:ncol,:pver) = (state%q(:ncol,:pver, 1) - tmp_q (:ncol,:pver))*rtdt + tmp_cldliq(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - tmp_cldliq(:ncol,:pver))*rtdt + tmp_cldice(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - tmp_cldice(:ncol,:pver))*rtdt + if ( cnst_cam_outfld( 1) ) call outfld (dmetendnam( 1), tmp_q , pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (dmetendnam(ixcldliq), tmp_cldliq, pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (dmetendnam(ixcldice), tmp_cldice, pcols, lchnk) + end if + + ! Total physics tendency for moisture and other tracers + + if ( cnst_cam_outfld( 1) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver, 1) - qini (:ncol,:pver) )*rtdt + call outfld (ptendnam( 1), ftem3, pcols, lchnk) + end if + if ( cnst_cam_outfld(ixcldliq) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - cldliqini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldliq), ftem3, pcols, lchnk) + end if + if ( cnst_cam_outfld(ixcldice) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - cldiceini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldice), ftem3, pcols, lchnk) + end if + + ! Total (physics+dynamics, everything!) tendency for Temperature + + !! get temperature stored in physics buffer + itim_old = pbuf_old_tim_idx() + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,itim_old/), kount=(/pcols,pver,1/)) + + !! calculate and outfld the total temperature tendency + ftem3(:ncol,:) = (state%t(:ncol,:) - t_ttend(:ncol,:))/ztodt + call outfld('TTEND_TOT', ftem3, pcols, lchnk) + + !! update physics buffer with this time-step's temperature + t_ttend(:ncol,:) = state%t(:ncol,:) + +end subroutine diag_phys_tend_writeout + +!####################################################################### + + subroutine diag_state_b4_phys_write (state) +! +!--------------------------------------------------------------- +! +! Purpose: Dump state just prior to executing physics +! +!--------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(in) :: state +! +!---------------------------Local workspace----------------------------- +! + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: lchnk ! chunk index +! +!----------------------------------------------------------------------- +! + lchnk = state%lchnk + + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call outfld('TBP', state%t, pcols, lchnk ) + if ( cnst_cam_outfld( 1) ) call outfld (bpcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (bpcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (bpcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) + + end subroutine diag_state_b4_phys_write + +end module cam_diagnostics diff --git a/SourceMods/src.cam/safe/cam_diagnostics.F90 b/SourceMods/src.cam/safe/cam_diagnostics.F90 new file mode 100644 index 0000000..5b1e9b1 --- /dev/null +++ b/SourceMods/src.cam/safe/cam_diagnostics.F90 @@ -0,0 +1,2387 @@ +module cam_diagnostics + +!--------------------------------------------------------------------------------- +! Module to compute a variety of diagnostics quantities for history files +!--------------------------------------------------------------------------------- + +use shr_kind_mod, only: r8 => shr_kind_r8 +use camsrfexch, only: cam_in_t, cam_out_t +use physics_types, only: physics_state, physics_tend +use ppgrid, only: pcols, pver, pverp, begchunk, endchunk +use physics_buffer, only: physics_buffer_desc, pbuf_add_field, dtype_r8, dyn_time_lvls, & + pbuf_get_field, pbuf_get_index, pbuf_old_tim_idx + + + +use cam_history, only: outfld, write_inithist, hist_fld_active +use constituents, only: pcnst, cnst_name, cnst_longname, cnst_cam_outfld, ptendnam, dmetendnam, apcnst, bpcnst, & + cnst_get_ind +use chemistry, only: chem_is +use dycore, only: dycore_is +use phys_control, only: phys_getopts +use wv_saturation, only: qsat, qsat_water, svp_ice +use time_manager, only: is_first_step + +use scamMod, only: single_column, wfld +use cam_abortutils, only: endrun + +implicit none +private +save + +! Public interfaces + +public :: & + diag_register, &! register pbuf space + diag_init, &! initialization + diag_allocate, &! allocate memory for module variables + diag_deallocate, &! deallocate memory for module variables + diag_conv_tend_ini, &! initialize convective tendency calcs + diag_phys_writeout, &! output diagnostics of the dynamics + diag_phys_tend_writeout, & ! output physics tendencies + diag_state_b4_phys_write,& ! output state before physics execution + diag_conv, &! output diagnostics of convective processes + diag_surf, &! output diagnostics of the surface + diag_export, &! output export state + diag_physvar_ic, & + diag_readnl ! read namelist options + +logical, public :: inithist_all = .false. ! Flag to indicate set of fields to be + ! included on IC file + ! .false. include only required fields + ! .true. include required *and* optional fields + +! Private data + +integer :: dqcond_num ! number of constituents to compute convective +character(len=16) :: dcconnam(pcnst) ! names of convection tendencies + ! tendencies for +real(r8), allocatable :: dtcond(:,:,:) ! temperature tendency due to convection +type dqcond_t + real(r8), allocatable :: cnst(:,:,:) ! constituent tendency due to convection +end type dqcond_t +type(dqcond_t), allocatable :: dqcond(:) + +character(len=8) :: diag_cnst_conv_tend = 'q_only' ! output constituent tendencies due to convection + ! 'none', 'q_only' or 'all' + +logical :: history_amwg ! output the variables used by the AMWG diag package +logical :: history_verbose ! produce verbose history output +logical :: history_vdiag ! output the variables used by the AMWG variability diag package +logical :: history_eddy ! output the eddy variables +logical :: history_budget ! output tendencies and state variables for CAM4 + ! temperature, water vapor, cloud ice and cloud + ! liquid budgets. +integer :: history_budget_histfile_num ! output history file number for budget fields + +!Physics buffer indices +integer :: qcwat_idx = 0 +integer :: tcwat_idx = 0 +integer :: lcwat_idx = 0 +integer :: cld_idx = 0 +integer :: concld_idx = 0 +integer :: tke_idx = 0 +integer :: kvm_idx = 0 +integer :: kvh_idx = 0 +integer :: cush_idx = 0 +integer :: t_ttend_idx = 0 + +integer :: prec_dp_idx = 0 +integer :: snow_dp_idx = 0 +integer :: prec_sh_idx = 0 +integer :: snow_sh_idx = 0 +integer :: prec_sed_idx = 0 +integer :: snow_sed_idx = 0 +integer :: prec_pcw_idx = 0 +integer :: snow_pcw_idx = 0 + + +integer :: tpert_idx=-1, qpert_idx=-1, pblh_idx=-1 +logical :: prog_modal_aero +contains + +! =============================================================================== + +subroutine diag_register + + ! Request physics buffer space for fields that persist across timesteps. + call pbuf_add_field('T_TTEND', 'global', dtype_r8, (/pcols,pver,dyn_time_lvls/), t_ttend_idx) + +end subroutine diag_register + +!=============================================================================== + +subroutine diag_readnl(nlfile) + use namelist_utils, only: find_group_name + use units, only: getunit, freeunit + use mpishorthand + use spmd_utils, only: masterproc + + character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input + + ! Local variables + integer :: unitn, ierr + character(len=*), parameter :: subname = 'diag_readnl' + + namelist /cam_diag_opts/ diag_cnst_conv_tend + !----------------------------------------------------------------------------- + + if (masterproc) then + unitn = getunit() + open( unitn, file=trim(nlfile), status='old' ) + call find_group_name(unitn, 'cam_diag_opts', status=ierr) + if (ierr == 0) then + read(unitn, cam_diag_opts, iostat=ierr) + if (ierr /= 0) then + call endrun(subname // ':: ERROR reading namelist') + end if + end if + close(unitn) + call freeunit(unitn) + end if + +#ifdef SPMD + ! Broadcast namelist variables + call mpibcast(diag_cnst_conv_tend, len(diag_cnst_conv_tend), mpichar, 0, mpicom) +#endif + +end subroutine diag_readnl + +!================================================================================================ + +subroutine diag_init() + + ! Declare the history fields for which this module contains outfld calls. + + use cam_history, only: addfld, horiz_only, add_default + use constituent_burden, only: constituent_burden_init + use cam_control_mod, only: moist_physics, ideal_phys + use tidal_diag, only: tidal_diag_init + + integer :: k, m + ! Note - this is a duplication of information in ice_constants + ! Cannot put in a use statement if want to swap ice models to cice4 + integer, parameter :: plevmx = 4 ! number of subsurface levels + character(len=8), parameter :: tsnam(plevmx) = (/ 'TS1', 'TS2', 'TS3', 'TS4' /) + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: ierr + + call phys_getopts(prog_modal_aero_out = prog_modal_aero ) + + ! outfld calls in diag_phys_writeout + + call addfld ('NSTEP',horiz_only, 'A','timestep','Model timestep') + call addfld ('PHIS',horiz_only, 'I','m2/s2','Surface geopotential') + + call addfld ('PS',horiz_only, 'A','Pa','Surface pressure') + call addfld ('T',(/ 'lev' /), 'A','K','Temperature') + call addfld ('U',(/ 'lev' /), 'A','m/s','Zonal wind') + call addfld ('V',(/ 'lev' /), 'A','m/s','Meridional wind') + call addfld (cnst_name(1),(/ 'lev' /), 'A','kg/kg',cnst_longname(1)) + + ! State before physics + call addfld ('TBP',(/ 'lev' /), 'A','K','Temperature (before physics)' ) + call addfld (bpcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (before physics)') + ! State after physics + call addfld ('TAP',(/ 'lev' /), 'A','K','Temperature (after physics)' ) + call addfld ('UAP',(/ 'lev' /), 'A','m/s','Zonal wind (after physics)' ) + call addfld ('VAP',(/ 'lev' /), 'A','m/s','Meridional wind (after physics)' ) + call addfld (apcnst(1) ,(/ 'lev' /), 'A','kg/kg',cnst_longname(1)//' (after physics)') + if ( dycore_is('LR') .or. dycore_is('SE') ) then + call addfld ('TFIX',horiz_only, 'A' ,'K/s','T fixer (T equivalent of Energy correction)') + call addfld ('PTTEND_RESID',(/ 'lev' /), 'A' ,'K/s',& + 'T-tendency due to BAB kluge at end of tphysac (diagnostic not part of T-budget)' ) + end if + call addfld ('TTEND_TOT',(/ 'lev' /), 'A','K/s' ,'Total temperature tendency' ) + + ! column burdens for all constituents except water vapor + call constituent_burden_init + + call addfld ('Z3',(/ 'lev' /), 'A','m','Geopotential Height (above sea level)') + call addfld ('Z1000',horiz_only, 'A','m','Geopotential Z at 1000 mbar pressure surface') + call addfld ('Z975',horiz_only, 'A','m','Geopotential Z at 975 mbar pressure surface') + call addfld ('Z950',horiz_only, 'A','m','Geopotential Z at 950 mbar pressure surface') + call addfld ('Z925',horiz_only, 'A','m','Geopotential Z at 925 mbar pressure surface') + call addfld ('Z900',horiz_only, 'A','m','Geopotential Z at 900 mbar pressure surface') + call addfld ('Z850',horiz_only, 'A','m','Geopotential Z at 850 mbar pressure surface') + call addfld ('Z800',horiz_only, 'A','m','Geopotential Z at 800 mbar pressure surface') + call addfld ('Z700',horiz_only, 'A','m','Geopotential Z at 700 mbar pressure surface') + call addfld ('Z600',horiz_only, 'A','m','Geopotential Z at 600 mbar pressure surface') + call addfld ('Z500',horiz_only, 'A','m','Geopotential Z at 500 mbar pressure surface') + call addfld ('Z400',horiz_only, 'A','m','Geopotential Z at 400 mbar pressure surface') + call addfld ('Z300',horiz_only, 'A','m','Geopotential Z at 300 mbar pressure surface') + call addfld ('Z200',horiz_only, 'A','m','Geopotential Z at 200 mbar pressure surface') + call addfld ('Z100',horiz_only, 'A','m','Geopotential Z at 100 mbar pressure surface') + call addfld ('Z050',horiz_only, 'A','m','Geopotential Z at 50 mbar pressure surface') + call addfld ('Z010',horiz_only, 'A','m','Geopotential Z at 10 mbar pressure surface') + + call addfld ('ZZ',(/ 'lev' /), 'A','m2','Eddy height variance' ) + call addfld ('VZ',(/ 'lev' /), 'A','m2/s','Meridional transport of geopotential energy') + call addfld ('VT',(/ 'lev' /), 'A','K m/s ','Meridional heat transport') + call addfld ('VU',(/ 'lev' /), 'A','m2/s2','Meridional flux of zonal momentum' ) + call addfld ('VV',(/ 'lev' /), 'A','m2/s2','Meridional velocity squared' ) + + if(prog_modal_aero) then !Only for prognostic aerosols + call addfld ('bc_a1_2',(/ 'lev' /), 'A','kg2/kg2','bc_a1 squared') + call addfld ('dst_a1_2',(/ 'lev' /), 'A','kg2/kg2','dst_a1 squared') + call addfld ('dst_a3_2',(/ 'lev' /), 'A','kg2/kg2','dst_a3 squared') + call addfld ('ncl_a1_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a1 squared') + call addfld ('ncl_a2_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a2 squared') + call addfld ('ncl_a3_2',(/ 'lev' /), 'A','kg2/kg2','ncl_a3 squared') + call addfld ('so4_a1_2',(/ 'lev' /), 'A','kg2/kg2','so4_a1 squared') + call addfld ('so4_a2_2',(/ 'lev' /), 'A','kg2/kg2','so4_a2 squared') + call addfld ('so4_a3_2',(/ 'lev' /), 'A','kg2/kg2','so4_a3 squared') + call addfld ('soa_a1_2',(/ 'lev' /), 'A','kg2/kg2','soa_a1 squared') + call addfld ('soa_a2_2',(/ 'lev' /), 'A','kg2/kg2','soa_a2 squared') + call addfld ('pom_a1_2',(/ 'lev' /), 'A','kg2/kg2','pom_a1 squared') + call addfld ('Vbc_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional bc_a1 transport') + call addfld ('Vdst_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a1 transport') + call addfld ('Vdst_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional dst_a3 transport') + call addfld ('Vncl_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a1 transport') + call addfld ('Vncl_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a2 transport') + call addfld ('Vncl_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional ncl_a3 transport') + call addfld ('Vso4_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a1 transport') + call addfld ('Vso4_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a2 transport') + call addfld ('Vso4_a3',(/ 'lev' /), 'A','m/skg/kg','Meridional so4_a3 transport') + call addfld ('Vsoa_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a1 transport') + call addfld ('Vsoa_a2',(/ 'lev' /), 'A','m/skg/kg','Meridional soa_a2 transport') + call addfld ('Vpom_a1',(/ 'lev' /), 'A','m/skg/kg','Meridional pom_a1 transport') + endif + call addfld ('VQ',(/ 'lev' /), 'A','m/skg/kg','Meridional water transport') + call addfld ('QQ',(/ 'lev' /), 'A','kg2/kg2','Eddy moisture variance') + call addfld ('OMEGAV',(/ 'lev' /) ,'A','m Pa/s2 ','Vertical flux of meridional momentum' ) + call addfld ('OMGAOMGA',(/ 'lev' /) ,'A','Pa2/s2','Vertical flux of vertical momentum' ) + call addfld ('OMEGAQ',(/ 'lev' /) ,'A','kgPa/kgs','Vertical water transport' ) + + call addfld ('UU',(/ 'lev' /), 'A','m2/s2','Zonal velocity squared' ) + call addfld ('WSPEED',(/ 'lev' /), 'X','m/s','Horizontal total wind speed maximum' ) + call addfld ('WSPDSRFMX',horiz_only, 'X','m/s','Horizontal total wind speed maximum at the surface' ) + call addfld ('WSPDSRFAV',horiz_only, 'A','m/s','Horizontal total wind speed average at the surface' ) + + call addfld ('OMEGA',(/ 'lev' /), 'A','Pa/s','Vertical velocity (pressure)') + call addfld ('OMEGAT',(/ 'lev' /), 'A','K Pa/s ','Vertical heat flux' ) + call addfld ('OMEGAU',(/ 'lev' /), 'A','m Pa/s2 ','Vertical flux of zonal momentum' ) + call addfld ('OMEGA1000',horiz_only, 'A','Pa/s','Vertical velocity at 1000 mbar pressure surface') + call addfld ('OMEGA975',horiz_only, 'A','Pa/s','Vertical velocity at 975 mbar pressure surface') + call addfld ('OMEGA950',horiz_only, 'A','Pa/s','Vertical velocity at 950 mbar pressure surface') + call addfld ('OMEGA925',horiz_only, 'A','Pa/s','Vertical velocity at 925 mbar pressure surface') + call addfld ('OMEGA900',horiz_only, 'A','Pa/s','Vertical velocity at 900 mbar pressure surface') + call addfld ('OMEGA850',horiz_only, 'A','Pa/s','Vertical velocity at 850 mbar pressure surface') + call addfld ('OMEGA800',horiz_only, 'A','Pa/s','Vertical velocity at 800 mbar pressure surface') + call addfld ('OMEGA700',horiz_only, 'A','Pa/s','Vertical velocity at 700 mbar pressure surface') + call addfld ('OMEGA600',horiz_only, 'A','Pa/s','Vertical velocity at 600 mbar pressure surface') + call addfld ('OMEGA500',horiz_only, 'A','Pa/s','Vertical velocity at 500 mbar pressure surface') + call addfld ('OMEGA400',horiz_only, 'A','Pa/s','Vertical velocity at 400 mbar pressure surface') + call addfld ('OMEGA300',horiz_only, 'A','Pa/s','Vertical velocity at 300 mbar pressure surface') + call addfld ('OMEGA200',horiz_only, 'A','Pa/s','Vertical velocity at 200 mbar pressure surface') + call addfld ('OMEGA100',horiz_only, 'A','Pa/s','Vertical velocity at 100 mbar pressure surface') + call addfld ('OMEGABOT',horiz_only, 'A','Pa/s','Lowest model level vertical velocity') + + call addfld ('RH1000',horiz_only, 'A','%','Relative humidity at 1000 mbar pressure surface') + call addfld ('RH975',horiz_only, 'A','%','Relative humidity at 975 mbar pressure surface') + call addfld ('RH950',horiz_only, 'A','%','Relative humidity at 950 mbar pressure surface') + call addfld ('RH925',horiz_only, 'A','%','Relative humidity at 925 mbar pressure surface') + call addfld ('RH900',horiz_only, 'A','%','Relative humidity at 900 mbar pressure surface') + call addfld ('RH850',horiz_only, 'A','%','Relative humidity at 850 mbar pressure surface') + call addfld ('RH800',horiz_only, 'A','%','Relative humidity at 800 mbar pressure surface') + call addfld ('RH700',horiz_only, 'A','%','Relative humidity at 700 mbar pressure surface') + call addfld ('RH600',horiz_only, 'A','%','Relative humidity at 600 mbar pressure surface') + call addfld ('RH500',horiz_only, 'A','%','Relative humidity at 500 mbar pressure surface') + call addfld ('RH400',horiz_only, 'A','%','Relative humidity at 400 mbar pressure surface') + call addfld ('RH300',horiz_only, 'A','%','Relative humidity at 300 mbar pressure surface') + call addfld ('RH200',horiz_only, 'A','%','Relative humidity at 200 mbar pressure surface') + call addfld ('RH100',horiz_only, 'A','%','Relative humidity at 100 mbar pressure surface') + call addfld ('RHBOT',horiz_only, 'A','%','Lowest model level relative humidity') + + call addfld ('MQ',(/ 'lev' /), 'A','kg/m2','Water vapor mass in layer') + call addfld ('TMQ',horiz_only, 'A','kg/m2','Total (vertically integrated) precipitable water') + +! cas fields + call addfld ('IVT',horiz_only, 'A', 'kg/m/s','Total (vertically integrated) vapor transport') + call addfld ('uIVT',horiz_only, 'A', 'kg/m/s','u-component (vertically integrated) vapor transport') + call addfld ('vIVT',horiz_only, 'A', 'kg/m/s','v-component (vertically integrated) vapor transport') +! end cas fields + + call addfld ('TUQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) zonal water flux') + call addfld ('TVQ',horiz_only, 'A','kg/m/s','Total (vertically integrated) meridional water flux') + call addfld ('TUH',horiz_only, 'A','W/m', 'Total (vertically integrated) zonal MSE flux') + call addfld ('TVH',horiz_only, 'A','W/m', 'Total (vertically integrated) meridional MSE flux') + call addfld ('DTENDTH', horiz_only, 'A', 'W/m2', 'Dynamic Tendency of Total (vertically integrated) moist static energy') + call addfld ('DTENDTQ', horiz_only, 'A', 'kg/m2/s','Dynamic Tendency of Total (vertically integrated) specific humidity') + call addfld ('RELHUM',(/ 'lev' /), 'A','percent','Relative humidity') + call addfld ('RHW',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to liquid') + call addfld ('RHI',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to ice') + call addfld ('RHCFMIP',(/ 'lev' /), 'A','percent' ,'Relative humidity with respect to water above 273 K, ice below 273 K') + call addfld ('PSL',horiz_only, 'A','Pa','Sea level pressure') + + call addfld ('T850',horiz_only, 'A','K','Temperature at 850 mbar pressure surface') + call addfld ('T500',horiz_only, 'A','K','Temperature at 500 mbar pressure surface') + call addfld ('T300',horiz_only, 'A','K','Temperature at 300 mbar pressure surface') + call addfld ('T200',horiz_only, 'A','K','Temperature at 200 mbar pressure surface') + call addfld ('Q850',horiz_only, 'A','kg/kg','Specific Humidity at 850 mbar pressure surface') + call addfld ('Q500',horiz_only, 'A','kg/kg','Specific Humidity at 500 mbar pressure surface') + call addfld ('Q300',horiz_only, 'A','kg/kg','Specific Humidity at 300 mbar pressure surface') + call addfld ('Q200',horiz_only, 'A','kg/kg','Specific Humidity at 200 mbar pressure surface') + call addfld ('U1000',horiz_only, 'A','m/s','Zonal wind at 1000 mbar pressure surface') + call addfld ('U975',horiz_only, 'A','m/s','Zonal wind at 975 mbar pressure surface') + call addfld ('U950',horiz_only, 'A','m/s','Zonal wind at 950 mbar pressure surface') + call addfld ('U925',horiz_only, 'A','m/s','Zonal wind at 925 mbar pressure surface') + call addfld ('U900',horiz_only, 'A','m/s','Zonal wind at 900 mbar pressure surface') + call addfld ('U850',horiz_only, 'A','m/s','Zonal wind at 850 mbar pressure surface') + call addfld ('U800',horiz_only, 'A','m/s','Zonal wind at 800 mbar pressure surface') + call addfld ('U700',horiz_only, 'A','m/s','Zonal wind at 700 mbar pressure surface') + call addfld ('U600',horiz_only, 'A','m/s','Zonal wind at 600 mbar pressure surface') + call addfld ('U500',horiz_only, 'A','m/s','Zonal wind at 500 mbar pressure surface') + call addfld ('U400',horiz_only, 'A','m/s','Zonal wind at 400 mbar pressure surface') + call addfld ('U300',horiz_only, 'A','m/s','Zonal wind at 300 mbar pressure surface') + call addfld ('U250',horiz_only, 'A','m/s','Zonal wind at 250 mbar pressure surface') + call addfld ('U200',horiz_only, 'A','m/s','Zonal wind at 200 mbar pressure surface') + call addfld ('U100',horiz_only, 'A','m/s','Zonal wind at 100 mbar pressure surface') + call addfld ('U050',horiz_only, 'A','m/s','Zonal wind at 50 mbar pressure surface') + call addfld ('U010',horiz_only, 'A','m/s','Zonal wind at 10 mbar pressure surface') + call addfld ('V1000',horiz_only, 'A','m/s','Meridional wind at 1000 mbar pressure surface') + call addfld ('V975',horiz_only, 'A','m/s','Meridional wind at 975 mbar pressure surface') + call addfld ('V950',horiz_only, 'A','m/s','Meridional wind at 950 mbar pressure surface') + call addfld ('V925',horiz_only, 'A','m/s','Meridional wind at 925 mbar pressure surface') + call addfld ('V900',horiz_only, 'A','m/s','Meridional wind at 900 mbar pressure surface') + call addfld ('V850',horiz_only, 'A','m/s','Meridional wind at 850 mbar pressure surface') + call addfld ('V800',horiz_only, 'A','m/s','Meridional wind at 800 mbar pressure surface') + call addfld ('V700',horiz_only, 'A','m/s','Meridional wind at 700 mbar pressure surface') + call addfld ('V600',horiz_only, 'A','m/s','Meridional wind at 600 mbar pressure surface') + call addfld ('V500',horiz_only, 'A','m/s','Meridional wind at 500 mbar pressure surface') + call addfld ('V400',horiz_only, 'A','m/s','Meridional wind at 400 mbar pressure surface') + call addfld ('V300',horiz_only, 'A','m/s','Meridional wind at 300 mbar pressure surface') + call addfld ('V250',horiz_only, 'A','m/s','Meridional wind at 250 mbar pressure surface') + call addfld ('V200',horiz_only, 'A','m/s','Meridional wind at 200 mbar pressure surface') + call addfld ('V100',horiz_only, 'A','m/s','Meridional wind at 100 mbar pressure surface') + call addfld ('V050',horiz_only, 'A','m/s','Meridional wind at 50 mbar pressure surface') + call addfld ('V010',horiz_only, 'A','m/s','Meridional wind at 10 mbar pressure surface') + + call addfld ('TT',(/ 'lev' /), 'A','K2','Eddy temperature variance' ) + + call addfld ('UBOT',horiz_only, 'A','m/s','Lowest model level zonal wind') + call addfld ('VBOT',horiz_only, 'A','m/s','Lowest model level meridional wind') + call addfld ('QBOT',horiz_only, 'A','kg/kg','Lowest model level water vapor mixing ratio') + call addfld ('ZBOT',horiz_only, 'A','m','Lowest model level height') + + call addfld ('ATMEINT',horiz_only, 'A','J/m2','Vertically integrated total atmospheric energy ') + + call addfld ('T1000',horiz_only, 'A','K','Temperature at 1000 mbar pressure surface') + call addfld ('T975',horiz_only, 'A','K','Temperature at 975 mbar pressure surface') + call addfld ('T950',horiz_only, 'A','K','Temperature at 950 mbar pressure surface') + call addfld ('T925',horiz_only, 'A','K','Temperature at 925 mbar pressure surface') + call addfld ('T900',horiz_only, 'A','K','Temperature at 900 mbar pressure surface') + call addfld ('T800',horiz_only, 'A','K','Temperature at 800 mbar pressure surface') + call addfld ('T700',horiz_only, 'A','K','Temperature at 700 mbar pressure surface') + call addfld ('T600',horiz_only, 'A','K','Temperature at 600 mbar pressure surface') + call addfld ('T400',horiz_only, 'A','K','Temperature at 400 mbar pressure surface') + call addfld ('T100',horiz_only, 'A','K','Temperature at 100 mbar pressure surface') + call addfld ('T050',horiz_only, 'A','K','Temperature at 50 mbar pressure surface') + call addfld ('T010',horiz_only, 'A','K','Temperature at 10 mbar pressure surface') + call addfld ('Q1000',horiz_only, 'A','kg/kg','Specific Humidity at 1000 mbar pressure surface') + call addfld ('Q975',horiz_only, 'A','kg/kg','Specific Humidity at 975 mbar pressure surface') + call addfld ('Q950',horiz_only, 'A','kg/kg','Specific Humidity at 950 mbar pressure surface') + call addfld ('Q925',horiz_only, 'A','kg/kg','Specific Humidity at 925 mbar pressure surface') + call addfld ('Q900',horiz_only, 'A','kg/kg','Specific Humidity at 900 mbar pressure surface') + call addfld ('Q800',horiz_only, 'A','kg/kg','Specific Humidity at 800 mbar pressure surface') + call addfld ('Q700',horiz_only, 'A','kg/kg','Specific Humidity at 700 mbar pressure surface') + call addfld ('Q600',horiz_only, 'A','kg/kg','Specific Humidity at 600 mbar pressure surface') + call addfld ('Q400',horiz_only, 'A','kg/kg','Specific Humidity at 400 mbar pressure surface') + call addfld ('Q100',horiz_only, 'A','kg/kg','Specific Humidity at 100 mbar pressure surface') + call addfld ('Q050',horiz_only, 'A','kg/kg','Specific Humidity at 050 mbar pressure surface') + call addfld ('Q010',horiz_only, 'A','kg/kg','Specific Humidity at 010 mbar pressure surface') + + call addfld ('T7001000',horiz_only, 'A','K','Temperature difference 700 mb - 1000 mb') + call addfld ('TH7001000',horiz_only, 'A','K','Theta difference 700 mb - 1000 mb') + call addfld ('THE7001000',horiz_only, 'A','K','ThetaE difference 700 mb - 1000 mb') + + call addfld ('T8501000',horiz_only, 'A','K','Temperature difference 850 mb - 1000 mb') + call addfld ('TH8501000',horiz_only, 'A','K','Theta difference 850 mb - 1000 mb') + call addfld ('THE8501000',horiz_only, 'A','K','ThetaE difference 850 mb - 1000 mb') + call addfld ('T9251000',horiz_only, 'A','K','Temperature difference 925 mb - 1000 mb') + call addfld ('TH9251000',horiz_only, 'A','K','Theta difference 925 mb - 1000 mb') + call addfld ('THE9251000',horiz_only, 'A','K','ThetaE difference 925 mb - 1000 mb') + + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call addfld('QRS', (/ 'lev' /), 'A', 'K/s', 'Solar heating rate') + end if + + ! ---------------------------- + ! determine default variables + ! ---------------------------- + call phys_getopts(history_amwg_out = history_amwg , & + history_verbose_out = history_verbose , & + history_vdiag_out = history_vdiag , & + history_eddy_out = history_eddy , & + history_budget_out = history_budget , & + history_budget_histfile_num_out = history_budget_histfile_num) + + if (history_amwg) then + call add_default ('PHIS ' , 1, ' ') + call add_default ('PS ' , 1, ' ') + call add_default ('T ' , 1, ' ') + call add_default ('U ' , 1, ' ') + call add_default ('V ' , 1, ' ') + call add_default (cnst_name(1), 1, ' ') + call add_default ('Z3 ' , 1, ' ') + call add_default ('OMEGA ' , 1, ' ') + call add_default ('VT ', 1, ' ') + call add_default ('VU ', 1, ' ') + call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') + + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('Vbc_a1 ', 1, ' ') + call add_default ('Vdst_a1 ', 1, ' ') + call add_default ('Vdst_a3 ', 1, ' ') + call add_default ('Vncl_a1 ', 1, ' ') + call add_default ('Vncl_a2 ', 1, ' ') + call add_default ('Vncl_a3 ', 1, ' ') + call add_default ('Vso4_a1 ', 1, ' ') + call add_default ('Vso4_a2 ', 1, ' ') + call add_default ('Vso4_a3 ', 1, ' ') + call add_default ('Vsoa_a1 ', 1, ' ') + call add_default ('Vsoa_a2 ', 1, ' ') + call add_default ('Vpom_a1 ', 1, ' ') + endif + call add_default ('UU ', 1, ' ') + call add_default ('OMEGAT ', 1, ' ') + if(prog_modal_aero .and. history_verbose) then !Only for prognostic aerosols + call add_default ('bc_a1_2 ', 1, ' ') + call add_default ('dst_a1_2', 1, ' ') + call add_default ('dst_a3_2', 1, ' ') + call add_default ('ncl_a1_2', 1, ' ') + call add_default ('ncl_a2_2', 1, ' ') + call add_default ('ncl_a3_2', 1, ' ') + call add_default ('so4_a1_2', 1, ' ') + call add_default ('so4_a2_2', 1, ' ') + call add_default ('so4_a3_2', 1, ' ') + call add_default ('soa_a1_2', 1, ' ') + call add_default ('soa_a2_2', 1, ' ') + call add_default ('pom_a1_2', 1, ' ') + endif + call add_default ('TMQ ', 1, ' ') + call add_default ('PSL ', 1, ' ') + if (moist_physics) then + call add_default ('RELHUM ', 1, ' ') + end if + + ! For Tier 1b global water cycle diagostics + call add_default ('TUQ ', 1, ' ') + call add_default ('TVQ ', 1, ' ') + call add_default ('TUH ', 1, ' ') + call add_default ('TVH ', 1, ' ') + call add_default ('DTENDTQ', 1, ' ') + call add_default ('DTENDTH', 1, ' ') + end if + + if (history_vdiag) then + call add_default ('U200', 2, ' ') + call add_default ('V200', 2, ' ') + call add_default ('U850', 2, ' ') + call add_default ('U200', 3, ' ') + call add_default ('U850', 3, ' ') + call add_default ('OMEGA500', 3, ' ') + end if + + if (history_eddy) then + call add_default ('VT ', 1, ' ') + call add_default ('VU ', 1, ' ') + call add_default ('VV ', 1, ' ') + call add_default ('VQ ', 1, ' ') + call add_default ('UU ', 1, ' ') + call add_default ('OMEGAT ', 1, ' ') + call add_default ('OMEGAQ ', 1, ' ') + call add_default ('OMEGAU ', 1, ' ') + call add_default ('OMEGAV ', 1, ' ') + endif + + if ( history_budget ) then + call add_default ('PHIS ' , history_budget_histfile_num, ' ') + call add_default ('PS ' , history_budget_histfile_num, ' ') + call add_default ('T ' , history_budget_histfile_num, ' ') + call add_default ('U ' , history_budget_histfile_num, ' ') + call add_default ('V ' , history_budget_histfile_num, ' ') + call add_default (cnst_name(1), history_budget_histfile_num, ' ') + call add_default ('TTEND_TOT' , history_budget_histfile_num, ' ') + + ! State before physics (FV) + call add_default ('TBP ' , history_budget_histfile_num, ' ') + call add_default (bpcnst(1) , history_budget_histfile_num, ' ') + ! State after physics (FV) + call add_default ('TAP ' , history_budget_histfile_num, ' ') + call add_default ('UAP ' , history_budget_histfile_num, ' ') + call add_default ('VAP ' , history_budget_histfile_num, ' ') + call add_default (apcnst(1) , history_budget_histfile_num, ' ') + if ( dycore_is('LR') .or. dycore_is('SE') ) then + call add_default ('TFIX ' , history_budget_histfile_num, ' ') + call add_default ('PTTEND_RESID', history_budget_histfile_num, ' ') + end if + end if + + ! This field is added by radiation when full physics is used + if ( ideal_phys )then + call add_default('QRS ', 1, ' ') + end if + + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + ! Exit here for adiabatic/ideal physics cases ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + if (.not. moist_physics) return + + + call addfld ('PDELDRY',(/ 'lev' /), 'A','Pa','Dry pressure difference between levels') + call addfld ('PSDRY',horiz_only, 'A','Pa','Surface pressure') + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + call add_default ('PS ', 2, ' ') + call add_default ('T ', 2, ' ') + end if + + ! outfld calls in diag_conv + + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call addfld ('DTCOND',(/ 'lev' /), 'A','K/s','T tendency - moist processes') + call addfld ('DTCOND_24_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. cos coeff.') + call addfld ('DTCOND_24_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 24hr. sin coeff.') + call addfld ('DTCOND_12_COS',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. cos coeff.') + call addfld ('DTCOND_12_SIN',(/ 'lev' /), 'A','K/s','T tendency - moist processes 12hr. sin coeff.') + + ! determine number of constituents for which convective tendencies must be computed + if (history_budget) then + dqcond_num = pcnst + else + if (diag_cnst_conv_tend == 'none') dqcond_num = 0 + if (diag_cnst_conv_tend == 'q_only') dqcond_num = 1 + if (diag_cnst_conv_tend == 'all') dqcond_num = pcnst + end if + + do m = 1, dqcond_num + dcconnam(m) = 'DC'//cnst_name(m) + end do + + if (diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' .or. history_budget) then + call addfld (dcconnam(1),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(1))//' tendency due to moist processes') + if ( diag_cnst_conv_tend == 'q_only' .or. diag_cnst_conv_tend == 'all' ) then + call add_default (dcconnam(1), 1, ' ') + end if + if( history_budget ) then + call add_default (dcconnam(1), history_budget_histfile_num, ' ') + end if + if (diag_cnst_conv_tend == 'all' .or. history_budget) then + do m = 2, pcnst + call addfld (dcconnam(m),(/ 'lev' /),'A', 'kg/kg/s',trim(cnst_name(m))//' tendency due to moist processes') + if( diag_cnst_conv_tend == 'all' ) then + call add_default (dcconnam(m), 1, ' ') + end if + if( history_budget .and. (m == ixcldliq .or. m == ixcldice) ) then + call add_default (dcconnam(m), history_budget_histfile_num, ' ') + end if + end do + end if + end if + + call addfld ('PRECL',horiz_only, 'A','m/s','Large-scale (stable) precipitation rate (liq + ice)' ) + call addfld ('PRECC',horiz_only, 'A','m/s','Convective precipitation rate (liq + ice)' ) + call addfld ('PRECT',horiz_only, 'A','m/s','Total (convective and large-scale) precipitation rate (liq + ice)' ) + call addfld ('PREC_PCW',horiz_only, 'A','m/s','LS_pcw precipitation rate') + call addfld ('PREC_zmc',horiz_only, 'A','m/s','CV_zmc precipitation rate') + call addfld ('PRECTMX',horiz_only, 'X','m/s','Maximum (convective and large-scale) precipitation rate (liq+ice)' ) + call addfld ('PRECSL',horiz_only, 'A','m/s','Large-scale (stable) snow rate (water equivalent)' ) + call addfld ('PRECSC',horiz_only, 'A','m/s','Convective snow rate (water equivalent)' ) + call addfld ('PRECCav',horiz_only, 'A','m/s','Average large-scale precipitation (liq + ice)' ) + call addfld ('PRECLav',horiz_only, 'A','m/s','Average convective precipitation (liq + ice)' ) + + ! outfld calls in diag_surf + + call addfld ('SHFLX',horiz_only, 'A','W/m2','Surface sensible heat flux') + call addfld ('LHFLX',horiz_only, 'A','W/m2','Surface latent heat flux') + call addfld ('QFLX',horiz_only, 'A','kg/m2/s','Surface water flux') + + call addfld ('TAUX',horiz_only, 'A','N/m2','Zonal surface stress') + call addfld ('TAUY',horiz_only, 'A','N/m2','Meridional surface stress') + call addfld ('TREFHT',horiz_only, 'A','K','Reference height temperature') + call addfld ('TREFHTMN',horiz_only, 'M','K','Minimum reference height temperature over output period') + call addfld ('TREFHTMX',horiz_only, 'X','K','Maximum reference height temperature over output period') + call addfld ('QREFHT', horiz_only, 'A','kg/kg','Reference height humidity') + call addfld ('U10', horiz_only, 'A','m/s','10m wind speed') + call addfld ('RHREFHT',horiz_only, 'A','fraction','Reference height relative humidity') + + call addfld ('LANDFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by land') + call addfld ('ICEFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('OCNFRAC',horiz_only, 'A','fraction','Fraction of sfc area covered by ocean') + + call addfld ('TREFMNAV',horiz_only, 'A','K','Average of TREFHT daily minimum') + call addfld ('TREFMXAV',horiz_only, 'A','K','Average of TREFHT daily maximum') + + call addfld ('TS',horiz_only, 'A','K','Surface temperature (radiative)') + call addfld ('TSMN',horiz_only, 'M','K','Minimum surface temperature over output period') + call addfld ('TSMX',horiz_only, 'X','K','Maximum surface temperature over output period') + call addfld ('SNOWHLND',horiz_only, 'A','m','Water equivalent snow depth') + call addfld ('SNOWHICE',horiz_only, 'A','m','Snow depth over ice', fill_value = 1.e30_r8) + call addfld ('TBOT',horiz_only, 'A','K','Lowest model level temperature') + + call addfld ('ASDIR', horiz_only, 'A', '1','albedo: shortwave, direct') + call addfld ('ASDIF', horiz_only, 'A', '1','albedo: shortwave, diffuse') + call addfld ('ALDIR', horiz_only, 'A', '1','albedo: longwave, direct') + call addfld ('ALDIF', horiz_only, 'A', '1','albedo: longwave, diffuse') + call addfld ('SST', horiz_only, 'A', 'K','sea surface temperature') + + ! defaults + if (history_amwg) then + call add_default ('DTCOND ', 1, ' ') + call add_default ('PRECL ', 1, ' ') + call add_default ('PRECC ', 1, ' ') + call add_default ('PRECSL ', 1, ' ') + call add_default ('PRECSC ', 1, ' ') + call add_default ('SHFLX ', 1, ' ') + call add_default ('LHFLX ', 1, ' ') + call add_default ('QFLX ', 1, ' ') + call add_default ('TAUX ', 1, ' ') + call add_default ('TAUY ', 1, ' ') + call add_default ('TREFHT ', 1, ' ') + call add_default ('LANDFRAC', 1, ' ') + call add_default ('OCNFRAC ', 1, ' ') + call add_default ('QREFHT ', 1, ' ') + call add_default ('U10 ', 1, ' ') + call add_default ('ICEFRAC ', 1, ' ') + call add_default ('TS ', 1, ' ') + call add_default ('TSMN ', 1, ' ') + call add_default ('TSMX ', 1, ' ') + call add_default ('SNOWHLND', 1, ' ') + call add_default ('SNOWHICE', 1, ' ') + endif + + call add_default ('OMEGA500', 1, ' ') + call add_default ('TH7001000', 1, ' ') + + if (history_vdiag) then + call add_default ('PRECT ', 2, ' ') + call add_default ('PRECT ', 3, ' ') + call add_default ('PRECT ', 4, ' ') + end if + + ! outfld calls in diag_phys_tend_writeout + + call addfld ('PTTEND' ,(/ 'lev' /), 'A','K/s','T total physics tendency' ) + call addfld (ptendnam( 1),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name( 1))//' total physics tendency ' ) + call addfld (ptendnam(ixcldliq),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldliq))//' total physics tendency ' ) + call addfld (ptendnam(ixcldice),(/ 'lev' /), 'A', 'kg/kg/s',trim(cnst_name(ixcldice))//' total physics tendency ' ) + if ( dycore_is('LR') )then + call addfld (dmetendnam( 1),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name( 1))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldliq),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldliq))//' dme adjustment tendency (FV) ') + call addfld (dmetendnam(ixcldice),(/ 'lev' /), 'A','kg/kg/s', & + trim(cnst_name(ixcldice))//' dme adjustment tendency (FV) ') + end if + + if ( history_budget ) then + call add_default ('PTTEND' , history_budget_histfile_num, ' ') + call add_default (ptendnam( 1), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default (ptendnam(ixcldice), history_budget_histfile_num, ' ') + if ( dycore_is('LR') )then + call add_default(dmetendnam(1) , history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldliq), history_budget_histfile_num, ' ') + call add_default(dmetendnam(ixcldice), history_budget_histfile_num, ' ') + end if + if( history_budget_histfile_num > 1 ) then + call add_default ('DTCOND ' , history_budget_histfile_num, ' ') + end if + end if + + ! outfld calls in diag_physvar_ic + + call addfld ('QCWAT&IC',(/ 'lev' /), 'I','kg/kg','q associated with cloud water' ) + call addfld ('TCWAT&IC',(/ 'lev' /), 'I','kg/kg','T associated with cloud water' ) + call addfld ('LCWAT&IC',(/ 'lev' /), 'I','kg/kg','Cloud water (ice + liq' ) + call addfld ('CLOUD&IC',(/ 'lev' /), 'I','fraction','Cloud fraction' ) + call addfld ('CONCLD&IC',(/ 'lev' /), 'I','fraction','Convective cloud fraction' ) + call addfld ('TKE&IC',(/ 'ilev' /),'I','m2/s2','Turbulent Kinetic Energy' ) + call addfld ('CUSH&IC',horiz_only, 'I','m','Convective Scale Height' ) + call addfld ('KVH&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (heat/moisture)') + call addfld ('KVM&IC',(/ 'ilev' /),'I','m2/s','Vertical diffusion diffusivities (momentum)' ) + call addfld ('PBLH&IC',horiz_only, 'I','m','PBL height' ) + call addfld ('TPERT&IC',horiz_only, 'I','K','Perturbation temperature (eddies in PBL)' ) + call addfld ('QPERT&IC',horiz_only, 'I','kg/kg','Perturbation specific humidity (eddies in PBL)' ) + call addfld ('TBOT&IC',horiz_only, 'I','K','Lowest model level temperature' ) + + + ! Initial file - Optional fields + + if (inithist_all) then + call add_default ('CONCLD&IC ',0, 'I') + call add_default ('QCWAT&IC ',0, 'I') + call add_default ('TCWAT&IC ',0, 'I') + call add_default ('LCWAT&IC ',0, 'I') + call add_default ('PBLH&IC ',0, 'I') + call add_default ('TPERT&IC ',0, 'I') + call add_default ('QPERT&IC ',0, 'I') + call add_default ('CLOUD&IC ',0, 'I') + call add_default ('TKE&IC ',0, 'I') + call add_default ('CUSH&IC ',0, 'I') + call add_default ('KVH&IC ',0, 'I') + call add_default ('KVM&IC ',0, 'I') + call add_default ('TBOT&IC ',0, 'I') + end if + + ! CAM export state + call addfld('a2x_BCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic black carbon') + call addfld('a2x_BCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic black carbon') + call addfld('a2x_BCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic black carbon') + call addfld('a2x_OCPHIWET', horiz_only, 'A', 'kg/m2/s', 'wetdep of hydrophilic organic carbon') + call addfld('a2x_OCPHIDRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophilic organic carbon') + call addfld('a2x_OCPHODRY', horiz_only, 'A', 'kg/m2/s', 'drydep of hydrophobic organic carbon') + call addfld('a2x_DSTWET1', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin1)') + call addfld('a2x_DSTDRY1', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin1)') + call addfld('a2x_DSTWET2', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin2)') + call addfld('a2x_DSTDRY2', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin2)') + call addfld('a2x_DSTWET3', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin3)') + call addfld('a2x_DSTDRY3', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin3)') + call addfld('a2x_DSTWET4', horiz_only, 'A', 'kg/m2/s', 'wetdep of dust (bin4)') + call addfld('a2x_DSTDRY4', horiz_only, 'A', 'kg/m2/s', 'drydep of dust (bin4)') + + !--------------------------------------------------------- + ! CAM history fields for CAM-DOM/CAM-CSIM + !--------------------------------------------------------- + + ! CAM-DOM history fields +#ifdef COUP_DOM + call addfld ('TSOCN&IC',horiz_only, 'I','m','Ocean tempertare') + call add_default ('TSOCN&IC ',0, 'I') +#endif + + ! CAM-CSIM history fields + + do k=1,plevmx + call addfld (tsnam(k),horiz_only,'A','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK' ,horiz_only,'A','m','Sea ice thickness') + call addfld ('TSICE' ,horiz_only,'A','K','Ice temperature') + do k = 1,plevmx + call addfld (trim(tsnam(k))//'&IC',horiz_only,'I','K',tsnam(k)//' subsoil temperature') + end do + call addfld ('SICTHK&IC',horiz_only,'I','m','Sea ice thickness' ) + call addfld ('TSICE&IC',horiz_only,'I','K','Ice temperature' ) + call addfld ('SNOWHICE&IC',horiz_only,'I','m','Water equivalent snow depth' ) + call addfld ('ICEFRAC&IC',horiz_only,'I','fraction','Fraction of sfc area covered by sea-ice') + call addfld ('TSICERAD&IC',horiz_only,'I','K','Radiatively equivalent ice temperature' ) + do k = 1,plevmx + call add_default(trim(tsnam(k))//'&IC',0, 'I') + end do + call add_default ('SICTHK&IC ',0, 'I') + call add_default ('TSICE&IC ',0, 'I') + call add_default ('SNOWHICE&IC',0, 'I') + call add_default ('ICEFRAC&IC ',0, 'I') + if (inithist_all) then + call add_default ('TSICERAD&IC',0, 'I') + end if + + !--------------------------------------------------------- + ! WACCM diagnostic history fields + !--------------------------------------------------------- + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + + ! create history variables for fourier coefficients of the diurnal + ! and semidiurnal tide in T, U, V, and Z3 + + call tidal_diag_init() + + endif + + qcwat_idx = pbuf_get_index('QCWAT',ierr) + tcwat_idx = pbuf_get_index('TCWAT',ierr) + lcwat_idx = pbuf_get_index('LCWAT',ierr) + cld_idx = pbuf_get_index('CLD') + concld_idx = pbuf_get_index('CONCLD') + + tke_idx = pbuf_get_index('tke') + kvm_idx = pbuf_get_index('kvm') + kvh_idx = pbuf_get_index('kvh') + cush_idx = pbuf_get_index('cush') + + pblh_idx = pbuf_get_index('pblh') + tpert_idx = pbuf_get_index('tpert') + qpert_idx = pbuf_get_index('qpert',ierr) + + prec_dp_idx = pbuf_get_index('PREC_DP') + snow_dp_idx = pbuf_get_index('SNOW_DP') + prec_sh_idx = pbuf_get_index('PREC_SH') + snow_sh_idx = pbuf_get_index('SNOW_SH') + prec_sed_idx = pbuf_get_index('PREC_SED') + snow_sed_idx = pbuf_get_index('SNOW_SED') + prec_pcw_idx = pbuf_get_index('PREC_PCW') + snow_pcw_idx = pbuf_get_index('SNOW_PCW') + +end subroutine diag_init + +!=============================================================================== + +subroutine diag_allocate() + use infnan, only: nan, assignment(=) + + ! Allocate memory for module variables. + ! Done at the begining of a physics step at same point as the pbuf allocate for + ! variables with "physpkg" scope. + + ! Local variables + character(len=*), parameter :: sub = 'diag_allocate' + integer :: i, istat + + allocate(dtcond(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dtcond = nan + + if (dqcond_num > 0) then + allocate(dqcond(dqcond_num)) + do i = 1, dqcond_num + allocate(dqcond(i)%cnst(pcols,pver,begchunk:endchunk), stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: allocate failed') + dqcond(i)%cnst = nan + end do + end if + +end subroutine diag_allocate + +!=============================================================================== + +subroutine diag_deallocate() + +! Deallocate memory for module variables. +! Done at the end of a physics step at same point as the pbuf deallocate for +! variables with "physpkg" scope. + +! Local variables + character(len=*), parameter :: sub = 'diag_deallocate' + integer :: i, istat + + deallocate(dtcond, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + + if (dqcond_num > 0) then + do i = 1, dqcond_num + deallocate(dqcond(i)%cnst, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + end do + deallocate(dqcond, stat=istat) + if ( istat /= 0 ) call endrun (sub//': ERROR: deallocate failed') + end if + +end subroutine diag_deallocate +!=============================================================================== + +subroutine diag_conv_tend_ini(state,pbuf) + +! Initialize convective tendency calcs. + + +! Argument: + + type(physics_state), intent(in) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) + +! Local variables: + + integer :: i, k, m, lchnk, ncol + real(r8), pointer, dimension(:,:) :: t_ttend + + lchnk = state%lchnk + ncol = state%ncol + + do k = 1, pver + do i = 1, ncol + dtcond(i,k,lchnk) = state%s(i,k) + end do + end do + + do m = 1, dqcond_num + do k = 1, pver + do i = 1, ncol + dqcond(m)%cnst(i,k,lchnk) = state%q(i,k,m) + end do + end do + end do + + !! initialize to pbuf T_TTEND to temperature at first timestep + if (is_first_step()) then + do m = 1, dyn_time_lvls + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,m/), kount=(/pcols,pver,1/)) + t_ttend(:ncol,:) = state%t(:ncol,:) + end do + end if + +end subroutine diag_conv_tend_ini +!=============================================================================== + + subroutine diag_phys_writeout(state, psl) + +!----------------------------------------------------------------------- +! +! Purpose: record dynamics variables on physics grid +! +!----------------------------------------------------------------------- + use physconst, only: gravit, rga, rair, cpair, latvap, rearth, pi, cappa + use time_manager, only: get_nstep + use interpolate_data, only: vertinterp + use constituent_burden, only: constituent_burden_comp + use cam_control_mod, only: moist_physics + use co2_cycle, only: c_i, co2_transport + + use tidal_diag, only: tidal_diag_write +!----------------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(inout) :: state + real(r8), optional , intent(out) :: psl(pcols) +! +!---------------------------Local workspace----------------------------- +! + real(r8) ftem(pcols,pver) ! temporary workspace + real(r8) ftem1(pcols,pver) ! another temporary workspace + real(r8) ftem2(pcols,pver) ! another temporary workspace + real(r8) ftem4(pcols,pver) ! another temporary workspace + real(r8) ftem5(pcols,pver) ! another temporary workspace + real(r8) psl_tmp(pcols) ! Sea Level Pressure + real(r8) z3(pcols,pver) ! geo-potential height + real(r8) p_surf(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_t2(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q1(pcols) ! data interpolated to a pressure surface + real(r8) p_surf_q2(pcols) ! data interpolated to a pressure surface + real(r8) tem2(pcols,pver) ! temporary workspace + real(r8) timestep(pcols) ! used for outfld call + real(r8) esl(pcols,pver) ! saturation vapor pressures + real(r8) esi(pcols,pver) ! + real(r8) dlon(pcols) ! width of grid cell (meters) + integer plon ! number of longitudes + + integer i, k, m, lchnk, ncol, nstep +! +!----------------------------------------------------------------------- +! + lchnk = state%lchnk + ncol = state%ncol + + ! Output NSTEP for debugging + nstep = get_nstep() + timestep(:ncol) = nstep + call outfld ('NSTEP ',timestep, pcols, lchnk) + + call outfld('T ',state%t , pcols ,lchnk ) + call outfld('PS ',state%ps, pcols ,lchnk ) + call outfld('U ',state%u , pcols ,lchnk ) + call outfld('V ',state%v , pcols ,lchnk ) + do m=1,pcnst + if ( cnst_cam_outfld(m) ) then + call outfld(cnst_name(m),state%q(1,1,m),pcols ,lchnk ) + end if + end do + + if (co2_transport()) then + do m = 1,4 + call outfld(trim(cnst_name(c_i(m)))//'_BOT', state%q(1,pver,c_i(m)), pcols, lchnk) + end do + end if + + ! column burdens of all constituents except water vapor + call constituent_burden_comp(state) + + if ( moist_physics) then + call outfld('PDELDRY ',state%pdeldry, pcols, lchnk) + call outfld('PSDRY', state%psdry, pcols, lchnk) + end if + + call outfld('PHIS ',state%phis, pcols, lchnk ) + + + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('phis ',state%phis, pcols, lchnk ) +#endif + +! +! Add height of surface to midpoint height above surface +! + do k = 1, pver + z3(:ncol,k) = state%zm(:ncol,k) + state%phis(:ncol)*rga + end do + call outfld('Z3 ',z3,pcols,lchnk) +! +! Output Z3 on pressure surfaces +! + if (hist_fld_active('Z1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, z3, p_surf) + call outfld('Z1000 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, z3, p_surf) + call outfld('Z975 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, z3, p_surf) + call outfld('Z950 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, z3, p_surf) + call outfld('Z925 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, z3, p_surf) + call outfld('Z900 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, z3, p_surf) + call outfld('Z850 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, z3, p_surf) + call outfld('Z800 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, z3, p_surf) + call outfld('Z700 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, z3, p_surf) + call outfld('Z600 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, z3, p_surf) + call outfld('Z500 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, z3, p_surf) + call outfld('Z400 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, z3, p_surf) + call outfld('Z300 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, z3, p_surf) + call outfld('Z200 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, z3, p_surf) + call outfld('Z100 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, z3, p_surf) + call outfld('Z050 ', p_surf, pcols, lchnk) + end if + if (hist_fld_active('Z010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, z3, p_surf) + call outfld('Z010 ', p_surf, pcols, lchnk) + end if +! +! Quadratic height fiels Z3*Z3 +! + ftem(:ncol,:) = z3(:ncol,:)*z3(:ncol,:) + call outfld('ZZ ',ftem,pcols,lchnk) + + ftem(:ncol,:) = z3(:ncol,:)*state%v(:ncol,:)*gravit + call outfld('VZ ',ftem, pcols,lchnk) +! +! Meridional advection fields +! + ftem(:ncol,:) = state%v(:ncol,:)*state%t(:ncol,:) + call outfld ('VT ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1) + call outfld ('VQ ',ftem ,pcols ,lchnk ) + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,14) + call outfld ('Vbc_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,15) + call outfld ('Vdst_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,22) + call outfld ('Vdst_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,16) + call outfld ('Vncl_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,20) + call outfld ('Vncl_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,23) + call outfld ('Vncl_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,11) + call outfld ('Vso4_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,18) + call outfld ('Vso4_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,24) + call outfld ('Vso4_a3 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,13) + call outfld ('Vsoa_a1 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,19) + call outfld ('Vsoa_a2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,12) + call outfld ('Vpom_a1 ',ftem ,pcols ,lchnk ) + endif + + ftem(:ncol,:) = state%q(:ncol,:,1)*state%q(:ncol,:,1) + call outfld ('QQ ',ftem ,pcols ,lchnk ) + + if(prog_modal_aero) then !Only for prognostic aerosols + ftem(:ncol,:) = state%q(:ncol,:,14)*state%q(:ncol,:,14) + call outfld ('bc_a1_2 ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,15)*state%q(:ncol,:,15) + call outfld ('dst_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,22)*state%q(:ncol,:,22) + call outfld ('dst_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,16)*state%q(:ncol,:,16) + call outfld ('ncl_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,20)*state%q(:ncol,:,20) + call outfld ('ncl_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,23)*state%q(:ncol,:,23) + call outfld ('ncl_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,11)*state%q(:ncol,:,11) + call outfld ('so4_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,18)*state%q(:ncol,:,18) + call outfld ('so4_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,24)*state%q(:ncol,:,24) + call outfld ('so4_a3_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,13)*state%q(:ncol,:,13) + call outfld ('soa_a1_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,19)*state%q(:ncol,:,19) + call outfld ('soa_a2_2',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%q(:ncol,:,12)*state%q(:ncol,:,12) + call outfld ('pom_a1_2',ftem ,pcols ,lchnk ) + endif + + ftem(:ncol,:) = state%v(:ncol,:)**2 + call outfld ('VV ',ftem ,pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:) * state%u(:ncol,:) + call outfld ('VU ',ftem ,pcols ,lchnk ) + +! zonal advection + + ftem(:ncol,:) = state%u(:ncol,:)**2 + call outfld ('UU ',ftem ,pcols ,lchnk ) + +! Wind speed + ftem(:ncol,:) = sqrt( state%u(:ncol,:)**2 + state%v(:ncol,:)**2) + call outfld ('WSPEED ',ftem ,pcols ,lchnk ) + call outfld ('WSPDSRFMX',ftem(:,pver) ,pcols ,lchnk ) + call outfld ('WSPDSRFAV',ftem(:,pver) ,pcols ,lchnk ) + +! Vertical velocity and advection + + if (single_column) then + call outfld('OMEGA ',wfld, pcols, lchnk ) + else + call outfld('OMEGA ',state%omega, pcols, lchnk ) + endif + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('omega ',state%omega, pcols, lchnk ) +#endif + + ftem(:ncol,:) = state%omega(:ncol,:)*state%t(:ncol,:) + call outfld('OMEGAT ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%u(:ncol,:) + call outfld('OMEGAU ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%v(:ncol,:) + call outfld('OMEGAV ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%q(:ncol,:,1) + call outfld('OMEGAQ ',ftem, pcols, lchnk ) + ftem(:ncol,:) = state%omega(:ncol,:)*state%omega(:ncol,:) + call outfld('OMGAOMGA',ftem, pcols, lchnk ) +! +! Output omega at 850 and 500 mb pressure levels +! + if (hist_fld_active('OMEGA1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%omega, p_surf) + call outfld('OMEGA1000', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%omega, p_surf) + call outfld('OMEGA975', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%omega, p_surf) + call outfld('OMEGA950', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%omega, p_surf) + call outfld('OMEGA925', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%omega, p_surf) + call outfld('OMEGA900', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%omega, p_surf) + call outfld('OMEGA850', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%omega, p_surf) + call outfld('OMEGA800', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%omega, p_surf) + call outfld('OMEGA700', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%omega, p_surf) + call outfld('OMEGA600', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%omega, p_surf) + call outfld('OMEGA500', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%omega, p_surf) + call outfld('OMEGA400', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%omega, p_surf) + call outfld('OMEGA300', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%omega, p_surf) + call outfld('OMEGA200', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGA100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%omega, p_surf) + call outfld('OMEGA100', p_surf, pcols, lchnk) + end if + if (hist_fld_active('OMEGABOT')) then + call outfld('OMEGABOT', state%omega(:,pver), pcols, lchnk) + end if +! +! Mass of q, by layer and vertically integrated +! + ftem(:ncol,:) = state%q(:ncol,:,1) * state%pdel(:ncol,:) * rga + call outfld ('MQ ',ftem ,pcols ,lchnk ) + + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TMQ ',ftem, pcols ,lchnk ) + +!CAS integrated vapor transport calculation + + !compute uq*dp/g and vq*dp/g + ftem4(:ncol,:) = state%q(:ncol,:,1) * state%u(:ncol,:) *state%pdel(:ncol,:) * rga + ftem5(:ncol,:) = state%q(:ncol,:,1) * state%v(:ncol,:) *state%pdel(:ncol,:) * rga + + !integrate each component + do k=2,pver + ftem4(:ncol,1) = ftem4(:ncol,1) + ftem4(:ncol,k) + ftem5(:ncol,1) = ftem5(:ncol,1) + ftem5(:ncol,k) + end do + !compute ivt + ftem(:ncol,1) = sqrt( ftem4(:ncol,1)**2 + ftem5(:ncol,1)**2) + + call outfld ('IVT ',ftem, pcols ,lchnk ) + + !just output uq*dp/g + ftem(:ncol,1) = ftem4(:ncol,1) + call outfld ('uIVT ',ftem, pcols ,lchnk ) + + !just output vq*dp/g + ftem(:ncol,1) = ftem5(:ncol,1) + call outfld ('vIVT ',ftem, pcols ,lchnk ) + +!CAS + + +! +! Mass of vertically integrated q flux +! + ftem(:ncol,:) = state%u(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TUQ ',ftem, pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*state%q(:ncol,:,1)*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TVQ ',ftem, pcols ,lchnk ) + +! +! Mass of vertically integrated MSE flux +! + ftem(:ncol,:) = state%u(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TUH ',ftem, pcols ,lchnk ) + + ftem(:ncol,:) = state%v(:ncol,:)*(state%s(:ncol,:)+latvap*state%q(:ncol,:,1))*state%pdel(:ncol,:)*rga + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('TVH ',ftem, pcols ,lchnk ) + + if (moist_physics) then + + ! Relative humidity + if (hist_fld_active('RELHUM')) then + call qsat(state%t(:ncol,:), state%pmid(:ncol,:), & + tem2(:ncol,:), ftem(:ncol,:)) + ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RELHUM ',ftem ,pcols ,lchnk ) + end if + + ! ftem defined above assuming RELHUM in one of the history tape + if (hist_fld_active('RH1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, ftem, p_surf) + call outfld('RH1000', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, ftem, p_surf) + call outfld('RH975', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, ftem, p_surf) + call outfld('RH950', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, ftem, p_surf) + call outfld('RH925', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, ftem, p_surf) + call outfld('RH900', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, ftem, p_surf) + call outfld('RH850', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, ftem, p_surf) + call outfld('RH800', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, ftem, p_surf) + call outfld('RH700', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, ftem, p_surf) + call outfld('RH600', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, ftem, p_surf) + call outfld('RH500', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, ftem, p_surf) + call outfld('RH400', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, ftem, p_surf) + call outfld('RH300', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, ftem, p_surf) + call outfld('RH200', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RH100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, ftem, p_surf) + call outfld('RH100', p_surf, pcols, lchnk) + end if + if (hist_fld_active('RHBOT')) then + call outfld('RHBOT', ftem(:,pver), pcols, lchnk) + end if + + if (hist_fld_active('RHW') .or. hist_fld_active('RHI') .or. hist_fld_active('RHCFMIP') ) then + + ! RH w.r.t liquid (water) + call qsat_water (state%t(:ncol,:), state%pmid(:ncol,:), & + esl(:ncol,:), ftem(:ncol,:)) + ftem(:ncol,:) = state%q(:ncol,:,1)/ftem(:ncol,:)*100._r8 + call outfld ('RHW ',ftem ,pcols ,lchnk ) + + ! Convert to RHI (ice) + do i=1,ncol + do k=1,pver + esi(i,k)=svp_ice(state%t(i,k)) + ftem1(i,k)=ftem(i,k)*esl(i,k)/esi(i,k) + end do + end do + call outfld ('RHI ',ftem1 ,pcols ,lchnk ) + + ! use temperature to decide if you populate with ftem (liquid, above 0 C) or ftem1 (ice, below 0 C) + + ftem2(:ncol,:)=ftem(:ncol,:) + + do i=1,ncol + do k=1,pver + if (state%t(i,k) .gt. 273) then + ftem2(i,k)=ftem(i,k) !!wrt water + else + ftem2(i,k)=ftem1(i,k) !!wrt ice + end if + end do + end do + + call outfld ('RHCFMIP ',ftem2 ,pcols ,lchnk ) + + end if + + end if +! +! Sea level pressure +! + if (present(psl) .or. hist_fld_active('PSL')) then + call cpslec (ncol, state%pmid, state%phis, state%ps, state%t,psl_tmp, gravit, rair) + call outfld ('PSL ',psl_tmp ,pcols, lchnk ) + if (present(psl)) then + psl(:ncol) = psl_tmp(:ncol) + end if + end if +! +! Output T,q,u,v fields on pressure surfaces +! + if (hist_fld_active('T975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%t, p_surf) + call outfld('T975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%t, p_surf) + call outfld('T950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%t, p_surf) + call outfld('T900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf) + call outfld('T850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%t, p_surf) + call outfld('T800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%t, p_surf) + call outfld('T600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%t, p_surf) + call outfld('T500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%t, p_surf) + call outfld('T400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%t, p_surf) + call outfld('T300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%t, p_surf) + call outfld('T200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%t, p_surf) + call outfld('T100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%q(1,1,1), p_surf) + call outfld('Q975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%q(1,1,1), p_surf) + call outfld('Q950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%q(1,1,1), p_surf) + call outfld('Q900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf) + call outfld('Q850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%q(1,1,1), p_surf) + call outfld('Q800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf) + call outfld('Q700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%q(1,1,1), p_surf) + call outfld('Q600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%q(1,1,1), p_surf) + call outfld('Q500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%q(1,1,1), p_surf) + call outfld('Q400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%q(1,1,1), p_surf) + call outfld('Q300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%q(1,1,1), p_surf) + call outfld('Q200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%q(1,1,1), p_surf) + call outfld('Q100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%q(1,1,1), p_surf) + call outfld('Q050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('Q010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%q(1,1,1), p_surf) + call outfld('Q010 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%u, p_surf) + call outfld('U1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%u, p_surf) + call outfld('U975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%u, p_surf) + call outfld('U950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%u, p_surf) + call outfld('U925 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%u, p_surf) + call outfld('U900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%u, p_surf) + call outfld('U850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%u, p_surf) + call outfld('U800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%u, p_surf) + call outfld('U700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%u, p_surf) + call outfld('U600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%u, p_surf) + call outfld('U500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%u, p_surf) + call outfld('U400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%u, p_surf) + call outfld('U300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%u, p_surf) + call outfld('U250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%u, p_surf) + call outfld('U200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%u, p_surf) + call outfld('U100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%u, p_surf) + call outfld('U050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('U010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%u, p_surf) + call outfld('U010 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V1000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%v, p_surf) + call outfld('V1000 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V975')) then + call vertinterp(ncol, pcols, pver, state%pmid, 97500._r8, state%v, p_surf) + call outfld('V975 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V950')) then + call vertinterp(ncol, pcols, pver, state%pmid, 95000._r8, state%v, p_surf) + call outfld('V950 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V925')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%v, p_surf) + call outfld('V925 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V900')) then + call vertinterp(ncol, pcols, pver, state%pmid, 90000._r8, state%v, p_surf) + call outfld('V900 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V850')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%v, p_surf) + call outfld('V850 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V800')) then + call vertinterp(ncol, pcols, pver, state%pmid, 80000._r8, state%v, p_surf) + call outfld('V800 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V700')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%v, p_surf) + call outfld('V700 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V600')) then + call vertinterp(ncol, pcols, pver, state%pmid, 60000._r8, state%v, p_surf) + call outfld('V600 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V500')) then + call vertinterp(ncol, pcols, pver, state%pmid, 50000._r8, state%v, p_surf) + call outfld('V500 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V400')) then + call vertinterp(ncol, pcols, pver, state%pmid, 40000._r8, state%v, p_surf) + call outfld('V400 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V300')) then + call vertinterp(ncol, pcols, pver, state%pmid, 30000._r8, state%v, p_surf) + call outfld('V300 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V250')) then + call vertinterp(ncol, pcols, pver, state%pmid, 25000._r8, state%v, p_surf) + call outfld('V250 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V200')) then + call vertinterp(ncol, pcols, pver, state%pmid, 20000._r8, state%v, p_surf) + call outfld('V200 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V100')) then + call vertinterp(ncol, pcols, pver, state%pmid, 10000._r8, state%v, p_surf) + call outfld('V100 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%v, p_surf) + call outfld('V050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('V010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%v, p_surf) + call outfld('V010 ', p_surf, pcols, lchnk ) + end if + + ftem(:ncol,:) = state%t(:ncol,:)*state%t(:ncol,:) + call outfld('TT ',ftem ,pcols ,lchnk ) +! +! Output U, V, T, Q, P and Z at bottom level +! + call outfld ('UBOT ', state%u(1,pver) , pcols, lchnk) + call outfld ('VBOT ', state%v(1,pver) , pcols, lchnk) + call outfld ('QBOT ', state%q(1,pver,1), pcols, lchnk) + call outfld ('ZBOT ', state%zm(1,pver) , pcols, lchnk) + +! Total energy of the atmospheric column for atmospheric heat storage calculations + + !! temporary variable to get surface geopotential in dimensions of (ncol,pver) + do k=1,pver + ftem1(:ncol,k)=state%phis(:ncol) !! surface geopotential in units (m2/s2) + end do + + !! calculate sum of sensible, kinetic, latent, and surface geopotential energy + !! E=CpT+PHIS+Lv*q+(0.5)*(u^2+v^2) + ftem(:ncol,:) = (cpair*state%t(:ncol,:) + ftem1(:ncol,:) + latvap*state%q(:ncol,:,1) + & + 0.5_r8*(state%u(:ncol,:)**2+state%v(:ncol,:)**2))*(state%pdel(:ncol,:)/gravit) + !! vertically integrate + do k=2,pver + ftem(:ncol,1) = ftem(:ncol,1) + ftem(:ncol,k) + end do + call outfld ('ATMEINT ',ftem(:ncol,1) ,pcols ,lchnk ) + +!! Boundary layer atmospheric stability, temperature, water vapor diagnostics + + if (hist_fld_active('T1000') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%t, p_surf_t1) + end if + + if (hist_fld_active('T925') .or. & + hist_fld_active('T9251000') .or. & + hist_fld_active('TH9251000') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%t, p_surf_t2) + end if + + if (hist_fld_active('Q1000') .or. & + hist_fld_active('THE9251000') .or. & + hist_fld_active('THE8501000') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 100000._r8, state%q(1,1,1), p_surf_q1) + end if + + if (hist_fld_active('Q925') .or. & + hist_fld_active('THE9251000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 92500._r8, state%q(1,1,1), p_surf_q2) + end if + + !!! at 1000 mb and 925 mb + if (hist_fld_active('T1000')) then + call outfld('T1000 ', p_surf_t1, pcols, lchnk ) + end if + + if (hist_fld_active('T925')) then + call outfld('T925 ', p_surf_t2, pcols, lchnk ) + end if + + if (hist_fld_active('Q1000')) then + call outfld('Q1000 ', p_surf_q1, pcols, lchnk ) + end if + + if (hist_fld_active('Q925')) then + call outfld('Q925 ', p_surf_q2, pcols, lchnk ) + end if + + if (hist_fld_active('T9251000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE9251000')) then + p_surf = (p_surf_t2*(1000.0_r8/925.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE9251000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T8501000') .or. & + hist_fld_active('TH8501000') .or. & + hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%t, p_surf_t2) + end if + + !!! at 1000 mb and 850 mb + if (hist_fld_active('T8501000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH8501000')) then + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE8501000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 85000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/850.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE8501000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T7001000') .or. & + hist_fld_active('TH7001000') .or. & + hist_fld_active('T700') .or. & + hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%t, p_surf_t2) + end if + + !!! at 700 mb + if (hist_fld_active('T700')) then + call outfld('T700 ', p_surf_t2, pcols, lchnk ) + end if + + !!! at 1000 mb and 700 mb + if (hist_fld_active('T7001000')) then + p_surf = p_surf_t2-p_surf_t1 + call outfld('T7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('TH7001000')) then + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)-(p_surf_t1*(1.0_r8)**cappa) + call outfld('TH7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('THE7001000')) then + call vertinterp(ncol, pcols, pver, state%pmid, 70000._r8, state%q(1,1,1), p_surf_q2) + p_surf = (p_surf_t2*(1000.0_r8/700.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q2)/(1004.0_r8*p_surf_t2))- & + (p_surf_t1*(1.0_r8)**cappa)*exp((2500000.0_r8*p_surf_q1)/(1004.0_r8*p_surf_t1)) + call outfld('THE7001000 ', p_surf, pcols, lchnk ) + end if + + if (hist_fld_active('T050')) then + call vertinterp(ncol, pcols, pver, state%pmid, 5000._r8, state%t, p_surf) + call outfld('T050 ', p_surf, pcols, lchnk ) + end if + if (hist_fld_active('T010')) then + call vertinterp(ncol, pcols, pver, state%pmid, 1000._r8, state%t, p_surf) + call outfld('T010 ', p_surf, pcols, lchnk ) + end if + + + !--------------------------------------------------------- + ! WACCM tidal diagnostics + !--------------------------------------------------------- + + if (chem_is('waccm_ghg') .or. chem_is('waccm_mozart') .or. chem_is('waccm_mozart_mam3')) then + + call tidal_diag_write(state) + + endif + + return + end subroutine diag_phys_writeout +!=============================================================================== + +subroutine diag_conv(state, ztodt, pbuf) + +!----------------------------------------------------------------------- +! +! Output diagnostics associated with all convective processes. +! +!----------------------------------------------------------------------- + use physconst, only: cpair + use tidal_diag, only: get_tidal_coeffs + +! Arguments: + + real(r8), intent(in) :: ztodt ! timestep for computing physics tendencies + type(physics_state), intent(in) :: state + type(physics_buffer_desc), pointer :: pbuf(:) + +! convective precipitation variables + real(r8), pointer :: prec_dp(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_dp(:) ! snow from ZM convection + real(r8), pointer :: prec_sh(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_sh(:) ! snow from Hack convection + real(r8), pointer :: prec_sed(:) ! total precipitation from ZM convection + real(r8), pointer :: snow_sed(:) ! snow from ZM convection + real(r8), pointer :: prec_pcw(:) ! total precipitation from Hack convection + real(r8), pointer :: snow_pcw(:) ! snow from Hack convection + +! Local variables: + + integer :: i, k, m, lchnk, ncol + + real(r8) :: rtdt + + real(r8):: precc(pcols) ! convective precip rate + real(r8):: precl(pcols) ! stratiform precip rate + real(r8):: snowc(pcols) ! convective snow rate + real(r8):: snowl(pcols) ! stratiform snow rate + real(r8):: prect(pcols) ! total (conv+large scale) precip rate + real(r8) :: dcoef(4) ! for tidal component of T tend + + lchnk = state%lchnk + ncol = state%ncol + + rtdt = 1._r8/ztodt + + call pbuf_get_field(pbuf, prec_dp_idx, prec_dp) + call pbuf_get_field(pbuf, snow_dp_idx, snow_dp) + call pbuf_get_field(pbuf, prec_sh_idx, prec_sh) + call pbuf_get_field(pbuf, snow_sh_idx, snow_sh) + call pbuf_get_field(pbuf, prec_sed_idx, prec_sed) + call pbuf_get_field(pbuf, snow_sed_idx, snow_sed) + call pbuf_get_field(pbuf, prec_pcw_idx, prec_pcw) + call pbuf_get_field(pbuf, snow_pcw_idx, snow_pcw) + +! Precipitation rates (multi-process) + precc(:ncol) = prec_dp(:ncol) + prec_sh(:ncol) + precl(:ncol) = prec_sed(:ncol) + prec_pcw(:ncol) + snowc(:ncol) = snow_dp(:ncol) + snow_sh(:ncol) + snowl(:ncol) = snow_sed(:ncol) + snow_pcw(:ncol) + prect(:ncol) = precc(:ncol) + precl(:ncol) + + call outfld('PRECC ', precc, pcols, lchnk ) + call outfld('PRECL ', precl, pcols, lchnk ) + call outfld('PREC_PCW', prec_pcw,pcols ,lchnk ) + call outfld('PREC_zmc', prec_dp ,pcols ,lchnk ) + call outfld('PRECSC ', snowc, pcols, lchnk ) + call outfld('PRECSL ', snowl, pcols, lchnk ) + call outfld('PRECT ', prect, pcols, lchnk ) + call outfld('PRECTMX ', prect, pcols, lchnk ) + + call outfld('PRECLav ', precl, pcols, lchnk ) + call outfld('PRECCav ', precc, pcols, lchnk ) + +#if ( defined BFB_CAM_SCAM_IOP ) + call outfld('Prec ' , prect, pcols, lchnk ) +#endif + + ! Total convection tendencies. + + do k = 1, pver + do i = 1, ncol + dtcond(i,k,lchnk) = (state%s(i,k) - dtcond(i,k,lchnk))*rtdt / cpair + end do + end do + call outfld('DTCOND ', dtcond(:,:,lchnk), pcols, lchnk) + + ! output tidal coefficients + call get_tidal_coeffs( dcoef ) + call outfld( 'DTCOND_24_SIN', dtcond(:ncol,:,lchnk)*dcoef(1), ncol, lchnk ) + call outfld( 'DTCOND_24_COS', dtcond(:ncol,:,lchnk)*dcoef(2), ncol, lchnk ) + call outfld( 'DTCOND_12_SIN', dtcond(:ncol,:,lchnk)*dcoef(3), ncol, lchnk ) + call outfld( 'DTCOND_12_COS', dtcond(:ncol,:,lchnk)*dcoef(4), ncol, lchnk ) + + do m = 1, dqcond_num + if ( cnst_cam_outfld(m) ) then + do k = 1, pver + do i = 1, ncol + dqcond(m)%cnst(i,k,lchnk) = (state%q(i,k,m) - dqcond(m)%cnst(i,k,lchnk))*rtdt + end do + end do + call outfld(dcconnam(m), dqcond(m)%cnst(:,:,lchnk), pcols, lchnk) + end if + end do + +end subroutine diag_conv + +!=============================================================================== + +subroutine diag_surf (cam_in, cam_out, ps, trefmxav, trefmnav ) + +!----------------------------------------------------------------------- +! +! Purpose: record surface diagnostics +! +!----------------------------------------------------------------------- + + use time_manager, only: is_end_curr_day + use co2_cycle, only: c_i, co2_transport + use constituents, only: sflxnam + +!----------------------------------------------------------------------- +! +! Input arguments +! + type(cam_in_t), intent(in) :: cam_in + type(cam_out_t), intent(in) :: cam_out + + real(r8), intent(inout) :: trefmnav(pcols) ! daily minimum tref + real(r8), intent(inout) :: trefmxav(pcols) ! daily maximum tref + + real(r8), intent(in) :: ps(pcols) ! Surface pressure. +! +!---------------------------Local workspace----------------------------- +! + integer :: i, k, m ! indexes + integer :: lchnk ! chunk identifier + integer :: ncol ! longitude dimension + real(r8) tem2(pcols) ! temporary workspace + real(r8) ftem(pcols) ! temporary workspace +! +!----------------------------------------------------------------------- +! + lchnk = cam_in%lchnk + ncol = cam_in%ncol + + call outfld('SHFLX', cam_in%shf, pcols, lchnk) + call outfld('LHFLX', cam_in%lhf, pcols, lchnk) + call outfld('QFLX', cam_in%cflx(1,1), pcols, lchnk) + + call outfld('TAUX', cam_in%wsx, pcols, lchnk) + call outfld('TAUY', cam_in%wsy, pcols, lchnk) + call outfld('TREFHT ', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMX', cam_in%tref, pcols, lchnk) + call outfld('TREFHTMN', cam_in%tref, pcols, lchnk) + call outfld('QREFHT', cam_in%qref, pcols, lchnk) + call outfld('U10', cam_in%u10, pcols, lchnk) +! +! Calculate and output reference height RH (RHREFHT) + + call qsat(cam_in%tref(:ncol), ps(:ncol), tem2(:ncol), ftem(:ncol)) + ftem(:ncol) = cam_in%qref(:ncol)/ftem(:ncol)*100._r8 + + + call outfld('RHREFHT', ftem, pcols, lchnk) + + +#if (defined BFB_CAM_SCAM_IOP ) + call outfld('shflx ',cam_in%shf, pcols, lchnk) + call outfld('lhflx ',cam_in%lhf, pcols, lchnk) + call outfld('trefht ',cam_in%tref, pcols, lchnk) +#endif +! +! Ouput ocn and ice fractions +! + call outfld('LANDFRAC', cam_in%landfrac, pcols, lchnk) + call outfld('ICEFRAC', cam_in%icefrac, pcols, lchnk) + call outfld('OCNFRAC', cam_in%ocnfrac, pcols, lchnk) +! +! Compute daily minimum and maximum of TREF +! + do i = 1,ncol + trefmxav(i) = max(cam_in%tref(i),trefmxav(i)) + trefmnav(i) = min(cam_in%tref(i),trefmnav(i)) + end do + if (is_end_curr_day()) then + call outfld('TREFMXAV', trefmxav,pcols, lchnk ) + call outfld('TREFMNAV', trefmnav,pcols, lchnk ) + trefmxav(:ncol) = -1.0e36_r8 + trefmnav(:ncol) = 1.0e36_r8 + endif + + call outfld('TBOT', cam_out%tbot, pcols, lchnk) + call outfld('TS', cam_in%ts, pcols, lchnk) + call outfld('TSMN', cam_in%ts, pcols, lchnk) + call outfld('TSMX', cam_in%ts, pcols, lchnk) + call outfld('SNOWHLND', cam_in%snowhland, pcols, lchnk) + call outfld('SNOWHICE', cam_in%snowhice, pcols, lchnk) + call outfld('ASDIR', cam_in%asdir, pcols, lchnk) + call outfld('ASDIF', cam_in%asdif, pcols, lchnk) + call outfld('ALDIR', cam_in%aldir, pcols, lchnk) + call outfld('ALDIF', cam_in%aldif, pcols, lchnk) + call outfld('SST', cam_in%sst, pcols, lchnk) + + if (co2_transport()) then + do m = 1,4 + call outfld(sflxnam(c_i(m)), cam_in%cflx(:,c_i(m)), pcols, lchnk) + end do + end if + +end subroutine diag_surf + +!=============================================================================== + +subroutine diag_export(cam_out) + +!----------------------------------------------------------------------- +! +! Purpose: Write export state to history file +! +!----------------------------------------------------------------------- + + ! arguments + type(cam_out_t), intent(inout) :: cam_out + + ! Local variables: + integer :: lchnk ! chunk identifier + logical :: atm_dep_flux ! true ==> sending deposition fluxes to coupler. + ! Otherwise, set them to zero. + !----------------------------------------------------------------------- + + lchnk = cam_out%lchnk + + call phys_getopts(atm_dep_flux_out=atm_dep_flux) + + if (.not. atm_dep_flux) then + ! set the fluxes to zero before outfld and sending them to the + ! coupler + cam_out%bcphiwet = 0.0_r8 + cam_out%bcphidry = 0.0_r8 + cam_out%bcphodry = 0.0_r8 + cam_out%ocphiwet = 0.0_r8 + cam_out%ocphidry = 0.0_r8 + cam_out%ocphodry = 0.0_r8 + cam_out%dstwet1 = 0.0_r8 + cam_out%dstdry1 = 0.0_r8 + cam_out%dstwet2 = 0.0_r8 + cam_out%dstdry2 = 0.0_r8 + cam_out%dstwet3 = 0.0_r8 + cam_out%dstdry3 = 0.0_r8 + cam_out%dstwet4 = 0.0_r8 + cam_out%dstdry4 = 0.0_r8 + end if + + call outfld('a2x_BCPHIWET', cam_out%bcphiwet, pcols, lchnk) + call outfld('a2x_BCPHIDRY', cam_out%bcphidry, pcols, lchnk) + call outfld('a2x_BCPHODRY', cam_out%bcphodry, pcols, lchnk) + call outfld('a2x_OCPHIWET', cam_out%ocphiwet, pcols, lchnk) + call outfld('a2x_OCPHIDRY', cam_out%ocphidry, pcols, lchnk) + call outfld('a2x_OCPHODRY', cam_out%ocphodry, pcols, lchnk) + call outfld('a2x_DSTWET1', cam_out%dstwet1, pcols, lchnk) + call outfld('a2x_DSTDRY1', cam_out%dstdry1, pcols, lchnk) + call outfld('a2x_DSTWET2', cam_out%dstwet2, pcols, lchnk) + call outfld('a2x_DSTDRY2', cam_out%dstdry2, pcols, lchnk) + call outfld('a2x_DSTWET3', cam_out%dstwet3, pcols, lchnk) + call outfld('a2x_DSTDRY3', cam_out%dstdry3, pcols, lchnk) + call outfld('a2x_DSTWET4', cam_out%dstwet4, pcols, lchnk) + call outfld('a2x_DSTDRY4', cam_out%dstdry4, pcols, lchnk) + +end subroutine diag_export + +!####################################################################### + + subroutine diag_physvar_ic (lchnk, pbuf, cam_out, cam_in) +! +!--------------------------------------------- +! +! Purpose: record physics variables on IC file +! +!--------------------------------------------- +! + +! +! Arguments +! + integer , intent(in) :: lchnk ! chunk identifier + type(physics_buffer_desc), pointer :: pbuf(:) + + type(cam_out_t), intent(inout) :: cam_out + type(cam_in_t), intent(inout) :: cam_in +! +!---------------------------Local workspace----------------------------- +! + integer :: k ! indices + integer :: itim_old ! indices + + real(r8), pointer, dimension(:,:) :: cwat_var + real(r8), pointer, dimension(:,:) :: conv_var_3d + real(r8), pointer, dimension(: ) :: conv_var_2d + real(r8), pointer :: tpert(:), pblh(:), qpert(:) +! +!----------------------------------------------------------------------- +! + if( write_inithist() ) then + + ! + ! Associate pointers with physics buffer fields + ! + itim_old = pbuf_old_tim_idx() + + if (qcwat_idx > 0) then + call pbuf_get_field(pbuf, qcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('QCWAT&IC ',cwat_var, pcols,lchnk) + end if + + if (tcwat_idx > 0) then + call pbuf_get_field(pbuf, tcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('TCWAT&IC ',cwat_var, pcols,lchnk) + end if + + if (lcwat_idx > 0) then + call pbuf_get_field(pbuf, lcwat_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('LCWAT&IC ',cwat_var, pcols,lchnk) + end if + + call pbuf_get_field(pbuf, cld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CLOUD&IC ',cwat_var, pcols,lchnk) + + call pbuf_get_field(pbuf, concld_idx, cwat_var, start=(/1,1,itim_old/), kount=(/pcols,pver,1/) ) + call outfld('CONCLD&IC ',cwat_var, pcols,lchnk) + + call pbuf_get_field(pbuf, tke_idx, conv_var_3d) + call outfld('TKE&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, kvm_idx, conv_var_3d) + call outfld('KVM&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, kvh_idx, conv_var_3d) + call outfld('KVH&IC ',conv_var_3d, pcols,lchnk) + + call pbuf_get_field(pbuf, cush_idx, conv_var_2d ,(/1,itim_old/), (/pcols,1/)) + call outfld('CUSH&IC ',conv_var_2d, pcols,lchnk) + + if (qpert_idx > 0) then + call pbuf_get_field(pbuf, qpert_idx, qpert) + call outfld('QPERT&IC ', qpert, pcols, lchnk) + end if + + call pbuf_get_field(pbuf, pblh_idx, pblh) + call outfld('PBLH&IC ', pblh, pcols, lchnk) + + call pbuf_get_field(pbuf, tpert_idx, tpert) + call outfld('TPERT&IC ', tpert, pcols, lchnk) + + ! The following is only needed for cam-csim + call outfld('TBOT&IC ', cam_out%tbot, pcols, lchnk) + end if + + end subroutine diag_physvar_ic + + +!####################################################################### + +subroutine diag_phys_tend_writeout(state, pbuf, tend, ztodt, tmp_q, tmp_cldliq, tmp_cldice, & + tmp_t, qini, cldliqini, cldiceini) + + !--------------------------------------------------------------- + ! + ! Purpose: Dump physics tendencies for moisture and temperature + ! + !--------------------------------------------------------------- + + use check_energy, only: check_energy_get_integrals + use physconst, only: cpair + + ! Arguments + + type(physics_state), intent(in ) :: state + + type(physics_buffer_desc), pointer :: pbuf(:) + type(physics_tend ), intent(in ) :: tend + real(r8) , intent(in ) :: ztodt ! physics timestep + real(r8) , intent(inout) :: tmp_q (pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldliq(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_cldice(pcols,pver) ! As input, holds pre-adjusted tracers (FV) + real(r8) , intent(inout) :: tmp_t (pcols,pver) ! holds last physics_updated T (FV) + real(r8) , intent(in ) :: qini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldliqini (pcols,pver) ! tracer fields at beginning of physics + real(r8) , intent(in ) :: cldiceini (pcols,pver) ! tracer fields at beginning of physics + + !---------------------------Local workspace----------------------------- + + integer :: m ! constituent index + integer :: lchnk ! chunk index + integer :: ncol ! number of columns in chunk + real(r8) :: ftem2(pcols ) ! Temporary workspace for outfld variables + real(r8) :: ftem3(pcols,pver) ! Temporary workspace for outfld variables + real(r8) :: rtdt + real(r8) :: heat_glob ! global energy integral (FV only) + integer :: ixcldice, ixcldliq! constituent indices for cloud liquid and ice water. + ! CAM pointers to get variables from the physics buffer + real(r8), pointer, dimension(:,:) :: t_ttend + integer :: itim_old + + !----------------------------------------------------------------------- + + lchnk = state%lchnk + ncol = state%ncol + rtdt = 1._r8/ztodt + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + + ! Dump out post-physics state (FV only) + + if (dycore_is('LR') .or. dycore_is('SE')) then + tmp_t(:ncol,:pver) = (tmp_t(:ncol,:pver) - state%t(:ncol,:pver))/ztodt + call outfld('PTTEND_RESID', tmp_t, pcols, lchnk ) + end if + call outfld('TAP', state%t, pcols, lchnk ) + call outfld('UAP', state%u, pcols, lchnk ) + call outfld('VAP', state%v, pcols, lchnk ) + + if ( cnst_cam_outfld( 1) ) call outfld (apcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (apcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (apcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) + + ! T-tendency due to FV Energy fixer (remove from total physics tendency diagnostic) + + if (dycore_is('LR') .or. dycore_is('SE')) then + call check_energy_get_integrals( heat_glob_out=heat_glob ) + ftem2(:ncol) = heat_glob/cpair + call outfld('TFIX', ftem2, pcols, lchnk ) + ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) - heat_glob/cpair + else + ftem3(:ncol,:pver) = tend%dtdt(:ncol,:pver) + end if + + ! Total physics tendency for Temperature + + call outfld('PTTEND',ftem3, pcols, lchnk ) + + ! Tendency for dry mass adjustment of q (valid for FV only) + + if (dycore_is('LR')) then + tmp_q (:ncol,:pver) = (state%q(:ncol,:pver, 1) - tmp_q (:ncol,:pver))*rtdt + tmp_cldliq(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - tmp_cldliq(:ncol,:pver))*rtdt + tmp_cldice(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - tmp_cldice(:ncol,:pver))*rtdt + if ( cnst_cam_outfld( 1) ) call outfld (dmetendnam( 1), tmp_q , pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (dmetendnam(ixcldliq), tmp_cldliq, pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (dmetendnam(ixcldice), tmp_cldice, pcols, lchnk) + end if + + ! Total physics tendency for moisture and other tracers + + if ( cnst_cam_outfld( 1) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver, 1) - qini (:ncol,:pver) )*rtdt + call outfld (ptendnam( 1), ftem3, pcols, lchnk) + end if + if ( cnst_cam_outfld(ixcldliq) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldliq) - cldliqini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldliq), ftem3, pcols, lchnk) + end if + if ( cnst_cam_outfld(ixcldice) ) then + ftem3(:ncol,:pver) = (state%q(:ncol,:pver,ixcldice) - cldiceini(:ncol,:pver) )*rtdt + call outfld (ptendnam(ixcldice), ftem3, pcols, lchnk) + end if + + ! Total (physics+dynamics, everything!) tendency for Temperature + + !! get temperature stored in physics buffer + itim_old = pbuf_old_tim_idx() + call pbuf_get_field(pbuf, t_ttend_idx, t_ttend, start=(/1,1,itim_old/), kount=(/pcols,pver,1/)) + + !! calculate and outfld the total temperature tendency + ftem3(:ncol,:) = (state%t(:ncol,:) - t_ttend(:ncol,:))/ztodt + call outfld('TTEND_TOT', ftem3, pcols, lchnk) + + !! update physics buffer with this time-step's temperature + t_ttend(:ncol,:) = state%t(:ncol,:) + +end subroutine diag_phys_tend_writeout + +!####################################################################### + + subroutine diag_state_b4_phys_write (state) +! +!--------------------------------------------------------------- +! +! Purpose: Dump state just prior to executing physics +! +!--------------------------------------------------------------- +! +! Arguments +! + type(physics_state), intent(in) :: state +! +!---------------------------Local workspace----------------------------- +! + integer :: ixcldice, ixcldliq ! constituent indices for cloud liquid and ice water. + integer :: lchnk ! chunk index +! +!----------------------------------------------------------------------- +! + lchnk = state%lchnk + + call cnst_get_ind('CLDLIQ', ixcldliq) + call cnst_get_ind('CLDICE', ixcldice) + call outfld('TBP', state%t, pcols, lchnk ) + if ( cnst_cam_outfld( 1) ) call outfld (bpcnst( 1), state%q(1,1, 1), pcols, lchnk) + if ( cnst_cam_outfld(ixcldliq) ) call outfld (bpcnst(ixcldliq), state%q(1,1,ixcldliq), pcols, lchnk) + if ( cnst_cam_outfld(ixcldice) ) call outfld (bpcnst(ixcldice), state%q(1,1,ixcldice), pcols, lchnk) + + end subroutine diag_state_b4_phys_write + +end module cam_diagnostics diff --git a/SourceMods/src.cam/safe/srcMods.tar b/SourceMods/src.cam/safe/srcMods.tar new file mode 100644 index 0000000..07910ed Binary files /dev/null and b/SourceMods/src.cam/safe/srcMods.tar differ diff --git a/SourceMods/src.cam/safe/tropopause.F90 b/SourceMods/src.cam/safe/tropopause.F90 new file mode 100755 index 0000000..ceb5c2e --- /dev/null +++ b/SourceMods/src.cam/safe/tropopause.F90 @@ -0,0 +1,1516 @@ + +! This module is used to diagnose the location of the tropopause. Multiple +! algorithms are provided, some of which may not be able to identify a +! tropopause in all situations. To handle these cases, an analytic +! definition and a climatology are provided that can be used to fill in +! when the original algorithm fails. The tropopause temperature and +! pressure are determined and can be output to the history file. +! +! These routines are based upon code in the WACCM chemistry module +! including mo_tropoause.F90 and llnl_set_chem_trop.F90. The code +! for the Reichler et al. [2003] algorithm is from: +! +! http://www.gfdl.noaa.gov/~tjr/TROPO/tropocode.htm +! +! Author: Charles Bardeen +! Created: April, 2009 + +module tropopause + !--------------------------------------------------------------- + ! ... variables for the tropopause module + !--------------------------------------------------------------- + + use shr_kind_mod, only : r8 => shr_kind_r8 + use shr_const_mod, only : pi => shr_const_pi + use ppgrid, only : pcols, pver, begchunk, endchunk + use cam_abortutils, only : endrun + use cam_logfile, only : iulog + use cam_history_support, only : fillvalue + use physics_types, only : physics_state + use physconst, only : cappa, rair, gravit + use spmd_utils, only : masterproc + + implicit none + + private + + public :: tropopause_readnl, tropopause_init, tropopause_find, tropopause_output + public :: tropopause_findChemTrop + public :: TROP_ALG_NONE, TROP_ALG_ANALYTIC, TROP_ALG_CLIMATE + public :: TROP_ALG_STOBIE, TROP_ALG_HYBSTOB, TROP_ALG_TWMO, TROP_ALG_WMO + public :: TROP_ALG_CPP + public :: NOTFOUND + + save + + ! These parameters define and enumeration to be used to define the primary + ! and backup algorithms to be used with the tropopause_find() method. The + ! backup algorithm is meant to provide a solution when the primary algorithm + ! fail. The algorithms that can't fail are: TROP_ALG_ANALYTIC, TROP_ALG_CLIMATE + ! and TROP_ALG_STOBIE. + integer, parameter :: TROP_ALG_NONE = 1 ! Don't evaluate + integer, parameter :: TROP_ALG_ANALYTIC = 2 ! Analytic Expression + integer, parameter :: TROP_ALG_CLIMATE = 3 ! Climatology + integer, parameter :: TROP_ALG_STOBIE = 4 ! Stobie Algorithm + integer, parameter :: TROP_ALG_TWMO = 5 ! WMO Definition, Reichler et al. [2003] + integer, parameter :: TROP_ALG_WMO = 6 ! WMO Definition + integer, parameter :: TROP_ALG_HYBSTOB = 7 ! Hybrid Stobie Algorithm + integer, parameter :: TROP_ALG_CPP = 8 ! Cold Point Parabolic + + integer, parameter :: TROP_NALG = 8 ! Number of Algorithms + character,parameter :: TROP_LETTER(TROP_NALG) = (/ ' ', 'A', 'C', 'S', 'T', 'W', 'H', 'F' /) + ! unique identifier for output, don't use P + + ! These variables should probably be controlled by namelist entries. + logical ,parameter :: output_all = .False. ! output tropopause info from all algorithms + integer ,parameter :: default_primary = TROP_ALG_TWMO ! default primary algorithm + integer ,parameter :: default_backup = TROP_ALG_CLIMATE ! default backup algorithm + + ! Namelist variables + character(len=256) :: tropopause_climo_file = 'trop_climo' ! absolute filepath of climatology file + + ! These variables are used to store the climatology data. + real(r8) :: days(12) ! days in the climatology + real(r8), pointer :: tropp_p_loc(:,:,:) ! climatological tropopause pressures + + integer, parameter :: NOTFOUND = -1 + + real(r8),parameter :: ALPHA = 0.03_r8 + + ! physical constants + ! These constants are set in module variables rather than as parameters + ! to support the aquaplanet mode in which the constants have values determined + ! by the experiment protocol + real(r8) :: cnst_kap ! = cappa + real(r8) :: cnst_faktor ! = -gravit/rair + real(r8) :: cnst_ka1 ! = cnst_kap - 1._r8 + +!================================================================================================ +contains +!================================================================================================ + + ! Read namelist variables. + subroutine tropopause_readnl(nlfile) + + use namelist_utils, only: find_group_name + use units, only: getunit, freeunit + use mpishorthand + + character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input + + ! Local variables + integer :: unitn, ierr + character(len=*), parameter :: subname = 'tropopause_readnl' + + namelist /tropopause_nl/ tropopause_climo_file + !----------------------------------------------------------------------------- + + if (masterproc) then + unitn = getunit() + open( unitn, file=trim(nlfile), status='old' ) + call find_group_name(unitn, 'tropopause_nl', status=ierr) + if (ierr == 0) then + read(unitn, tropopause_nl, iostat=ierr) + if (ierr /= 0) then + call endrun(subname // ':: ERROR reading namelist') + end if + end if + close(unitn) + call freeunit(unitn) + end if + +#ifdef SPMD + ! Broadcast namelist variables + call mpibcast(tropopause_climo_file, len(tropopause_climo_file), mpichar, 0, mpicom) +#endif + + end subroutine tropopause_readnl + + + ! This routine is called during intialization and must be called before the + ! other methods in this module can be used. Its main tasks are to read in the + ! climatology from a file and to define the output fields. Much of this code + ! is taken from mo_tropopause. + subroutine tropopause_init() + + + use ppgrid, only: pver + use cam_pio_utils, only: + use cam_history, only: addfld, horiz_only, add_default + + + implicit none + + ! define physical constants + cnst_kap = cappa + cnst_faktor = -gravit/rair + cnst_ka1 = cnst_kap - 1._r8 + + ! Define the output fields. + call addfld('TROP_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure', flag_xyfill=.True.) + call addfld('TROP_T', horiz_only, 'A', 'K', 'Tropopause Temperature', flag_xyfill=.True.) + call addfld('TROP_Z', horiz_only, 'A', 'm', 'Tropopause Height', flag_xyfill=.True.) + call addfld('TROP_DZ', (/ 'lev' /), 'A', 'm', 'Relative Tropopause Height') + call addfld('TROP_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Probabilty') + call addfld('TROP_FD', horiz_only, 'A', 'probability', 'Tropopause Found') + + call addfld('TROPP_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (primary)', flag_xyfill=.True.) + call addfld('TROPP_T', horiz_only, 'A', 'K', 'Tropopause Temperature (primary)', flag_xyfill=.True.) + call addfld('TROPP_Z', horiz_only, 'A', 'm', 'Tropopause Height (primary)', flag_xyfill=.True.) + call addfld('TROPP_DZ', (/ 'lev' /), 'A', 'm', 'Relalive Tropopause Height (primary)') + call addfld('TROPP_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (primary)') + call addfld('TROPP_FD', horiz_only, 'A', 'probability', 'Tropopause Found (primary)') + +!! start nanr for Rich + call addfld('TROPF_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (cold point)', flag_xyfill=.True.) + call addfld('TROPF_T', horiz_only, 'A', 'K', 'Tropopause Temperature (cold point)', flag_xyfill=.True.) + call addfld('TROPF_Z', horiz_only, 'A', 'm', 'Tropopause Height (cold point)', flag_xyfill=.True.) + call addfld('TROPF_DZ', (/ 'lev' /), 'A', 'm', 'Relative Tropopause Height (cold point)', flag_xyfill=.True.) + call addfld('TROPF_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (cold point)') + call addfld('TROPF_FD', horiz_only, 'A', 'probability', 'Tropopause Found (cold point)') +!! end nanr + + call addfld( 'hstobie_trop', (/ 'lev' /), 'I', 'fraction of model time', 'Lowest level with stratospheric chemsitry' ) + call addfld( 'hstobie_linoz', (/ 'lev' /), 'I', 'fraction of model time', 'Lowest possible Linoz level' ) + call addfld( 'hstobie_tropop', (/ 'lev' /), 'I', 'fraction of model time', & + 'Troposphere boundary calculated in chemistry' ) + + ! If requested, be prepared to output results from all of the methods. + if (output_all) then + call addfld('TROPA_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (analytic)', flag_xyfill=.True.) + call addfld('TROPA_T', horiz_only, 'A', 'K', 'Tropopause Temperature (analytic)', flag_xyfill=.True.) + call addfld('TROPA_Z', horiz_only, 'A', 'm', 'Tropopause Height (analytic)', flag_xyfill=.True.) + call addfld('TROPA_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (analytic)') + call addfld('TROPA_FD', horiz_only, 'A', 'probability', 'Tropopause Found (analytic)') + + call addfld('TROPC_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (climatology)', flag_xyfill=.True.) + call addfld('TROPC_T', horiz_only, 'A', 'K', 'Tropopause Temperature (climatology)', flag_xyfill=.True.) + call addfld('TROPC_Z', horiz_only, 'A', 'm', 'Tropopause Height (climatology)', flag_xyfill=.True.) + call addfld('TROPC_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (climatology)') + call addfld('TROPC_FD', horiz_only, 'A', 'probability', 'Tropopause Found (climatology)') + + call addfld('TROPS_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (stobie)', flag_xyfill=.True.) + call addfld('TROPS_T', horiz_only, 'A', 'K', 'Tropopause Temperature (stobie)', flag_xyfill=.True.) + call addfld('TROPS_Z', horiz_only, 'A', 'm', 'Tropopause Height (stobie)', flag_xyfill=.True.) + call addfld('TROPS_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (stobie)') + call addfld('TROPS_FD', horiz_only, 'A', 'probability', 'Tropopause Found (stobie)') + + call addfld('TROPT_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (twmo)', flag_xyfill=.True.) + call addfld('TROPT_T', horiz_only, 'A', 'K', 'Tropopause Temperature (twmo)', flag_xyfill=.True.) + call addfld('TROPT_Z', horiz_only, 'A', 'm', 'Tropopause Height (twmo)', flag_xyfill=.True.) + call addfld('TROPT_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (twmo)') + call addfld('TROPT_FD', horiz_only, 'A', 'probability', 'Tropopause Found (twmo)') + + call addfld('TROPW_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (WMO)', flag_xyfill=.True.) + call addfld('TROPW_T', horiz_only, 'A', 'K', 'Tropopause Temperature (WMO)', flag_xyfill=.True.) + call addfld('TROPW_Z', horiz_only, 'A', 'm', 'Tropopause Height (WMO)', flag_xyfill=.True.) + call addfld('TROPW_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (WMO)') + call addfld('TROPW_FD', horiz_only, 'A', 'probability', 'Tropopause Found (WMO)') + + call addfld('TROPH_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_T', horiz_only, 'A', 'K', 'Tropopause Temperature (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_Z', horiz_only, 'A', 'm', 'Tropopause Height (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (Hybrid Stobie)') + call addfld('TROPH_FD', horiz_only, 'A', 'probability', 'Tropopause Found (Hybrid Stobie)') + end if + + call add_default('TROP_P', 1, ' ') + call add_default('TROP_T', 1, ' ') + call add_default('hstobie_linoz', 1, ' ') + + call tropopause_read_file() + + + end subroutine tropopause_init + + + subroutine tropopause_read_file + !------------------------------------------------------------------ + ! ... initialize upper boundary values + !------------------------------------------------------------------ + use interpolate_data, only : lininterp_init, lininterp, interp_type, lininterp_finish + use dyn_grid, only : get_dyn_grid_parm + use phys_grid, only : get_ncols_p, get_rlat_all_p, get_rlon_all_p + use ioFileMod, only : getfil + use time_manager, only : get_calday + use physconst, only : pi + use cam_pio_utils, only: cam_pio_openfile + use pio, only : file_desc_t, var_desc_t, pio_inq_dimid, pio_inq_dimlen, & + pio_inq_varid, pio_get_var, pio_closefile, pio_nowrite + + !------------------------------------------------------------------ + ! ... local variables + !------------------------------------------------------------------ + integer :: i, j, n + integer :: ierr + type(file_desc_t) :: pio_id + integer :: dimid + type(var_desc_t) :: vid + integer :: nlon, nlat, ntimes + integer :: start(3) + integer :: count(3) + integer, parameter :: dates(12) = (/ 116, 214, 316, 415, 516, 615, & + 716, 816, 915, 1016, 1115, 1216 /) + integer :: plon, plat + type(interp_type) :: lon_wgts, lat_wgts + real(r8), allocatable :: tropp_p_in(:,:,:) + real(r8), allocatable :: lat(:) + real(r8), allocatable :: lon(:) + real(r8) :: to_lats(pcols), to_lons(pcols) + real(r8), parameter :: d2r=pi/180._r8, zero=0._r8, twopi=pi*2._r8 + character(len=256) :: locfn + integer :: c, ncols + + + plon = get_dyn_grid_parm('plon') + plat = get_dyn_grid_parm('plat') + + + !----------------------------------------------------------------------- + ! ... open netcdf file + !----------------------------------------------------------------------- + call getfil (tropopause_climo_file, locfn, 0) + call cam_pio_openfile(pio_id, trim(locfn), PIO_NOWRITE) + + !----------------------------------------------------------------------- + ! ... get time dimension + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'time', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, ntimes ) + if( ntimes /= 12 )then + write(iulog,*) 'tropopause_init: number of months = ',ntimes,'; expecting 12' + call endrun + end if + !----------------------------------------------------------------------- + ! ... get latitudes + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'lat', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, nlat ) + allocate( lat(nlat), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: lat allocation error = ',ierr + call endrun + end if + ierr = pio_inq_varid( pio_id, 'lat', vid ) + ierr = pio_get_var( pio_id, vid, lat ) + lat(:nlat) = lat(:nlat) * d2r + !----------------------------------------------------------------------- + ! ... get longitudes + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'lon', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, nlon ) + allocate( lon(nlon), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: lon allocation error = ',ierr + call endrun + end if + ierr = pio_inq_varid( pio_id, 'lon', vid ) + ierr = pio_get_var( pio_id, vid, lon ) + lon(:nlon) = lon(:nlon) * d2r + + !------------------------------------------------------------------ + ! ... allocate arrays + !------------------------------------------------------------------ + allocate( tropp_p_in(nlon,nlat,ntimes), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: tropp_p_in allocation error = ',ierr + call endrun + end if + !------------------------------------------------------------------ + ! ... read in the tropopause pressure + !------------------------------------------------------------------ + ierr = pio_inq_varid( pio_id, 'trop_p', vid ) + start = (/ 1, 1, 1 /) + count = (/ nlon, nlat, ntimes /) + ierr = pio_get_var( pio_id, vid, start, count, tropp_p_in ) + + !------------------------------------------------------------------ + ! ... close the netcdf file + !------------------------------------------------------------------ + call pio_closefile( pio_id ) + + !-------------------------------------------------------------------- + ! ... regrid + !-------------------------------------------------------------------- + + allocate( tropp_p_loc(pcols,begchunk:endchunk,ntimes), stat=ierr ) + + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: tropp_p_loc allocation error = ',ierr + call endrun + end if + + do c=begchunk,endchunk + ncols = get_ncols_p(c) + call get_rlat_all_p(c, pcols, to_lats) + call get_rlon_all_p(c, pcols, to_lons) + call lininterp_init(lon, nlon, to_lons, ncols, 2, lon_wgts, zero, twopi) + call lininterp_init(lat, nlat, to_lats, ncols, 1, lat_wgts) + do n=1,ntimes + call lininterp(tropp_p_in(:,:,n), nlon, nlat, tropp_p_loc(1:ncols,c,n), ncols, lon_wgts, lat_wgts) + end do + call lininterp_finish(lon_wgts) + call lininterp_finish(lat_wgts) + end do + deallocate(lon) + deallocate(lat) + deallocate(tropp_p_in) + + !-------------------------------------------------------- + ! ... initialize the monthly day of year times + !-------------------------------------------------------- + + do n = 1,12 + days(n) = get_calday( dates(n), 0 ) + end do + if (masterproc) then + write(iulog,*) 'tropopause_init : days' + write(iulog,'(1p,5g15.8)') days(:) + endif + + end subroutine tropopause_read_file + + + ! This analytic expression closely matches the mean tropopause determined + ! by the NCEP reanalysis and has been used by the radiation code. + subroutine tropopause_analytic(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + integer :: i + integer :: k + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + ! Calculate the pressure of the tropopause. + tP = (25000.0_r8 - 15000.0_r8 * (cos(pstate%lat(i)))**2) + + ! Find the level that contains the tropopause. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end do + end subroutine tropopause_analytic + + + ! Read the tropopause pressure in from a file containging a climatology. The + ! data is interpolated to the current dat of year and latitude. + ! + ! NOTE: The data is read in during tropopause_init and stored in the module + ! variable trop + subroutine tropopause_climate(pstate, tropLev, tropP, tropT, tropZ) + use time_manager, only : get_curr_calday + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + integer :: i + integer :: k + integer :: m + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + real(r8) :: calday ! day of year including fraction + real(r8) :: dels + integer :: last + integer :: next + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! If any columns remain to be indentified, the nget the current + ! day from the calendar. + + if (any(tropLev == NOTFOUND)) then + + ! Determine the calendar day. + calday = get_curr_calday() + + !-------------------------------------------------------- + ! ... setup the time interpolation + !-------------------------------------------------------- + if( calday < days(1) ) then + next = 1 + last = 12 + dels = (365._r8 + calday - days(12)) / (365._r8 + days(1) - days(12)) + else if( calday >= days(12) ) then + next = 1 + last = 12 + dels = (calday - days(12)) / (365._r8 + days(1) - days(12)) + else + do m = 11,1,-1 + if( calday >= days(m) ) then + exit + end if + end do + last = m + next = m + 1 + dels = (calday - days(m)) / (days(m+1) - days(m)) + end if + + dels = max( min( 1._r8,dels ),0._r8 ) + + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + !-------------------------------------------------------- + ! ... get tropopause level from climatology + !-------------------------------------------------------- + ! Interpolate the tropopause pressure. + tP = tropp_p_loc(i,lchnk,last) & + + dels * (tropp_p_loc(i,lchnk,next) - tropp_p_loc(i,lchnk,last)) + + ! Find the associated level. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end do + end if + + return + end subroutine tropopause_climate + + !----------------------------------------------------------------------- + !----------------------------------------------------------------------- + subroutine tropopause_hybridstobie(pstate, tropLev, tropP, tropT, tropZ) + use cam_history, only : outfld + + !----------------------------------------------------------------------- + ! Originally written by Philip Cameron-Smith, LLNL + ! + ! Stobie-Linoz hybrid: the highest altitude of + ! a) Stobie algorithm, or + ! b) minimum Linoz pressure. + ! + ! NOTE: the ltrop(i) gridbox itself is assumed to be a STRATOSPHERIC gridbox. + !----------------------------------------------------------------------- + !----------------------------------------------------------------------- + ! ... Local variables + !----------------------------------------------------------------------- + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + real(r8),parameter :: min_Stobie_Pressure= 40.E2_r8 !For case 2 & 4. [Pa] + real(r8),parameter :: max_Linoz_Pressure =208.E2_r8 !For case 4. [Pa] + + integer :: i, k, ncol + real(r8) :: stobie_min, shybrid_temp !temporary variable for case 2 & 3. + integer :: ltrop_linoz(pcols) !Lowest possible Linoz vertical level + integer :: ltrop_trop(pcols) !Tropopause level for hybrid case. + logical :: ltrop_linoz_set !Flag that lowest linoz level already found. + real(r8) :: trop_output(pcols,pver) !For output purposes only. + real(r8) :: trop_linoz_output(pcols,pver) !For output purposes only. + real(r8) :: trop_trop_output(pcols,pver) !For output purposes only. + + ! write(iulog,*) 'In set_chem_trop, o3_ndx =',o3_ndx + ltrop_linoz(:) = 1 ! Initialize to default value. + ltrop_trop(:) = 1 ! Initialize to default value. + ncol = pstate%ncol + + LOOP_COL4: do i=1,ncol + + ! Skip column in which the tropopause has already been found. + not_found: if (tropLev(i) == NOTFOUND) then + + stobie_min = 1.e10_r8 ! An impossibly large number + ltrop_linoz_set = .FALSE. + LOOP_LEV: do k=pver,1,-1 + IF (pstate%pmid(i,k) < min_stobie_pressure) cycle + shybrid_temp = ALPHA * pstate%t(i,k) - Log10(pstate%pmid(i,k)) + !PJC_NOTE: the units of pmid won't matter, because it is just an additive offset. + IF (shybrid_temp= 55000._r8) then + cycle + end if + + if ((tLev == -1) .or. (stobie(k) < sTrop)) then + tLev = k + tP = pstate%pmid(i, k) + sTrop = stobie(k) + end if + end do + + if (tLev /= -1) then + tropLev(i) = tLev + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end if + end do + + return + end subroutine tropopause_stobie + + + ! This routine is an implementation of Reichler et al. [2003] done by + ! Reichler and downloaded from his web site. Minimal modifications were + ! made to have the routine work within the CAM framework (i.e. using + ! CAM constants and types). + ! + ! NOTE: I am not a big fan of the goto's and multiple returns in this + ! code, but for the moment I have left them to preserve as much of the + ! original and presumably well tested code as possible. + ! UPDATE: The most "obvious" substitutions have been made to replace + ! goto/return statements with cycle/exit. The structure is still + ! somewhat tangled. + ! UPDATE 2: "gamma" renamed to "gam" in order to avoid confusion + ! with the Fortran 2008 intrinsic. "level" argument removed because + ! a physics column is not contiguous, so using explicit dimensions + ! will cause the data to be needlessly copied. + ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + ! + ! determination of tropopause height from gridded temperature data + ! + ! reference: Reichler, T., M. Dameris, and R. Sausen (2003) + ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + subroutine twmo(t, p, plimu, pliml, gam, trp) + + real(r8), intent(in), dimension(:) :: t, p + real(r8), intent(in) :: plimu, pliml, gam + real(r8), intent(out) :: trp + + real(r8), parameter :: deltaz = 2000.0_r8 + + real(r8) :: pmk, pm, a, b, tm, dtdp, dtdz + real(r8) :: ag, bg, ptph + real(r8) :: pm0, pmk0, dtdz0 + real(r8) :: p2km, asum, aquer + real(r8) :: pmk2, pm2, a2, b2, tm2, dtdp2, dtdz2 + integer :: level + integer :: icount, jj + integer :: j + + + trp=-99.0_r8 ! negative means not valid + + ! initialize start level + ! dt/dz + level = size(t) + pmk= .5_r8 * (p(level-1)**cnst_kap+p(level)**cnst_kap) + pm = pmk**(1/cnst_kap) + a = (t(level-1)-t(level))/(p(level-1)**cnst_kap-p(level)**cnst_kap) + b = t(level)-(a*p(level)**cnst_kap) + tm = a * pmk + b + dtdp = a * cnst_kap * (pm**cnst_ka1) + dtdz = cnst_faktor*dtdp*pm/tm + + main_loop: do j=level-1,2,-1 + pm0 = pm + pmk0 = pmk + dtdz0 = dtdz + + ! dt/dz + pmk= .5_r8 * (p(j-1)**cnst_kap+p(j)**cnst_kap) + pm = pmk**(1/cnst_kap) + a = (t(j-1)-t(j))/(p(j-1)**cnst_kap-p(j)**cnst_kap) + b = t(j)-(a*p(j)**cnst_kap) + tm = a * pmk + b + dtdp = a * cnst_kap * (pm**cnst_ka1) + dtdz = cnst_faktor*dtdp*pm/tm + ! dt/dz valid? + if (dtdz.le.gam) cycle main_loop ! no, dt/dz < -2 K/km + if (pm.gt.plimu) cycle main_loop ! no, too low + + ! dtdz is valid, calculate tropopause pressure + if (dtdz0.lt.gam) then + ag = (dtdz-dtdz0) / (pmk-pmk0) + bg = dtdz0 - (ag * pmk0) + ptph = exp(log((gam-bg)/ag)/cnst_kap) + else + ptph = pm + endif + + if (ptph.lt.pliml) cycle main_loop + if (ptph.gt.plimu) cycle main_loop + + ! 2nd test: dtdz above 2 km must not exceed gam + p2km = ptph + deltaz*(pm/tm)*cnst_faktor ! p at ptph + 2km + asum = 0.0_r8 ! dtdz above + icount = 0 ! number of levels above + + ! test until apm < p2km + in_loop: do jj=j,2,-1 + + pmk2 = .5_r8 * (p(jj-1)**cnst_kap+p(jj)**cnst_kap) ! p mean ^kappa + pm2 = pmk2**(1/cnst_kap) ! p mean + if(pm2.gt.ptph) cycle in_loop ! doesn't happen + if(pm2.lt.p2km) exit in_loop ! ptropo is valid + + a2 = (t(jj-1)-t(jj)) ! a + a2 = a2/(p(jj-1)**cnst_kap-p(jj)**cnst_kap) + b2 = t(jj)-(a2*p(jj)**cnst_kap) ! b + tm2 = a2 * pmk2 + b2 ! T mean + dtdp2 = a2 * cnst_kap * (pm2**(cnst_kap-1)) ! dt/dp + dtdz2 = cnst_faktor*dtdp2*pm2/tm2 + asum = asum+dtdz2 + icount = icount+1 + aquer = asum/float(icount) ! dt/dz mean + + ! discard ptropo ? + if (aquer.le.gam) cycle main_loop ! dt/dz above < gam + + enddo in_loop ! test next level + + trp = ptph + exit main_loop + enddo main_loop + + end subroutine twmo + + + ! This routine uses an implementation of Reichler et al. [2003] done by + ! Reichler and downloaded from his web site. This is similar to the WMO + ! routines, but is designed for GCMs with a coarse vertical grid. + subroutine tropopause_twmo(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: gam = -0.002_r8 ! K/m + real(r8), parameter :: plimu = 45000._r8 ! Pa + real(r8), parameter :: pliml = 7500._r8 ! Pa + + integer :: i + integer :: k + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + ! Use the routine from Reichler. + call twmo(pstate%t(i, :), pstate%pmid(i, :), plimu, pliml, gam, tP) + + ! if successful, store of the results and find the level and temperature. + if (tP > 0) then + + ! Find the associated level. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end if + end do + + return + end subroutine tropopause_twmo + + ! This routine implements the WMO definition of the tropopause (WMO, 1957; Seidel and Randel, 2006). + ! This requires that the lapse rate be less than 2 K/km for an altitude range + ! of 2 km. The search starts at the surface and stops the first time this + ! criteria is met. + ! + ! NOTE: This code was modeled after the code in mo_tropopause; however, the + ! requirement that dt be greater than 0 was removed and the check to make + ! sure that the lapse rate is maintained for 2 km was added. + subroutine tropopause_wmo(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: ztrop_low = 5000._r8 ! lowest tropopause level allowed (m) + real(r8), parameter :: ztrop_high = 20000._r8 ! highest tropopause level allowed (m) + real(r8), parameter :: max_dtdz = 0.002_r8 ! max dt/dz for tropopause level (K/m) + real(r8), parameter :: min_trop_dz = 2000._r8 ! min tropopause thickness (m) + + integer :: i + integer :: k + integer :: k2 + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + real(r8) :: dt + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + kloop: do k = pver-1, 2, -1 + + ! Skip levels below the minimum and stop if nothing is found + ! before the maximum. + if (pstate%zm(i, k) < ztrop_low) then + cycle kloop + else if (pstate%zm(i, k) > ztrop_high) then + exit kloop + end if + + ! Compare the actual lapse rate to the threshold + dt = pstate%t(i, k) - pstate%t(i, k-1) + + if (dt <= (max_dtdz * (pstate%zm(i, k-1) - pstate%zm(i, k)))) then + + ! Make sure that the lapse rate stays below the threshold for the + ! specified range. + k2loop: do k2 = k-1, 2, -1 + if ((pstate%zm(i, k2) - pstate%zm(i, k)) >= min_trop_dz) then + tP = pstate%pmid(i, k) + tropLev(i) = k + exit k2loop + end if + + dt = pstate%t(i, k) - pstate%t(i, k2) + if (dt > (max_dtdz * (pstate%zm(i, k2) - pstate%zm(i, k)))) then + exit k2loop + end if + end do k2loop + + if (tropLev(i) == NOTFOUND) then + cycle kloop + else + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + + exit kloop + end if + end if + end do kloop + end if + end do + + return + end subroutine tropopause_wmo + + ! This routine searches for the cold point tropopause, and uses a parabolic + ! fit of the coldest point and two adjacent points to interpolate the cold point + ! between model levels. + subroutine tropopause_cpp(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: ztrop_low = 5000._r8 ! lowest tropopause level allowed (m) + real(r8), parameter :: ztrop_high = 25000._r8 ! highest tropopause level allowed (m) + + integer :: i + integer :: k, firstk, lastk + integer :: k2 + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tZ ! tropopause height (m) + real(r8) :: tmin + real(r8) :: f0, f1, f2 + real(r8) :: x0, x1, x2 + real(r8) :: c0, c1, c2 + real(r8) :: a, b, c + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + firstk = 0 + lastk = pver+1 + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + tmin = 1e6_r8 + + kloop: do k = pver-1, 2, -1 + + ! Skip levels below the minimum and stop if nothing is found + ! before the maximum. + if (pstate%zm(i, k) < ztrop_low) then + firstk = k + cycle kloop + else if (pstate%zm(i, k) > ztrop_high) then + lastk = k + exit kloop + end if + + ! Find the coldest point + if (pstate%t(i, k) < tmin) then + tropLev(i) = k + tmin = pstate%t(i,k) + end if + end do kloop + + ! If the minimum is at the edge of the search range, then don't + ! consider this to be a minima + if ((tropLev(i) >= (firstk-1)) .or. (tropLev(i) <= (lastk+1))) then + tropLev(i) = NOTFOUND + else + + ! If returning P, Z, or T, then do a parabolic fit using the + ! cold point and it its 2 surrounding points to interpolate + ! between model levels. + if (present(tropP) .or. present(tropZ) .or. present(tropT)) then + f0 = pstate%t(i, tropLev(i)-1) + f1 = pstate%t(i, tropLev(i)) + f2 = pstate%t(i, tropLev(i)+1) + + x0 = pstate%zm(i, tropLev(i)-1) + x1 = pstate%zm(i, tropLev(i)) + x2 = pstate%zm(i, tropLev(i)+1) + + c0 = (x0-x1)*(x0-x2) + c1 = (x1-x0)*(x1-x2) + c2 = (x2-x0)*(x2-x1) + + ! Determine the quadratic coefficients of: + ! T = a * z^2 - b*z + c + a = (f0/c0 + f1/c1 + f2/c2) + b = (f0/c0*(x1+x2) + f1/c1*(x0+x2) + f2/c2*(x0+x1)) + c = f0/c0*x1*x2 + f1/c1*x0*x2 + f2/c2*x0*x1 + + ! Find the altitude of the minimum temperature + tZ = 0.5_r8 * b / a + + ! The fit should be between the upper and lower points, + ! so skip the point if the fit fails. + if ((tZ >= x0) .or. (tZ <= x2)) then + tropLev(i) = NOTFOUND + else + ! Return the optional outputs + if (present(tropP)) then + tropP(i) = tropopause_interpolateP(pstate, i, tropLev(i), tZ) + end if + + if (present(tropT)) then + tropT(i) = a * tZ*tZ - b*tZ + c + end if + + if (present(tropZ)) then + tropZ(i) = tZ + end if + end if + end if + end if + end if + end do + + return + end subroutine tropopause_cpp + + + ! Searches all the columns in the chunk and attempts to identify the tropopause. + ! Two routines can be specifed, a primary routine which is tried first and a + ! backup routine which will be tried only if the first routine fails. If the + ! tropopause can not be identified by either routine, then a NOTFOUND is returned + ! for the tropopause level, temperature and pressure. + subroutine tropopause_find(pstate, tropLev, tropP, tropT, tropZ, primary, backup) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, optional, intent(in) :: primary ! primary detection algorithm + integer, optional, intent(in) :: backup ! backup detection algorithm + integer, intent(out) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(out) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(out) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(out) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variable + integer :: primAlg ! Primary algorithm + integer :: backAlg ! Backup algorithm + + ! Initialize the results to a missing value, so that the algorithms will + ! attempt to find the tropopause for all of them. + tropLev(:) = NOTFOUND + if (present(tropP)) tropP(:) = fillvalue + if (present(tropT)) tropT(:) = fillvalue + if (present(tropZ)) tropZ(:) = fillvalue + + ! Set the algorithms to be used, either the ones provided or the defaults. + if (present(primary)) then + primAlg = primary + else + primAlg = default_primary + end if + + if (present(backup)) then + backAlg = backup + else + backAlg = default_backup + end if + + ! Try to find the tropopause using the primary algorithm. + if (primAlg /= TROP_ALG_NONE) then + call tropopause_findUsing(pstate, primAlg, tropLev, tropP, tropT, tropZ) + end if + + if ((backAlg /= TROP_ALG_NONE) .and. any(tropLev(:) == NOTFOUND)) then + call tropopause_findUsing(pstate, backAlg, tropLev, tropP, tropT, tropZ) + end if + + return + end subroutine tropopause_find + + ! Searches all the columns in the chunk and attempts to identify the "chemical" + ! tropopause. This is the lapse rate tropopause, backed up by the climatology + ! if the lapse rate fails to find the tropopause at pressures higher than a certain + ! threshold. This pressure threshold depends on latitude. Between 50S and 50N, + ! the climatology is used if the lapse rate tropopause is not found at P > 75 hPa. + ! At high latitude (poleward of 50), the threshold is increased to 125 hPa to + ! eliminate false events that are sometimes detected in the cold polar stratosphere. + ! + ! NOTE: This routine was adapted from code in chemistry.F90 and mo_gasphase_chemdr.F90. + subroutine tropopause_findChemTrop(pstate, tropLev, primary, backup) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, optional, intent(in) :: primary ! primary detection algorithm + integer, optional, intent(in) :: backup ! backup detection algorithm + integer, intent(out) :: tropLev(pcols) ! tropopause level index + + ! Local Variable + real(r8), parameter :: rad2deg = 180._r8/pi ! radians to degrees conversion factor + real(r8) :: dlats(pcols) + integer :: i + integer :: ncol + integer :: backAlg + + ! First use the lapse rate tropopause. + ncol = pstate%ncol + call tropopause_find(pstate, tropLev, primary=primary, backup=TROP_ALG_NONE) + + ! Now check high latitudes (poleward of 50) and set the level to the + ! climatology if the level was not found or is at P <= 125 hPa. + dlats(:ncol) = pstate%lat(:ncol) * rad2deg ! convert to degrees + + if (present(backup)) then + backAlg = backup + else + backAlg = default_backup + end if + + do i = 1, ncol + if (abs(dlats(i)) > 50._r8) then + if (tropLev(i) .ne. NOTFOUND) then + if (pstate%pmid(i, tropLev(i)) <= 12500._r8) then + tropLev(i) = NOTFOUND + end if + end if + end if + end do + + ! Now use the backup algorithm + if ((backAlg /= TROP_ALG_NONE) .and. any(tropLev(:) == NOTFOUND)) then + call tropopause_findUsing(pstate, backAlg, tropLev) + end if + + return + end subroutine tropopause_findChemTrop + + + + + ! Call the appropriate tropopause detection routine based upon the algorithm + ! specifed. + ! + ! NOTE: It is assumed that the output fields have been initialized by the + ! caller, and only output values set to fillvalue will be detected. + subroutine tropopause_findUsing(pstate, algorithm, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: algorithm ! detection algorithm + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Dispatch the request to the appropriate routine. + select case(algorithm) + case(TROP_ALG_ANALYTIC) + call tropopause_analytic(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_CLIMATE) + call tropopause_climate(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_STOBIE) + call tropopause_stobie(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_HYBSTOB) + call tropopause_hybridstobie(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_TWMO) + call tropopause_twmo(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_WMO) + call tropopause_wmo(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_CPP) + call tropopause_cpp(pstate, tropLev, tropP, tropT, tropZ) + + case default + write(iulog, *) 'tropopause: Invalid detection algorithm (', algorithm, ') specified.' + call endrun + end select + + return + end subroutine tropopause_findUsing + + ! This routine interpolates the pressures in the physics state to + ! find the pressure at the specified tropopause altitude. + function tropopause_interpolateP(pstate, icol, tropLev, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropZ ! tropopause pressure (m) + real(r8) :: tropopause_interpolateP + + ! Local Variables + real(r8) :: tropP ! tropopause pressure (Pa) + real(r8) :: dlogPdZ ! dlog(p)/dZ + + ! Interpolate the temperature linearly against log(P) + + ! Is the tropopause at the midpoint? + if (tropZ == pstate%zm(icol, tropLev)) then + tropP = pstate%pmid(icol, tropLev) + + else if (tropZ > pstate%zm(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + if (tropLev > 1) then + dlogPdZ = (log(pstate%pmid(icol, tropLev)) - log(pstate%pmid(icol, tropLev - 1))) / & + (pstate%zm(icol, tropLev) - pstate%zm(icol, tropLev - 1)) + tropP = pstate%pmid(icol, tropLev) + exp((tropZ - pstate%zm(icol, tropLev)) * dlogPdZ) + end if + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + if (tropLev < pver) then + dlogPdZ = (log(pstate%pmid(icol, tropLev + 1)) - log(pstate%pmid(icol, tropLev))) / & + (pstate%zm(icol, tropLev + 1) - pstate%zm(icol, tropLev)) + tropP = pstate%pmid(icol, tropLev) + exp((tropZ - pstate%zm(icol, tropLev)) * dlogPdZ) + end if + end if + + tropopause_interpolateP = tropP + end function tropopause_interpolateP + + + + ! This routine interpolates the temperatures in the physics state to + ! find the temperature at the specified tropopause pressure. + function tropopause_interpolateT(pstate, icol, tropLev, tropP) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropP ! tropopause pressure (Pa) + real(r8) :: tropopause_interpolateT + + ! Local Variables + real(r8) :: tropT ! tropopause temperature (K) + real(r8) :: dTdlogP ! dT/dlog(P) + + ! Intrepolate the temperature linearly against log(P) + + ! Is the tropopause at the midpoint? + if (tropP == pstate%pmid(icol, tropLev)) then + tropT = pstate%t(icol, tropLev) + + else if (tropP < pstate%pmid(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + if (tropLev > 1) then + dTdlogP = (pstate%t(icol, tropLev) - pstate%t(icol, tropLev - 1)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pmid(icol, tropLev - 1))) + tropT = pstate%t(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dTdlogP + end if + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + if (tropLev < pver) then + dTdlogP = (pstate%t(icol, tropLev + 1) - pstate%t(icol, tropLev)) / & + (log(pstate%pmid(icol, tropLev + 1)) - log(pstate%pmid(icol, tropLev))) + tropT = pstate%t(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dTdlogP + end if + end if + + tropopause_interpolateT = tropT + end function tropopause_interpolateT + + + ! This routine interpolates the geopotential height in the physics state to + ! find the geopotential height at the specified tropopause pressure. + function tropopause_interpolateZ(pstate, icol, tropLev, tropP) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropP ! tropopause pressure (Pa) + real(r8) :: tropopause_interpolateZ + + ! Local Variables + real(r8) :: tropZ ! tropopause geopotential height (m) + real(r8) :: dZdlogP ! dZ/dlog(P) + + ! Intrepolate the geopotential height linearly against log(P) + + ! Is the tropoause at the midpoint? + if (tropP == pstate%pmid(icol, tropLev)) then + tropZ = pstate%zm(icol, tropLev) + + else if (tropP < pstate%pmid(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + dZdlogP = (pstate%zm(icol, tropLev) - pstate%zi(icol, tropLev)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pint(icol, tropLev))) + tropZ = pstate%zm(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dZdlogP + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + dZdlogP = (pstate%zm(icol, tropLev) - pstate%zi(icol, tropLev+1)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pint(icol, tropLev+1))) + tropZ = pstate%zm(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dZdlogP + end if + + tropopause_interpolateZ = tropZ + end function tropopause_interpolateZ + + + ! Output the tropopause pressure and temperature to the history files. Two sets + ! of output will be generated, one for the default algorithm and another one + ! using the default routine, but backed by a climatology when the default + ! algorithm fails. + subroutine tropopause_output(pstate) + use cam_history, only : outfld + + implicit none + + type(physics_state), intent(in) :: pstate + + ! Local Variables + integer :: i + integer :: alg + integer :: ncol ! number of cloumns in the chunk + integer :: lchnk ! chunk identifier + integer :: tropLev(pcols) ! tropopause level index + real(r8) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8) :: tropT(pcols) ! tropopause temperature (K) + real(r8) :: tropZ(pcols) ! tropopause height (m) + real(r8) :: tropFound(pcols) ! tropopause found + real(r8) :: tropDZ(pcols, pver) ! relative tropopause height (m) + real(r8) :: tropPdf(pcols, pver) ! tropopause probability distribution + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Find the tropopause using the default algorithm backed by the climatology. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROP_P', tropP(:ncol), ncol, lchnk) + call outfld('TROP_T', tropT(:ncol), ncol, lchnk) + call outfld('TROP_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROP_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROP_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROP_FD', tropFound(:ncol), ncol, lchnk) + + + ! Find the tropopause using just the primary algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROPP_P', tropP(:ncol), ncol, lchnk) + call outfld('TROPP_T', tropT(:ncol), ncol, lchnk) + call outfld('TROPP_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROPP_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROPP_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROPP_FD', tropFound(:ncol), ncol, lchnk) + + ! Find the tropopause using just the cold point algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, primary=TROP_ALG_CPP, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROPF_P', tropP(:ncol), ncol, lchnk) + call outfld('TROPF_T', tropT(:ncol), ncol, lchnk) + call outfld('TROPF_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROPF_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROPF_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROPF_FD', tropFound(:ncol), ncol, lchnk) + + + ! If requested, do all of the algorithms. + if (output_all) then + + do alg = 2, TROP_NALG + + ! Find the tropopause using just the analytic algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, primary=alg, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + end if + end do + + call outfld('TROP' // TROP_LETTER(alg) // '_P', tropP(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_T', tropT(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_FD', tropFound(:ncol), ncol, lchnk) + end do + end if + + return + end subroutine tropopause_output +end module tropopause diff --git a/SourceMods/src.cam/srcMods.tar b/SourceMods/src.cam/srcMods.tar new file mode 100644 index 0000000..07910ed Binary files /dev/null and b/SourceMods/src.cam/srcMods.tar differ diff --git a/SourceMods/src.cam/tropopause.F90 b/SourceMods/src.cam/tropopause.F90 new file mode 100755 index 0000000..ceb5c2e --- /dev/null +++ b/SourceMods/src.cam/tropopause.F90 @@ -0,0 +1,1516 @@ + +! This module is used to diagnose the location of the tropopause. Multiple +! algorithms are provided, some of which may not be able to identify a +! tropopause in all situations. To handle these cases, an analytic +! definition and a climatology are provided that can be used to fill in +! when the original algorithm fails. The tropopause temperature and +! pressure are determined and can be output to the history file. +! +! These routines are based upon code in the WACCM chemistry module +! including mo_tropoause.F90 and llnl_set_chem_trop.F90. The code +! for the Reichler et al. [2003] algorithm is from: +! +! http://www.gfdl.noaa.gov/~tjr/TROPO/tropocode.htm +! +! Author: Charles Bardeen +! Created: April, 2009 + +module tropopause + !--------------------------------------------------------------- + ! ... variables for the tropopause module + !--------------------------------------------------------------- + + use shr_kind_mod, only : r8 => shr_kind_r8 + use shr_const_mod, only : pi => shr_const_pi + use ppgrid, only : pcols, pver, begchunk, endchunk + use cam_abortutils, only : endrun + use cam_logfile, only : iulog + use cam_history_support, only : fillvalue + use physics_types, only : physics_state + use physconst, only : cappa, rair, gravit + use spmd_utils, only : masterproc + + implicit none + + private + + public :: tropopause_readnl, tropopause_init, tropopause_find, tropopause_output + public :: tropopause_findChemTrop + public :: TROP_ALG_NONE, TROP_ALG_ANALYTIC, TROP_ALG_CLIMATE + public :: TROP_ALG_STOBIE, TROP_ALG_HYBSTOB, TROP_ALG_TWMO, TROP_ALG_WMO + public :: TROP_ALG_CPP + public :: NOTFOUND + + save + + ! These parameters define and enumeration to be used to define the primary + ! and backup algorithms to be used with the tropopause_find() method. The + ! backup algorithm is meant to provide a solution when the primary algorithm + ! fail. The algorithms that can't fail are: TROP_ALG_ANALYTIC, TROP_ALG_CLIMATE + ! and TROP_ALG_STOBIE. + integer, parameter :: TROP_ALG_NONE = 1 ! Don't evaluate + integer, parameter :: TROP_ALG_ANALYTIC = 2 ! Analytic Expression + integer, parameter :: TROP_ALG_CLIMATE = 3 ! Climatology + integer, parameter :: TROP_ALG_STOBIE = 4 ! Stobie Algorithm + integer, parameter :: TROP_ALG_TWMO = 5 ! WMO Definition, Reichler et al. [2003] + integer, parameter :: TROP_ALG_WMO = 6 ! WMO Definition + integer, parameter :: TROP_ALG_HYBSTOB = 7 ! Hybrid Stobie Algorithm + integer, parameter :: TROP_ALG_CPP = 8 ! Cold Point Parabolic + + integer, parameter :: TROP_NALG = 8 ! Number of Algorithms + character,parameter :: TROP_LETTER(TROP_NALG) = (/ ' ', 'A', 'C', 'S', 'T', 'W', 'H', 'F' /) + ! unique identifier for output, don't use P + + ! These variables should probably be controlled by namelist entries. + logical ,parameter :: output_all = .False. ! output tropopause info from all algorithms + integer ,parameter :: default_primary = TROP_ALG_TWMO ! default primary algorithm + integer ,parameter :: default_backup = TROP_ALG_CLIMATE ! default backup algorithm + + ! Namelist variables + character(len=256) :: tropopause_climo_file = 'trop_climo' ! absolute filepath of climatology file + + ! These variables are used to store the climatology data. + real(r8) :: days(12) ! days in the climatology + real(r8), pointer :: tropp_p_loc(:,:,:) ! climatological tropopause pressures + + integer, parameter :: NOTFOUND = -1 + + real(r8),parameter :: ALPHA = 0.03_r8 + + ! physical constants + ! These constants are set in module variables rather than as parameters + ! to support the aquaplanet mode in which the constants have values determined + ! by the experiment protocol + real(r8) :: cnst_kap ! = cappa + real(r8) :: cnst_faktor ! = -gravit/rair + real(r8) :: cnst_ka1 ! = cnst_kap - 1._r8 + +!================================================================================================ +contains +!================================================================================================ + + ! Read namelist variables. + subroutine tropopause_readnl(nlfile) + + use namelist_utils, only: find_group_name + use units, only: getunit, freeunit + use mpishorthand + + character(len=*), intent(in) :: nlfile ! filepath for file containing namelist input + + ! Local variables + integer :: unitn, ierr + character(len=*), parameter :: subname = 'tropopause_readnl' + + namelist /tropopause_nl/ tropopause_climo_file + !----------------------------------------------------------------------------- + + if (masterproc) then + unitn = getunit() + open( unitn, file=trim(nlfile), status='old' ) + call find_group_name(unitn, 'tropopause_nl', status=ierr) + if (ierr == 0) then + read(unitn, tropopause_nl, iostat=ierr) + if (ierr /= 0) then + call endrun(subname // ':: ERROR reading namelist') + end if + end if + close(unitn) + call freeunit(unitn) + end if + +#ifdef SPMD + ! Broadcast namelist variables + call mpibcast(tropopause_climo_file, len(tropopause_climo_file), mpichar, 0, mpicom) +#endif + + end subroutine tropopause_readnl + + + ! This routine is called during intialization and must be called before the + ! other methods in this module can be used. Its main tasks are to read in the + ! climatology from a file and to define the output fields. Much of this code + ! is taken from mo_tropopause. + subroutine tropopause_init() + + + use ppgrid, only: pver + use cam_pio_utils, only: + use cam_history, only: addfld, horiz_only, add_default + + + implicit none + + ! define physical constants + cnst_kap = cappa + cnst_faktor = -gravit/rair + cnst_ka1 = cnst_kap - 1._r8 + + ! Define the output fields. + call addfld('TROP_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure', flag_xyfill=.True.) + call addfld('TROP_T', horiz_only, 'A', 'K', 'Tropopause Temperature', flag_xyfill=.True.) + call addfld('TROP_Z', horiz_only, 'A', 'm', 'Tropopause Height', flag_xyfill=.True.) + call addfld('TROP_DZ', (/ 'lev' /), 'A', 'm', 'Relative Tropopause Height') + call addfld('TROP_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Probabilty') + call addfld('TROP_FD', horiz_only, 'A', 'probability', 'Tropopause Found') + + call addfld('TROPP_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (primary)', flag_xyfill=.True.) + call addfld('TROPP_T', horiz_only, 'A', 'K', 'Tropopause Temperature (primary)', flag_xyfill=.True.) + call addfld('TROPP_Z', horiz_only, 'A', 'm', 'Tropopause Height (primary)', flag_xyfill=.True.) + call addfld('TROPP_DZ', (/ 'lev' /), 'A', 'm', 'Relalive Tropopause Height (primary)') + call addfld('TROPP_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (primary)') + call addfld('TROPP_FD', horiz_only, 'A', 'probability', 'Tropopause Found (primary)') + +!! start nanr for Rich + call addfld('TROPF_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (cold point)', flag_xyfill=.True.) + call addfld('TROPF_T', horiz_only, 'A', 'K', 'Tropopause Temperature (cold point)', flag_xyfill=.True.) + call addfld('TROPF_Z', horiz_only, 'A', 'm', 'Tropopause Height (cold point)', flag_xyfill=.True.) + call addfld('TROPF_DZ', (/ 'lev' /), 'A', 'm', 'Relative Tropopause Height (cold point)', flag_xyfill=.True.) + call addfld('TROPF_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (cold point)') + call addfld('TROPF_FD', horiz_only, 'A', 'probability', 'Tropopause Found (cold point)') +!! end nanr + + call addfld( 'hstobie_trop', (/ 'lev' /), 'I', 'fraction of model time', 'Lowest level with stratospheric chemsitry' ) + call addfld( 'hstobie_linoz', (/ 'lev' /), 'I', 'fraction of model time', 'Lowest possible Linoz level' ) + call addfld( 'hstobie_tropop', (/ 'lev' /), 'I', 'fraction of model time', & + 'Troposphere boundary calculated in chemistry' ) + + ! If requested, be prepared to output results from all of the methods. + if (output_all) then + call addfld('TROPA_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (analytic)', flag_xyfill=.True.) + call addfld('TROPA_T', horiz_only, 'A', 'K', 'Tropopause Temperature (analytic)', flag_xyfill=.True.) + call addfld('TROPA_Z', horiz_only, 'A', 'm', 'Tropopause Height (analytic)', flag_xyfill=.True.) + call addfld('TROPA_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (analytic)') + call addfld('TROPA_FD', horiz_only, 'A', 'probability', 'Tropopause Found (analytic)') + + call addfld('TROPC_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (climatology)', flag_xyfill=.True.) + call addfld('TROPC_T', horiz_only, 'A', 'K', 'Tropopause Temperature (climatology)', flag_xyfill=.True.) + call addfld('TROPC_Z', horiz_only, 'A', 'm', 'Tropopause Height (climatology)', flag_xyfill=.True.) + call addfld('TROPC_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (climatology)') + call addfld('TROPC_FD', horiz_only, 'A', 'probability', 'Tropopause Found (climatology)') + + call addfld('TROPS_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (stobie)', flag_xyfill=.True.) + call addfld('TROPS_T', horiz_only, 'A', 'K', 'Tropopause Temperature (stobie)', flag_xyfill=.True.) + call addfld('TROPS_Z', horiz_only, 'A', 'm', 'Tropopause Height (stobie)', flag_xyfill=.True.) + call addfld('TROPS_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (stobie)') + call addfld('TROPS_FD', horiz_only, 'A', 'probability', 'Tropopause Found (stobie)') + + call addfld('TROPT_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (twmo)', flag_xyfill=.True.) + call addfld('TROPT_T', horiz_only, 'A', 'K', 'Tropopause Temperature (twmo)', flag_xyfill=.True.) + call addfld('TROPT_Z', horiz_only, 'A', 'm', 'Tropopause Height (twmo)', flag_xyfill=.True.) + call addfld('TROPT_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (twmo)') + call addfld('TROPT_FD', horiz_only, 'A', 'probability', 'Tropopause Found (twmo)') + + call addfld('TROPW_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (WMO)', flag_xyfill=.True.) + call addfld('TROPW_T', horiz_only, 'A', 'K', 'Tropopause Temperature (WMO)', flag_xyfill=.True.) + call addfld('TROPW_Z', horiz_only, 'A', 'm', 'Tropopause Height (WMO)', flag_xyfill=.True.) + call addfld('TROPW_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (WMO)') + call addfld('TROPW_FD', horiz_only, 'A', 'probability', 'Tropopause Found (WMO)') + + call addfld('TROPH_P', horiz_only, 'A', 'Pa', 'Tropopause Pressure (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_T', horiz_only, 'A', 'K', 'Tropopause Temperature (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_Z', horiz_only, 'A', 'm', 'Tropopause Height (Hybrid Stobie)', flag_xyfill=.True.) + call addfld('TROPH_PD', (/ 'lev' /), 'A', 'probability', 'Tropopause Distribution (Hybrid Stobie)') + call addfld('TROPH_FD', horiz_only, 'A', 'probability', 'Tropopause Found (Hybrid Stobie)') + end if + + call add_default('TROP_P', 1, ' ') + call add_default('TROP_T', 1, ' ') + call add_default('hstobie_linoz', 1, ' ') + + call tropopause_read_file() + + + end subroutine tropopause_init + + + subroutine tropopause_read_file + !------------------------------------------------------------------ + ! ... initialize upper boundary values + !------------------------------------------------------------------ + use interpolate_data, only : lininterp_init, lininterp, interp_type, lininterp_finish + use dyn_grid, only : get_dyn_grid_parm + use phys_grid, only : get_ncols_p, get_rlat_all_p, get_rlon_all_p + use ioFileMod, only : getfil + use time_manager, only : get_calday + use physconst, only : pi + use cam_pio_utils, only: cam_pio_openfile + use pio, only : file_desc_t, var_desc_t, pio_inq_dimid, pio_inq_dimlen, & + pio_inq_varid, pio_get_var, pio_closefile, pio_nowrite + + !------------------------------------------------------------------ + ! ... local variables + !------------------------------------------------------------------ + integer :: i, j, n + integer :: ierr + type(file_desc_t) :: pio_id + integer :: dimid + type(var_desc_t) :: vid + integer :: nlon, nlat, ntimes + integer :: start(3) + integer :: count(3) + integer, parameter :: dates(12) = (/ 116, 214, 316, 415, 516, 615, & + 716, 816, 915, 1016, 1115, 1216 /) + integer :: plon, plat + type(interp_type) :: lon_wgts, lat_wgts + real(r8), allocatable :: tropp_p_in(:,:,:) + real(r8), allocatable :: lat(:) + real(r8), allocatable :: lon(:) + real(r8) :: to_lats(pcols), to_lons(pcols) + real(r8), parameter :: d2r=pi/180._r8, zero=0._r8, twopi=pi*2._r8 + character(len=256) :: locfn + integer :: c, ncols + + + plon = get_dyn_grid_parm('plon') + plat = get_dyn_grid_parm('plat') + + + !----------------------------------------------------------------------- + ! ... open netcdf file + !----------------------------------------------------------------------- + call getfil (tropopause_climo_file, locfn, 0) + call cam_pio_openfile(pio_id, trim(locfn), PIO_NOWRITE) + + !----------------------------------------------------------------------- + ! ... get time dimension + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'time', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, ntimes ) + if( ntimes /= 12 )then + write(iulog,*) 'tropopause_init: number of months = ',ntimes,'; expecting 12' + call endrun + end if + !----------------------------------------------------------------------- + ! ... get latitudes + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'lat', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, nlat ) + allocate( lat(nlat), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: lat allocation error = ',ierr + call endrun + end if + ierr = pio_inq_varid( pio_id, 'lat', vid ) + ierr = pio_get_var( pio_id, vid, lat ) + lat(:nlat) = lat(:nlat) * d2r + !----------------------------------------------------------------------- + ! ... get longitudes + !----------------------------------------------------------------------- + ierr = pio_inq_dimid( pio_id, 'lon', dimid ) + ierr = pio_inq_dimlen( pio_id, dimid, nlon ) + allocate( lon(nlon), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: lon allocation error = ',ierr + call endrun + end if + ierr = pio_inq_varid( pio_id, 'lon', vid ) + ierr = pio_get_var( pio_id, vid, lon ) + lon(:nlon) = lon(:nlon) * d2r + + !------------------------------------------------------------------ + ! ... allocate arrays + !------------------------------------------------------------------ + allocate( tropp_p_in(nlon,nlat,ntimes), stat=ierr ) + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: tropp_p_in allocation error = ',ierr + call endrun + end if + !------------------------------------------------------------------ + ! ... read in the tropopause pressure + !------------------------------------------------------------------ + ierr = pio_inq_varid( pio_id, 'trop_p', vid ) + start = (/ 1, 1, 1 /) + count = (/ nlon, nlat, ntimes /) + ierr = pio_get_var( pio_id, vid, start, count, tropp_p_in ) + + !------------------------------------------------------------------ + ! ... close the netcdf file + !------------------------------------------------------------------ + call pio_closefile( pio_id ) + + !-------------------------------------------------------------------- + ! ... regrid + !-------------------------------------------------------------------- + + allocate( tropp_p_loc(pcols,begchunk:endchunk,ntimes), stat=ierr ) + + if( ierr /= 0 ) then + write(iulog,*) 'tropopause_init: tropp_p_loc allocation error = ',ierr + call endrun + end if + + do c=begchunk,endchunk + ncols = get_ncols_p(c) + call get_rlat_all_p(c, pcols, to_lats) + call get_rlon_all_p(c, pcols, to_lons) + call lininterp_init(lon, nlon, to_lons, ncols, 2, lon_wgts, zero, twopi) + call lininterp_init(lat, nlat, to_lats, ncols, 1, lat_wgts) + do n=1,ntimes + call lininterp(tropp_p_in(:,:,n), nlon, nlat, tropp_p_loc(1:ncols,c,n), ncols, lon_wgts, lat_wgts) + end do + call lininterp_finish(lon_wgts) + call lininterp_finish(lat_wgts) + end do + deallocate(lon) + deallocate(lat) + deallocate(tropp_p_in) + + !-------------------------------------------------------- + ! ... initialize the monthly day of year times + !-------------------------------------------------------- + + do n = 1,12 + days(n) = get_calday( dates(n), 0 ) + end do + if (masterproc) then + write(iulog,*) 'tropopause_init : days' + write(iulog,'(1p,5g15.8)') days(:) + endif + + end subroutine tropopause_read_file + + + ! This analytic expression closely matches the mean tropopause determined + ! by the NCEP reanalysis and has been used by the radiation code. + subroutine tropopause_analytic(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + integer :: i + integer :: k + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + ! Calculate the pressure of the tropopause. + tP = (25000.0_r8 - 15000.0_r8 * (cos(pstate%lat(i)))**2) + + ! Find the level that contains the tropopause. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end do + end subroutine tropopause_analytic + + + ! Read the tropopause pressure in from a file containging a climatology. The + ! data is interpolated to the current dat of year and latitude. + ! + ! NOTE: The data is read in during tropopause_init and stored in the module + ! variable trop + subroutine tropopause_climate(pstate, tropLev, tropP, tropT, tropZ) + use time_manager, only : get_curr_calday + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + integer :: i + integer :: k + integer :: m + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + real(r8) :: calday ! day of year including fraction + real(r8) :: dels + integer :: last + integer :: next + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! If any columns remain to be indentified, the nget the current + ! day from the calendar. + + if (any(tropLev == NOTFOUND)) then + + ! Determine the calendar day. + calday = get_curr_calday() + + !-------------------------------------------------------- + ! ... setup the time interpolation + !-------------------------------------------------------- + if( calday < days(1) ) then + next = 1 + last = 12 + dels = (365._r8 + calday - days(12)) / (365._r8 + days(1) - days(12)) + else if( calday >= days(12) ) then + next = 1 + last = 12 + dels = (calday - days(12)) / (365._r8 + days(1) - days(12)) + else + do m = 11,1,-1 + if( calday >= days(m) ) then + exit + end if + end do + last = m + next = m + 1 + dels = (calday - days(m)) / (days(m+1) - days(m)) + end if + + dels = max( min( 1._r8,dels ),0._r8 ) + + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + !-------------------------------------------------------- + ! ... get tropopause level from climatology + !-------------------------------------------------------- + ! Interpolate the tropopause pressure. + tP = tropp_p_loc(i,lchnk,last) & + + dels * (tropp_p_loc(i,lchnk,next) - tropp_p_loc(i,lchnk,last)) + + ! Find the associated level. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end do + end if + + return + end subroutine tropopause_climate + + !----------------------------------------------------------------------- + !----------------------------------------------------------------------- + subroutine tropopause_hybridstobie(pstate, tropLev, tropP, tropT, tropZ) + use cam_history, only : outfld + + !----------------------------------------------------------------------- + ! Originally written by Philip Cameron-Smith, LLNL + ! + ! Stobie-Linoz hybrid: the highest altitude of + ! a) Stobie algorithm, or + ! b) minimum Linoz pressure. + ! + ! NOTE: the ltrop(i) gridbox itself is assumed to be a STRATOSPHERIC gridbox. + !----------------------------------------------------------------------- + !----------------------------------------------------------------------- + ! ... Local variables + !----------------------------------------------------------------------- + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + real(r8),parameter :: min_Stobie_Pressure= 40.E2_r8 !For case 2 & 4. [Pa] + real(r8),parameter :: max_Linoz_Pressure =208.E2_r8 !For case 4. [Pa] + + integer :: i, k, ncol + real(r8) :: stobie_min, shybrid_temp !temporary variable for case 2 & 3. + integer :: ltrop_linoz(pcols) !Lowest possible Linoz vertical level + integer :: ltrop_trop(pcols) !Tropopause level for hybrid case. + logical :: ltrop_linoz_set !Flag that lowest linoz level already found. + real(r8) :: trop_output(pcols,pver) !For output purposes only. + real(r8) :: trop_linoz_output(pcols,pver) !For output purposes only. + real(r8) :: trop_trop_output(pcols,pver) !For output purposes only. + + ! write(iulog,*) 'In set_chem_trop, o3_ndx =',o3_ndx + ltrop_linoz(:) = 1 ! Initialize to default value. + ltrop_trop(:) = 1 ! Initialize to default value. + ncol = pstate%ncol + + LOOP_COL4: do i=1,ncol + + ! Skip column in which the tropopause has already been found. + not_found: if (tropLev(i) == NOTFOUND) then + + stobie_min = 1.e10_r8 ! An impossibly large number + ltrop_linoz_set = .FALSE. + LOOP_LEV: do k=pver,1,-1 + IF (pstate%pmid(i,k) < min_stobie_pressure) cycle + shybrid_temp = ALPHA * pstate%t(i,k) - Log10(pstate%pmid(i,k)) + !PJC_NOTE: the units of pmid won't matter, because it is just an additive offset. + IF (shybrid_temp= 55000._r8) then + cycle + end if + + if ((tLev == -1) .or. (stobie(k) < sTrop)) then + tLev = k + tP = pstate%pmid(i, k) + sTrop = stobie(k) + end if + end do + + if (tLev /= -1) then + tropLev(i) = tLev + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end if + end do + + return + end subroutine tropopause_stobie + + + ! This routine is an implementation of Reichler et al. [2003] done by + ! Reichler and downloaded from his web site. Minimal modifications were + ! made to have the routine work within the CAM framework (i.e. using + ! CAM constants and types). + ! + ! NOTE: I am not a big fan of the goto's and multiple returns in this + ! code, but for the moment I have left them to preserve as much of the + ! original and presumably well tested code as possible. + ! UPDATE: The most "obvious" substitutions have been made to replace + ! goto/return statements with cycle/exit. The structure is still + ! somewhat tangled. + ! UPDATE 2: "gamma" renamed to "gam" in order to avoid confusion + ! with the Fortran 2008 intrinsic. "level" argument removed because + ! a physics column is not contiguous, so using explicit dimensions + ! will cause the data to be needlessly copied. + ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + ! + ! determination of tropopause height from gridded temperature data + ! + ! reference: Reichler, T., M. Dameris, and R. Sausen (2003) + ! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + subroutine twmo(t, p, plimu, pliml, gam, trp) + + real(r8), intent(in), dimension(:) :: t, p + real(r8), intent(in) :: plimu, pliml, gam + real(r8), intent(out) :: trp + + real(r8), parameter :: deltaz = 2000.0_r8 + + real(r8) :: pmk, pm, a, b, tm, dtdp, dtdz + real(r8) :: ag, bg, ptph + real(r8) :: pm0, pmk0, dtdz0 + real(r8) :: p2km, asum, aquer + real(r8) :: pmk2, pm2, a2, b2, tm2, dtdp2, dtdz2 + integer :: level + integer :: icount, jj + integer :: j + + + trp=-99.0_r8 ! negative means not valid + + ! initialize start level + ! dt/dz + level = size(t) + pmk= .5_r8 * (p(level-1)**cnst_kap+p(level)**cnst_kap) + pm = pmk**(1/cnst_kap) + a = (t(level-1)-t(level))/(p(level-1)**cnst_kap-p(level)**cnst_kap) + b = t(level)-(a*p(level)**cnst_kap) + tm = a * pmk + b + dtdp = a * cnst_kap * (pm**cnst_ka1) + dtdz = cnst_faktor*dtdp*pm/tm + + main_loop: do j=level-1,2,-1 + pm0 = pm + pmk0 = pmk + dtdz0 = dtdz + + ! dt/dz + pmk= .5_r8 * (p(j-1)**cnst_kap+p(j)**cnst_kap) + pm = pmk**(1/cnst_kap) + a = (t(j-1)-t(j))/(p(j-1)**cnst_kap-p(j)**cnst_kap) + b = t(j)-(a*p(j)**cnst_kap) + tm = a * pmk + b + dtdp = a * cnst_kap * (pm**cnst_ka1) + dtdz = cnst_faktor*dtdp*pm/tm + ! dt/dz valid? + if (dtdz.le.gam) cycle main_loop ! no, dt/dz < -2 K/km + if (pm.gt.plimu) cycle main_loop ! no, too low + + ! dtdz is valid, calculate tropopause pressure + if (dtdz0.lt.gam) then + ag = (dtdz-dtdz0) / (pmk-pmk0) + bg = dtdz0 - (ag * pmk0) + ptph = exp(log((gam-bg)/ag)/cnst_kap) + else + ptph = pm + endif + + if (ptph.lt.pliml) cycle main_loop + if (ptph.gt.plimu) cycle main_loop + + ! 2nd test: dtdz above 2 km must not exceed gam + p2km = ptph + deltaz*(pm/tm)*cnst_faktor ! p at ptph + 2km + asum = 0.0_r8 ! dtdz above + icount = 0 ! number of levels above + + ! test until apm < p2km + in_loop: do jj=j,2,-1 + + pmk2 = .5_r8 * (p(jj-1)**cnst_kap+p(jj)**cnst_kap) ! p mean ^kappa + pm2 = pmk2**(1/cnst_kap) ! p mean + if(pm2.gt.ptph) cycle in_loop ! doesn't happen + if(pm2.lt.p2km) exit in_loop ! ptropo is valid + + a2 = (t(jj-1)-t(jj)) ! a + a2 = a2/(p(jj-1)**cnst_kap-p(jj)**cnst_kap) + b2 = t(jj)-(a2*p(jj)**cnst_kap) ! b + tm2 = a2 * pmk2 + b2 ! T mean + dtdp2 = a2 * cnst_kap * (pm2**(cnst_kap-1)) ! dt/dp + dtdz2 = cnst_faktor*dtdp2*pm2/tm2 + asum = asum+dtdz2 + icount = icount+1 + aquer = asum/float(icount) ! dt/dz mean + + ! discard ptropo ? + if (aquer.le.gam) cycle main_loop ! dt/dz above < gam + + enddo in_loop ! test next level + + trp = ptph + exit main_loop + enddo main_loop + + end subroutine twmo + + + ! This routine uses an implementation of Reichler et al. [2003] done by + ! Reichler and downloaded from his web site. This is similar to the WMO + ! routines, but is designed for GCMs with a coarse vertical grid. + subroutine tropopause_twmo(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: gam = -0.002_r8 ! K/m + real(r8), parameter :: plimu = 45000._r8 ! Pa + real(r8), parameter :: pliml = 7500._r8 ! Pa + + integer :: i + integer :: k + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + ! Use the routine from Reichler. + call twmo(pstate%t(i, :), pstate%pmid(i, :), plimu, pliml, gam, tP) + + ! if successful, store of the results and find the level and temperature. + if (tP > 0) then + + ! Find the associated level. + do k = pver, 2, -1 + if (tP >= pstate%pint(i, k)) then + tropLev(i) = k + exit + end if + end do + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + end if + end if + end do + + return + end subroutine tropopause_twmo + + ! This routine implements the WMO definition of the tropopause (WMO, 1957; Seidel and Randel, 2006). + ! This requires that the lapse rate be less than 2 K/km for an altitude range + ! of 2 km. The search starts at the surface and stops the first time this + ! criteria is met. + ! + ! NOTE: This code was modeled after the code in mo_tropopause; however, the + ! requirement that dt be greater than 0 was removed and the check to make + ! sure that the lapse rate is maintained for 2 km was added. + subroutine tropopause_wmo(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: ztrop_low = 5000._r8 ! lowest tropopause level allowed (m) + real(r8), parameter :: ztrop_high = 20000._r8 ! highest tropopause level allowed (m) + real(r8), parameter :: max_dtdz = 0.002_r8 ! max dt/dz for tropopause level (K/m) + real(r8), parameter :: min_trop_dz = 2000._r8 ! min tropopause thickness (m) + + integer :: i + integer :: k + integer :: k2 + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tP ! tropopause pressure (Pa) + real(r8) :: dt + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + + kloop: do k = pver-1, 2, -1 + + ! Skip levels below the minimum and stop if nothing is found + ! before the maximum. + if (pstate%zm(i, k) < ztrop_low) then + cycle kloop + else if (pstate%zm(i, k) > ztrop_high) then + exit kloop + end if + + ! Compare the actual lapse rate to the threshold + dt = pstate%t(i, k) - pstate%t(i, k-1) + + if (dt <= (max_dtdz * (pstate%zm(i, k-1) - pstate%zm(i, k)))) then + + ! Make sure that the lapse rate stays below the threshold for the + ! specified range. + k2loop: do k2 = k-1, 2, -1 + if ((pstate%zm(i, k2) - pstate%zm(i, k)) >= min_trop_dz) then + tP = pstate%pmid(i, k) + tropLev(i) = k + exit k2loop + end if + + dt = pstate%t(i, k) - pstate%t(i, k2) + if (dt > (max_dtdz * (pstate%zm(i, k2) - pstate%zm(i, k)))) then + exit k2loop + end if + end do k2loop + + if (tropLev(i) == NOTFOUND) then + cycle kloop + else + + ! Return the optional outputs + if (present(tropP)) tropP(i) = tP + + if (present(tropT)) then + tropT(i) = tropopause_interpolateT(pstate, i, tropLev(i), tP) + end if + + if (present(tropZ)) then + tropZ(i) = tropopause_interpolateZ(pstate, i, tropLev(i), tP) + end if + + exit kloop + end if + end if + end do kloop + end if + end do + + return + end subroutine tropopause_wmo + + ! This routine searches for the cold point tropopause, and uses a parabolic + ! fit of the coldest point and two adjacent points to interpolate the cold point + ! between model levels. + subroutine tropopause_cpp(pstate, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variables + real(r8), parameter :: ztrop_low = 5000._r8 ! lowest tropopause level allowed (m) + real(r8), parameter :: ztrop_high = 25000._r8 ! highest tropopause level allowed (m) + + integer :: i + integer :: k, firstk, lastk + integer :: k2 + integer :: ncol ! number of columns in the chunk + integer :: lchnk ! chunk identifier + real(r8) :: tZ ! tropopause height (m) + real(r8) :: tmin + real(r8) :: f0, f1, f2 + real(r8) :: x0, x1, x2 + real(r8) :: c0, c1, c2 + real(r8) :: a, b, c + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Iterate over all of the columns. + do i = 1, ncol + + firstk = 0 + lastk = pver+1 + + ! Skip column in which the tropopause has already been found. + if (tropLev(i) == NOTFOUND) then + tmin = 1e6_r8 + + kloop: do k = pver-1, 2, -1 + + ! Skip levels below the minimum and stop if nothing is found + ! before the maximum. + if (pstate%zm(i, k) < ztrop_low) then + firstk = k + cycle kloop + else if (pstate%zm(i, k) > ztrop_high) then + lastk = k + exit kloop + end if + + ! Find the coldest point + if (pstate%t(i, k) < tmin) then + tropLev(i) = k + tmin = pstate%t(i,k) + end if + end do kloop + + ! If the minimum is at the edge of the search range, then don't + ! consider this to be a minima + if ((tropLev(i) >= (firstk-1)) .or. (tropLev(i) <= (lastk+1))) then + tropLev(i) = NOTFOUND + else + + ! If returning P, Z, or T, then do a parabolic fit using the + ! cold point and it its 2 surrounding points to interpolate + ! between model levels. + if (present(tropP) .or. present(tropZ) .or. present(tropT)) then + f0 = pstate%t(i, tropLev(i)-1) + f1 = pstate%t(i, tropLev(i)) + f2 = pstate%t(i, tropLev(i)+1) + + x0 = pstate%zm(i, tropLev(i)-1) + x1 = pstate%zm(i, tropLev(i)) + x2 = pstate%zm(i, tropLev(i)+1) + + c0 = (x0-x1)*(x0-x2) + c1 = (x1-x0)*(x1-x2) + c2 = (x2-x0)*(x2-x1) + + ! Determine the quadratic coefficients of: + ! T = a * z^2 - b*z + c + a = (f0/c0 + f1/c1 + f2/c2) + b = (f0/c0*(x1+x2) + f1/c1*(x0+x2) + f2/c2*(x0+x1)) + c = f0/c0*x1*x2 + f1/c1*x0*x2 + f2/c2*x0*x1 + + ! Find the altitude of the minimum temperature + tZ = 0.5_r8 * b / a + + ! The fit should be between the upper and lower points, + ! so skip the point if the fit fails. + if ((tZ >= x0) .or. (tZ <= x2)) then + tropLev(i) = NOTFOUND + else + ! Return the optional outputs + if (present(tropP)) then + tropP(i) = tropopause_interpolateP(pstate, i, tropLev(i), tZ) + end if + + if (present(tropT)) then + tropT(i) = a * tZ*tZ - b*tZ + c + end if + + if (present(tropZ)) then + tropZ(i) = tZ + end if + end if + end if + end if + end if + end do + + return + end subroutine tropopause_cpp + + + ! Searches all the columns in the chunk and attempts to identify the tropopause. + ! Two routines can be specifed, a primary routine which is tried first and a + ! backup routine which will be tried only if the first routine fails. If the + ! tropopause can not be identified by either routine, then a NOTFOUND is returned + ! for the tropopause level, temperature and pressure. + subroutine tropopause_find(pstate, tropLev, tropP, tropT, tropZ, primary, backup) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, optional, intent(in) :: primary ! primary detection algorithm + integer, optional, intent(in) :: backup ! backup detection algorithm + integer, intent(out) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(out) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(out) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(out) :: tropZ(pcols) ! tropopause height (m) + + ! Local Variable + integer :: primAlg ! Primary algorithm + integer :: backAlg ! Backup algorithm + + ! Initialize the results to a missing value, so that the algorithms will + ! attempt to find the tropopause for all of them. + tropLev(:) = NOTFOUND + if (present(tropP)) tropP(:) = fillvalue + if (present(tropT)) tropT(:) = fillvalue + if (present(tropZ)) tropZ(:) = fillvalue + + ! Set the algorithms to be used, either the ones provided or the defaults. + if (present(primary)) then + primAlg = primary + else + primAlg = default_primary + end if + + if (present(backup)) then + backAlg = backup + else + backAlg = default_backup + end if + + ! Try to find the tropopause using the primary algorithm. + if (primAlg /= TROP_ALG_NONE) then + call tropopause_findUsing(pstate, primAlg, tropLev, tropP, tropT, tropZ) + end if + + if ((backAlg /= TROP_ALG_NONE) .and. any(tropLev(:) == NOTFOUND)) then + call tropopause_findUsing(pstate, backAlg, tropLev, tropP, tropT, tropZ) + end if + + return + end subroutine tropopause_find + + ! Searches all the columns in the chunk and attempts to identify the "chemical" + ! tropopause. This is the lapse rate tropopause, backed up by the climatology + ! if the lapse rate fails to find the tropopause at pressures higher than a certain + ! threshold. This pressure threshold depends on latitude. Between 50S and 50N, + ! the climatology is used if the lapse rate tropopause is not found at P > 75 hPa. + ! At high latitude (poleward of 50), the threshold is increased to 125 hPa to + ! eliminate false events that are sometimes detected in the cold polar stratosphere. + ! + ! NOTE: This routine was adapted from code in chemistry.F90 and mo_gasphase_chemdr.F90. + subroutine tropopause_findChemTrop(pstate, tropLev, primary, backup) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, optional, intent(in) :: primary ! primary detection algorithm + integer, optional, intent(in) :: backup ! backup detection algorithm + integer, intent(out) :: tropLev(pcols) ! tropopause level index + + ! Local Variable + real(r8), parameter :: rad2deg = 180._r8/pi ! radians to degrees conversion factor + real(r8) :: dlats(pcols) + integer :: i + integer :: ncol + integer :: backAlg + + ! First use the lapse rate tropopause. + ncol = pstate%ncol + call tropopause_find(pstate, tropLev, primary=primary, backup=TROP_ALG_NONE) + + ! Now check high latitudes (poleward of 50) and set the level to the + ! climatology if the level was not found or is at P <= 125 hPa. + dlats(:ncol) = pstate%lat(:ncol) * rad2deg ! convert to degrees + + if (present(backup)) then + backAlg = backup + else + backAlg = default_backup + end if + + do i = 1, ncol + if (abs(dlats(i)) > 50._r8) then + if (tropLev(i) .ne. NOTFOUND) then + if (pstate%pmid(i, tropLev(i)) <= 12500._r8) then + tropLev(i) = NOTFOUND + end if + end if + end if + end do + + ! Now use the backup algorithm + if ((backAlg /= TROP_ALG_NONE) .and. any(tropLev(:) == NOTFOUND)) then + call tropopause_findUsing(pstate, backAlg, tropLev) + end if + + return + end subroutine tropopause_findChemTrop + + + + + ! Call the appropriate tropopause detection routine based upon the algorithm + ! specifed. + ! + ! NOTE: It is assumed that the output fields have been initialized by the + ! caller, and only output values set to fillvalue will be detected. + subroutine tropopause_findUsing(pstate, algorithm, tropLev, tropP, tropT, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: algorithm ! detection algorithm + integer, intent(inout) :: tropLev(pcols) ! tropopause level index + real(r8), optional, intent(inout) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8), optional, intent(inout) :: tropT(pcols) ! tropopause temperature (K) + real(r8), optional, intent(inout) :: tropZ(pcols) ! tropopause height (m) + + ! Dispatch the request to the appropriate routine. + select case(algorithm) + case(TROP_ALG_ANALYTIC) + call tropopause_analytic(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_CLIMATE) + call tropopause_climate(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_STOBIE) + call tropopause_stobie(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_HYBSTOB) + call tropopause_hybridstobie(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_TWMO) + call tropopause_twmo(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_WMO) + call tropopause_wmo(pstate, tropLev, tropP, tropT, tropZ) + + case(TROP_ALG_CPP) + call tropopause_cpp(pstate, tropLev, tropP, tropT, tropZ) + + case default + write(iulog, *) 'tropopause: Invalid detection algorithm (', algorithm, ') specified.' + call endrun + end select + + return + end subroutine tropopause_findUsing + + ! This routine interpolates the pressures in the physics state to + ! find the pressure at the specified tropopause altitude. + function tropopause_interpolateP(pstate, icol, tropLev, tropZ) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropZ ! tropopause pressure (m) + real(r8) :: tropopause_interpolateP + + ! Local Variables + real(r8) :: tropP ! tropopause pressure (Pa) + real(r8) :: dlogPdZ ! dlog(p)/dZ + + ! Interpolate the temperature linearly against log(P) + + ! Is the tropopause at the midpoint? + if (tropZ == pstate%zm(icol, tropLev)) then + tropP = pstate%pmid(icol, tropLev) + + else if (tropZ > pstate%zm(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + if (tropLev > 1) then + dlogPdZ = (log(pstate%pmid(icol, tropLev)) - log(pstate%pmid(icol, tropLev - 1))) / & + (pstate%zm(icol, tropLev) - pstate%zm(icol, tropLev - 1)) + tropP = pstate%pmid(icol, tropLev) + exp((tropZ - pstate%zm(icol, tropLev)) * dlogPdZ) + end if + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + if (tropLev < pver) then + dlogPdZ = (log(pstate%pmid(icol, tropLev + 1)) - log(pstate%pmid(icol, tropLev))) / & + (pstate%zm(icol, tropLev + 1) - pstate%zm(icol, tropLev)) + tropP = pstate%pmid(icol, tropLev) + exp((tropZ - pstate%zm(icol, tropLev)) * dlogPdZ) + end if + end if + + tropopause_interpolateP = tropP + end function tropopause_interpolateP + + + + ! This routine interpolates the temperatures in the physics state to + ! find the temperature at the specified tropopause pressure. + function tropopause_interpolateT(pstate, icol, tropLev, tropP) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropP ! tropopause pressure (Pa) + real(r8) :: tropopause_interpolateT + + ! Local Variables + real(r8) :: tropT ! tropopause temperature (K) + real(r8) :: dTdlogP ! dT/dlog(P) + + ! Intrepolate the temperature linearly against log(P) + + ! Is the tropopause at the midpoint? + if (tropP == pstate%pmid(icol, tropLev)) then + tropT = pstate%t(icol, tropLev) + + else if (tropP < pstate%pmid(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + if (tropLev > 1) then + dTdlogP = (pstate%t(icol, tropLev) - pstate%t(icol, tropLev - 1)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pmid(icol, tropLev - 1))) + tropT = pstate%t(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dTdlogP + end if + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + if (tropLev < pver) then + dTdlogP = (pstate%t(icol, tropLev + 1) - pstate%t(icol, tropLev)) / & + (log(pstate%pmid(icol, tropLev + 1)) - log(pstate%pmid(icol, tropLev))) + tropT = pstate%t(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dTdlogP + end if + end if + + tropopause_interpolateT = tropT + end function tropopause_interpolateT + + + ! This routine interpolates the geopotential height in the physics state to + ! find the geopotential height at the specified tropopause pressure. + function tropopause_interpolateZ(pstate, icol, tropLev, tropP) + + implicit none + + type(physics_state), intent(in) :: pstate + integer, intent(in) :: icol ! column being processed + integer, intent(in) :: tropLev ! tropopause level index + real(r8), optional, intent(in) :: tropP ! tropopause pressure (Pa) + real(r8) :: tropopause_interpolateZ + + ! Local Variables + real(r8) :: tropZ ! tropopause geopotential height (m) + real(r8) :: dZdlogP ! dZ/dlog(P) + + ! Intrepolate the geopotential height linearly against log(P) + + ! Is the tropoause at the midpoint? + if (tropP == pstate%pmid(icol, tropLev)) then + tropZ = pstate%zm(icol, tropLev) + + else if (tropP < pstate%pmid(icol, tropLev)) then + + ! It is above the midpoint? Make sure we aren't at the top. + dZdlogP = (pstate%zm(icol, tropLev) - pstate%zi(icol, tropLev)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pint(icol, tropLev))) + tropZ = pstate%zm(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dZdlogP + else + + ! It is below the midpoint. Make sure we aren't at the bottom. + dZdlogP = (pstate%zm(icol, tropLev) - pstate%zi(icol, tropLev+1)) / & + (log(pstate%pmid(icol, tropLev)) - log(pstate%pint(icol, tropLev+1))) + tropZ = pstate%zm(icol, tropLev) + (log(tropP) - log(pstate%pmid(icol, tropLev))) * dZdlogP + end if + + tropopause_interpolateZ = tropZ + end function tropopause_interpolateZ + + + ! Output the tropopause pressure and temperature to the history files. Two sets + ! of output will be generated, one for the default algorithm and another one + ! using the default routine, but backed by a climatology when the default + ! algorithm fails. + subroutine tropopause_output(pstate) + use cam_history, only : outfld + + implicit none + + type(physics_state), intent(in) :: pstate + + ! Local Variables + integer :: i + integer :: alg + integer :: ncol ! number of cloumns in the chunk + integer :: lchnk ! chunk identifier + integer :: tropLev(pcols) ! tropopause level index + real(r8) :: tropP(pcols) ! tropopause pressure (Pa) + real(r8) :: tropT(pcols) ! tropopause temperature (K) + real(r8) :: tropZ(pcols) ! tropopause height (m) + real(r8) :: tropFound(pcols) ! tropopause found + real(r8) :: tropDZ(pcols, pver) ! relative tropopause height (m) + real(r8) :: tropPdf(pcols, pver) ! tropopause probability distribution + + ! Information about the chunk. + lchnk = pstate%lchnk + ncol = pstate%ncol + + ! Find the tropopause using the default algorithm backed by the climatology. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROP_P', tropP(:ncol), ncol, lchnk) + call outfld('TROP_T', tropT(:ncol), ncol, lchnk) + call outfld('TROP_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROP_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROP_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROP_FD', tropFound(:ncol), ncol, lchnk) + + + ! Find the tropopause using just the primary algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROPP_P', tropP(:ncol), ncol, lchnk) + call outfld('TROPP_T', tropT(:ncol), ncol, lchnk) + call outfld('TROPP_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROPP_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROPP_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROPP_FD', tropFound(:ncol), ncol, lchnk) + + ! Find the tropopause using just the cold point algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, primary=TROP_ALG_CPP, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + tropDZ(:,:) = fillvalue + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + tropDZ(i,:) = pstate%zm(i,:) - tropZ(i) + end if + end do + + call outfld('TROPF_P', tropP(:ncol), ncol, lchnk) + call outfld('TROPF_T', tropT(:ncol), ncol, lchnk) + call outfld('TROPF_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROPF_DZ', tropDZ(:ncol, :), ncol, lchnk) + call outfld('TROPF_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROPF_FD', tropFound(:ncol), ncol, lchnk) + + + ! If requested, do all of the algorithms. + if (output_all) then + + do alg = 2, TROP_NALG + + ! Find the tropopause using just the analytic algorithm. + call tropopause_find(pstate, tropLev, tropP=tropP, tropT=tropT, tropZ=tropZ, primary=alg, backup=TROP_ALG_NONE) + + tropPdf(:,:) = 0._r8 + tropFound(:) = 0._r8 + + do i = 1, ncol + if (tropLev(i) /= NOTFOUND) then + tropPdf(i, tropLev(i)) = 1._r8 + tropFound(i) = 1._r8 + end if + end do + + call outfld('TROP' // TROP_LETTER(alg) // '_P', tropP(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_T', tropT(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_Z', tropZ(:ncol), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_PD', tropPdf(:ncol, :), ncol, lchnk) + call outfld('TROP' // TROP_LETTER(alg) // '_FD', tropFound(:ncol), ncol, lchnk) + end do + end if + + return + end subroutine tropopause_output +end module tropopause diff --git a/SourceMods/src.clm/README b/SourceMods/src.clm/README deleted file mode 100644 index 8a169bd..0000000 --- a/SourceMods/src.clm/README +++ /dev/null @@ -1,9 +0,0 @@ -Put source mods for the clm library in this directory. - -WARNING: SourceMods are not kept under version control, and can easily -become out of date if changes are made to the source code on which they -are based. We only recommend using SourceMods for small, short-term -changes that just apply to one or two cases. For larger or longer-term -changes, including gradual, incremental changes towards a final -solution, we highly recommend making changes in the main source tree, -leveraging version control (git or svn). diff --git a/SourceMods/src.pop/README b/SourceMods/src.pop/README deleted file mode 100644 index 0332e82..0000000 --- a/SourceMods/src.pop/README +++ /dev/null @@ -1,9 +0,0 @@ -Put source mods for the pop library in this directory. - -WARNING: SourceMods are not kept under version control, and can easily -become out of date if changes are made to the source code on which they -are based. We only recommend using SourceMods for small, short-term -changes that just apply to one or two cases. For larger or longer-term -changes, including gradual, incremental changes towards a final -solution, we highly recommend making changes in the main source tree, -leveraging version control (git or svn). diff --git a/git_scripts/checkout_E3SM-master.sh b/git_scripts/checkout_E3SM-master.sh new file mode 100755 index 0000000..9c1c266 --- /dev/null +++ b/git_scripts/checkout_E3SM-master.sh @@ -0,0 +1,112 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Code and compilation +#readonly CHECKOUT="v2.1.0-beta.3" +readonly CHECKOUT="v2.1.0-beta.1" +readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly BRANCH="8d668b7c015a17608d59d9513e9b425ed8995e26" # v2.1.0-beta.3 as of 2022-12-16 +#readonly BRANCH="master" # v2.1.0-beta.3 as of 2022-12-16 +readonly CHERRY=( ) + +# Set paths +readonly CODE_ROOT="/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1-hosing/${CHECKOUT}" + +# --- Toggle flags for what to do ---- +do_fetch_code=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# All done +echo $'\n----- All done -----\n' + +} + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/git_scripts/checkout_E3SM.sh b/git_scripts/checkout_E3SM.sh new file mode 100755 index 0000000..76d8db9 --- /dev/null +++ b/git_scripts/checkout_E3SM.sh @@ -0,0 +1,110 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Code and compilation +readonly CHECKOUT="v2.1.0-beta.3" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +readonly BRANCH="8d668b7c015a17608d59d9513e9b425ed8995e26" # v2.1.0-beta.3 as of 2022-12-16 +readonly CHERRY=( ) + +# Set paths +readonly CODE_ROOT="/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv21beta/code/${CHECKOUT}" + +# --- Toggle flags for what to do ---- +do_fetch_code=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# All done +echo $'\n----- All done -----\n' + +} + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/git_scripts/run.v2.LR.historical_0101.sh b/git_scripts/run.v2.LR.historical_0101.sh new file mode 100644 index 0000000..47867be --- /dev/null +++ b/git_scripts/run.v2.LR.historical_0101.sh @@ -0,0 +1,463 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=chrysalis +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="WCYCL20TR" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.historical_0101" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20210702" +readonly BRANCH="master" +readonly CHERRY=( "7e4d1c9fec40ce1cf2c272d671f5d9111fa4dea7" "a5b1d42d7cd24924d0dbda95e24ad8d4556d93f1" ) # PR4349 +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/lcrc/group/e3sm/ac.forsyth2/E3SMv2/v2.LR.historical_0101/init" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0101-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +readonly CASE_ROOT="/lcrc/group/e3sm/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' # xs < 20 nodes => fits in debug queue. M80 uses 80 nodes, L uses 105. +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="M80" + readonly WALLTIME="30:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="30" # How often to stop the model, should be a multiple of REST_N + readonly REST_OPTION="nyears" + readonly REST_N="5" # How often to write a restart file + readonly RESUBMIT="4" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="/lcrc/group/e3sm/ac.forsyth2/E3SMv2/v2.LR.piControl/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +! Additional retuning + clubb_tk1 = 268.15D0 + gw_convect_hcf = 10.0 +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + + flanduse_timeseries = '${input_data_dir}/lnd/clm2/surfdata_map/landuse.timeseries_ne30np4.pg2_hist_simyr1850-2015_c210113.nc' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/git_scripts/run.v2.LR.historical_0111.sh b/git_scripts/run.v2.LR.historical_0111.sh new file mode 100755 index 0000000..2e05791 --- /dev/null +++ b/git_scripts/run.v2.LR.historical_0111.sh @@ -0,0 +1,477 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="WCYCL20TR" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.historical_0111" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20211221" +readonly BRANCH="6528b786d13e10d26cc7c90a916bcee01e74a53c" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/golaz/E3SMv2/v2.LR.historical_0111/init" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0111-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/git_scripts/v2.LR.LE_bundle.sh b/git_scripts/v2.LR.LE_bundle.sh new file mode 100755 index 0000000..d2393a8 --- /dev/null +++ b/git_scripts/v2.LR.LE_bundle.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.LE +#SBATCH --account=e3sm +#SBATCH --nodes=1618 +#SBATCH --output=/global/cscratch1/sd/golaz/E3SMv2/v2.LR.LE.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.LE_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0241 0261 0271 0281 0291 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/golaz/E3SMv2/v2.LR.historical_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="12" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/homes/g/golaz/E3SMv2/scripts + . ./v2.LR.LE_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/homes/g/golaz/E3SMv2/scripts +. ./v2.LR.LE_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.LE_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/HICCUP b/scripts/HICCUP new file mode 160000 index 0000000..9a59643 --- /dev/null +++ b/scripts/HICCUP @@ -0,0 +1 @@ +Subproject commit 9a596432728c762e7811adc3f802e8ccb7793cd0 diff --git a/scripts/HICCUP-2024 b/scripts/HICCUP-2024 new file mode 160000 index 0000000..0c99230 --- /dev/null +++ b/scripts/HICCUP-2024 @@ -0,0 +1 @@ +Subproject commit 0c99230deaa2f3254256184dfecbd2e2f86baece diff --git a/scripts/README.ENDWD_COL b/scripts/README.ENDWD_COL new file mode 100644 index 0000000..4838a11 --- /dev/null +++ b/scripts/README.ENDWD_COL @@ -0,0 +1,31 @@ +Jon Wolf: +I'm looking at what they did to "fix" the default restart file and it seems like they just stuck on an array with no values in it. This is what I get from the fixed file: + +ncdump -h v2.LR.historical_0101.elm.r.2015-01-01-00000.nc | grep -i endwb +double ENDWB_COL(column) ; +ENDWB_COL:long_name = "col-level water mass end of the time step" ; +ENDWB_COL:units = "mm" ; +ENDWB_COL:interpinic_flag = 1 ; +ENDWB_COL:interpinic_flag_meanings = "1=nearest neighbor, 2=copy directly, 3=skip" ; +ENDWB_COL:_FillValue = 1.e+36 ; +ENDWB_COL:missing_value = 1.e+36 ; + +but looking at it with ncview says "All values are missing!" + +--------------- +srcuni + +# Make a safe copy +cp v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.ORIG.nc + +# Extract variable to use as template fpr ENDWB_COL +ncks -v URBAN_HEAT v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc URBAN_HEAT.nc +ncrename -v URBAN_HEAT,ENDWB_COL URBAN_HEAT.nc +cp URBAN_HEAT.nc input.nc + +# Set ENDWB_COL to 0.0 +ncflint -O -v ENDWB_COL -w 0,0 input.nc input.nc output.nc +mv output.nc ENDWB_COL.nc + +# Append ENDWB_COL to restart file +ncks -A ENDWB_COL.nc v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc diff --git a/scripts/README.mfilt b/scripts/README.mfilt new file mode 100644 index 0000000..2f2183e --- /dev/null +++ b/scripts/README.mfilt @@ -0,0 +1,5 @@ +Please change to the following: + + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 diff --git a/scripts/create_b.e21.CESM2-LE-10-macro b/scripts/create_b.e21.CESM2-LE-10-macro deleted file mode 100755 index ba02b9a..0000000 --- a/scripts/create_b.e21.CESM2-LE-10-macro +++ /dev/null @@ -1,59 +0,0 @@ -#!/bin/csh -fx -### set env variables -setenv CESM2_LE_TOOLS_ROOT $HOME/CESM2_LE_Tools -setenv POSTPROCESS_PATH /home/jedwards/workflow/CESM_postprocessing -source $POSTPROCESS_PATH/cesm-env2/bin/activate - -setenv CESMROOT /home/jedwards/cesm2_1/ - -set COMPSET = BHISTcmip6 -set MACHINE = aleph -set RESOLN = f09_g17 - -set smbr = 2 -set embr = 2 - -@ mb = $smbr -@ me = $embr - -foreach mbr ( `seq $mb $me` ) - -@ USE_REFDATE = 1001 + ( $mbr - 1 ) * 10 - -if ($mbr < 10) then - setenv CASENAME b.e21.${COMPSET}.${RESOLN}.LE2-${USE_REFDATE}.00${mbr} -else if - ($mbr >= 10) then - setenv CASENAME b.e21.${COMPSET}.${RESOLN}.LE2-${USE_REFDATE}.0${mbr} -endif - -setenv CASEROOT /proj/$USER/$CASENAME -echo $CASEROOT - -$CESMROOT/cime/scripts/create_newcase --case $CASEROOT --res $RESOLN --mach $MACHINE --compset $COMPSET --workflow=timeseries - -cd $CASEROOT - -./case.setup - -./xmlchange RUN_REFCASE=b.e21.B1850.f09_g17.CMIP6-piControl.001 -./xmlchange RUN_REFDATE=${USE_REFDATE}-01-01 - -mv user_nl_cam user_nl_cam.`date +%m%d-%H%M` -mv user_nl_clm user_nl_clm.`date +%m%d-%H%M` -mv user_nl_cpl user_nl_cpl.`date +%m%d-%H%M` -mv user_nl_cice user_nl_cice.`date +%m%d-%H%M` - -cp $CESM2_LE_TOOLS_ROOT/user_nl_files/* $CASEROOT/ -mv SourceMods SourceMods.`date +%m%d-%H%M` -cp -r $CESM2_LE_TOOLS_ROOT/SourceMods $CASEROOT/ - -./preview_namelists - -./case.build >& bld.`date +%m%d-%H%M` - - $POSTPROCESS_PATH/cesm-env2/bin/create_postprocess --caseroot $CASEROOT -end # member loop - -exit - diff --git a/scripts/extractData/extractData.scr b/scripts/extractData/extractData.scr new file mode 100755 index 0000000..4ce7343 --- /dev/null +++ b/scripts/extractData/extractData.scr @@ -0,0 +1,27 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#bash +#source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh + +#array=( 0111 0121 ) +array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0101 0151 0201 0251 0301 ) + + +for mbr in "${array[@]}" +do + +echo ${mbr} + +cd /pscratch/sd/n/nanr/mpaso-forAixue/hist.$mbr +zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_$mbr "archive/ocn/hist/*hist.am.meridionalHeatTransport*" & + +done diff --git a/scripts/extractData/extractZstash.scr b/scripts/extractData/extractZstash.scr new file mode 100755 index 0000000..45e995a --- /dev/null +++ b/scripts/extractData/extractZstash.scr @@ -0,0 +1,58 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 10 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +mv $CASEROOT/user_nl_eam $CASEROOT/user_nl_eam-aufiresON +cp $E3SM_TOOLS_ROOT/user_nl_files/user_nl_eam-aufiresOFF $CASEROOT/user_nl_eam +./xmlchange CONTINUE_RUN=TRUE +# end in July 2021 +#./xmlchange STOP_N=13 + +# end in July 2022 +#./xmlchange STOP_N=25 + +# end in December 2024 (54 months) +./xmlchange STOP_DATE=20250101 +./xmlchange STOP_N=27 +./xmlchange RESUBMIT=1 +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/hist-v2/run.v2.LR.historical_0111.sh b/scripts/hist-v2/run.v2.LR.historical_0111.sh new file mode 100755 index 0000000..9dd7ef5 --- /dev/null +++ b/scripts/hist-v2/run.v2.LR.historical_0111.sh @@ -0,0 +1,481 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL20TR" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.historical_0111-test" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220318" +readonly BRANCH="maint-2.0" +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.piControl/archive/rest/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" + + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/toss/v1-old/setup_e3sm-hist-le.csh b/scripts/toss/v1-old/setup_e3sm-hist-le.csh new file mode 100755 index 0000000..55d8b0a --- /dev/null +++ b/scripts/toss/v1-old/setup_e3sm-hist-le.csh @@ -0,0 +1,40 @@ +#!/bin/csh -f + +set MACHINE = cori-knl + +set EXPERIMENT = hist +set RESOLN = ne30_oECv3_ICG +set RESOLNSHORT = ne30 +set COMPSET = A_WCYCL20TRS_CMIP6 +set COMPSETSHORT = 20TR_CMIP6 +set PROJECT = mp9 +set TAG = E3SM + +set mbr=1 + +if ($mbr < 10) then + setenv CASE b.e3smv1.${COMPSETSHORT}.${RESOLNSHORT}.${EXPERIMENT}.00${mbr} +else + setenv CASE b.e3smv1.${COMPSETSHORT}.${RESOLNSHORT}.${EXPERIMENT}.0${mbr} +endif + +set PATH = /global/project/projectdirs/ccsm1/people/$USER/ +set RUNDIR = $SCRATCH/$USER/$CASE/run +set BLDDIR = $SCRATCH/$USER/$CASE/bld +setenv CASEROOT $PATH/cases/e3smv1/$CASE +set TOOLSROOT = /global/homes/n/nanr/CESM_tools/e3sm/ + +setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ + +#$E3SMROOT/cime/scripts/create_newcase --case MCSP_CMT_5_coupled --compset A_WCYCL20TRS_CMIP6 --res ne30_oECv3_ICG --pecount L --handle-preexisting-dirs u --mach cori-knl --output-root /global/cscratch1/sd/sglanvil/ --script-root /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled -project mp9 + +$E3SMROOT/cime/scripts/create_newcase --case $CASE --compset $COMPSET --res $RESOLN --pecount L --handle-preexisting-dirs u --mach $MACHINE --output-root $SCRATCH --script-root $CASEROOT -project $PROJECT + +cd $CASEROOT + +cp $TOOLSROOT/SourceMods/src.cam/cam_diagnostics.F90 ./SourceMods/src.cam/ +cp $TOOLSROOT/user_nl_files/le-hist/user_nl_cam $CASEROOT/ + +./case.setup +./case.build + diff --git a/scripts/toss/v1-old/setup_e3sm-le-v1-SSP370.csh b/scripts/toss/v1-old/setup_e3sm-le-v1-SSP370.csh new file mode 100755 index 0000000..1f6ddeb --- /dev/null +++ b/scripts/toss/v1-old/setup_e3sm-le-v1-SSP370.csh @@ -0,0 +1,67 @@ +#!/bin/csh -f + +set MACHINE = cori-knl + +set EXPERIMENT = LE-v1 +set RESOLN = ne30_oECv3_ICG +set RESOLNSHORT = ne30_oECv3_ICG +set COMPSET = A_WCYCL20TRS_CMIP6 +set COMPSET = A_WCYCLSSP370_CMIP6 +set COMPSETSHORT = 20TR_CMIP6 +set PROJECT = mp9 +set TAG = E3SM +set TAG = e3sm-le-v1 + +set mbr=2 + +if ($mbr < 10) then + setenv CASE b.e3smv1.${COMPSETSHORT}.${RESOLNSHORT}.${EXPERIMENT}.00${mbr} +else + setenv CASE b.e3smv1.${COMPSETSHORT}.${RESOLNSHORT}.${EXPERIMENT}.0${mbr} +endif + +set PATH = /global/project/projectdirs/ccsm1/people/$USER/ +set RUNDIR = $SCRATCH/$CASE/run +set BLDDIR = $SCRATCH/$CASE/bld +setenv CASEROOT $PATH/cases/e3smv1-le/$CASE +set TOOLSROOT = /global/homes/n/nanr/CESM_tools/e3sm/ +set REFCASE = 20180129.DECKv1b_piControl.ne30_oEC.edison +set RESTDIR = $SCRATCH/archive/ +set REFDATE = "0200-01-01" +set STARTDATE = "1850-01-01" + +#setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ +#setenv E3SMROOT /global/u2/x/xyhuang/e3sm-le-v1/ +setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ + +#$E3SMROOT/cime/scripts/create_newcase --case MCSP_CMT_5_coupled --compset A_WCYCL20TRS_CMIP6 --res ne30_oECv3_ICG --pecount L --handle-preexisting-dirs u --mach cori-knl --output-root /global/cscratch1/sd/sglanvil/ --script-root /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled -project mp9 + +$E3SMROOT/cime/scripts/create_newcase --case $CASE --compset $COMPSET --res $RESOLN --pecount L --handle-preexisting-dirs u --mach $MACHINE --output-root $SCRATCH --script-root $CASEROOT -project $PROJECT + +cd $CASEROOT + +cp $TOOLSROOT/SourceMods/src.cam/cam_diagnostics.F90 ./SourceMods/src.cam/ +cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_* $CASEROOT/ + +if (${mbr} == 1) then + cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_cam-mbr1 $CASEROOT/user_nl_cam +else + cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_cam-mbr2-3 $CASEROOT/user_nl_cam +endif + +./xmlchange RUN_TYPE=hybrid +./xmlchange RUN_REFCASE=$REFCASE +./xmlchange RUN_REFDATE=$REFDATE +./xmlchange RUN_STARTDATE=$STARTDATE +./xmlchange GET_REFCASE=FALSE +./xmlchange JOB_WALLCLOCK_TIME=24:00:00 +./xmlchange STOP_OPTION=nyears +./xmlchange STOP_N=3 + + +#echo "$RESTDIR/$REFCASE/${REFDATE}-00000/* $RUNDIR/" +mkdir -p $RUNDIR +cp $RESTDIR/$REFCASE/${REFDATE}-00000/* $RUNDIR/ +./case.setup +./case.build + diff --git a/scripts/toss/v1-old/setup_e3sm-le-v1.csh b/scripts/toss/v1-old/setup_e3sm-le-v1.csh new file mode 100755 index 0000000..6e8a0d8 --- /dev/null +++ b/scripts/toss/v1-old/setup_e3sm-le-v1.csh @@ -0,0 +1,92 @@ +#!/bin/csh -f + +set MACHINE = cori-knl + +set EXPERIMENT = LE-v1 +set RESOLN = ne30_oECv3_ICG +set COMPSET = A_WCYCL20TRS_CMIP6 +set COMPSETSHORT = 20TR_CMIP6 +set PROJECT = mp9 +set TAG = E3SM +set TAG = e3sm-le-v1 + +set mbr=5 + +#Start Years 210, 320, 365, 410, 480 +if (${mbr} == 1) then + set REFYEAR = "0210" + set REFDATE = $REFYEAR-"01-01" +endif +if (${mbr} == 2) then + set REFYEAR = "0480" + set REFDATE = $REFYEAR-"01-01" +endif +if (${mbr} == 3) then + set REFYEAR = "0320" + set REFDATE = $REFYEAR-"01-01" +endif +if (${mbr} == 4) then + set REFYEAR = "0365" + set REFDATE = $REFYEAR-"01-01" +endif +if (${mbr} == 5) then + set REFYEAR = "0410" + set REFDATE = $REFYEAR-"01-01" +endif + + + +if ($mbr < 10) then + setenv CASE e3smv1.${COMPSETSHORT}.${RESOLN}.${EXPERIMENT}.$REFYEAR.00${mbr} +else + setenv CASE e3smv1.${COMPSETSHORT}.${RESOLN}.${EXPERIMENT}.0${mbr} +endif + +set PATH = /global/project/projectdirs/ccsm1/people/$USER/ +set RUNDIR = $SCRATCH/$CASE/run +set BLDDIR = $SCRATCH/$CASE/bld +setenv CASEROOT $PATH/cases/e3smv1-le/$CASE +set TOOLSROOT = /global/homes/n/nanr/CESM_tools/e3sm/ +set REFCASE = 20180129.DECKv1b_piControl.ne30_oEC.edison +set RESTDIR = $SCRATCH/archive/ +set STARTDATE = "1850-01-01" + +#setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ +#setenv E3SMROOT /global/u2/x/xyhuang/e3sm-le-v1/ +setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ + + +$E3SMROOT/cime/scripts/create_newcase --case $CASE --compset $COMPSET --res $RESOLN --pecount L --handle-preexisting-dirs u --mach $MACHINE --output-root $SCRATCH --script-root $CASEROOT -project $PROJECT + +cd $CASEROOT + +cp $TOOLSROOT/SourceMods/src.cam/*.F90 ./SourceMods/src.cam/ +cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_* $CASEROOT/ + +if (${mbr} == 1) then + cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_cam-mbr1 $CASEROOT/user_nl_cam +else + cp $TOOLSROOT/user_nl_files/le-v1-catalyst/user_nl_cam-mbr2-3 $CASEROOT/user_nl_cam +endif + +./xmlchange RUN_TYPE=hybrid +./xmlchange RUN_REFCASE=$REFCASE +./xmlchange RUN_REFDATE=$REFDATE +./xmlchange RUN_STARTDATE=$STARTDATE +./xmlchange GET_REFCASE=FALSE +./xmlchange JOB_WALLCLOCK_TIME=24:00:00 --subgroup case.run +./xmlchange STOP_OPTION=nyears +./xmlchange STOP_N=2 +./xmlchange DOUT_S=TRUE + +./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive +./xmlchange JOB_QUEUE=debug --subgroup case.st_archive + + + +echo "$RESTDIR/$REFCASE/rest/${REFDATE}-00000/* $RUNDIR/" +mkdir -p $RUNDIR +cp $RESTDIR/$REFCASE/rest/${REFDATE}-00000/* $RUNDIR/ +./case.setup +./case.build + diff --git a/scripts/toss/v1-old/setup_e3sm.csh b/scripts/toss/v1-old/setup_e3sm.csh new file mode 100755 index 0000000..3717e38 --- /dev/null +++ b/scripts/toss/v1-old/setup_e3sm.csh @@ -0,0 +1,36 @@ +#!/bin/csh -f + +set MACHINE = cori-knl + +set EXPERIMENT = test +set RESOLN = ne30_oECv3_ICG +set COMPSET = A_WCYCL20TRS_CMIP6 +set PROJECT = mp9 +set TAG = E3SM + +set mbr=2 + +if ($mbr < 10) then + setenv CASE v1.${COMPSET}.${RESOLN}.${EXPERIMENT}.00${mbr} +else + setenv CASE v1.${COMPSET}.${RESOLN}.${EXPERIMENT}.0${mbr} +endif + +set PATH = /global/project/projectdirs/ccsm1/people/$USER/ +set RUNDIR = $SCRATCH/$USER/$CASE/run +set BLDDIR = $SCRATCH/$USER/$CASE/bld +setenv CASEROOT $PATH/cases/e3smv1/$CASE +set TOOLSROOT = /global/homes/n/nanr/CESM_tools/e3sm/ + +setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ + +#$E3SMROOT/cime/scripts/create_newcase --case MCSP_CMT_5_coupled --compset A_WCYCL20TRS_CMIP6 --res ne30_oECv3_ICG --pecount L --handle-preexisting-dirs u --mach cori-knl --output-root /global/cscratch1/sd/sglanvil/ --script-root /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled -project mp9 + +$E3SMROOT/cime/scripts/create_newcase --case $CASE --compset $COMPSET --res $RESOLN --pecount L --handle-preexisting-dirs u --mach $MACHINE --output-root $SCRATCH --script-root $CASEROOT -project $PROJECT + +cd $CASEROOT + +cp $TOOLSROOT/SourceMods/src.cam/cam_diagnostics.F90 ./SourceMods/src.cam/ + + + diff --git a/scripts/toss/v2.SMYLE-toss/IC/atm-IC/run.v2.LR.cami.sh b/scripts/toss/v2.SMYLE-toss/IC/atm-IC/run.v2.LR.cami.sh new file mode 100755 index 0000000..058b9ec --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/IC/atm-IC/run.v2.LR.cami.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + + array=( 0241 ) +refarray=( 1991 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${refarray[$ctr]} +echo ${refyear} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="m4195" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical_daily-cami_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-cami/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam +inithist = 'DAILY' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + ctr=$((ctr+1)) + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh b/scripts/toss/v2.SMYLE-toss/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh new file mode 100755 index 0000000..4213896 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + + array=( 0201 ) +refarray=( 1970 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${refarray[$ctr]} +echo ${refyear} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical_monthly-restarts_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-monthlyRestarts/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="3" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam +inithist = 'DAILY' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + ctr=$((ctr+1)) + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/README.levels b/scripts/toss/v2.SMYLE-toss/README.levels new file mode 100644 index 0000000..e69de29 diff --git a/scripts/toss/v2.SMYLE-toss/env_mach/env_mach_specific.xml b/scripts/toss/v2.SMYLE-toss/env_mach/env_mach_specific.xml new file mode 100644 index 0000000..5d01f82 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/env_mach/env_mach_specific.xml @@ -0,0 +1,98 @@ + + +
+ These variables control the machine dependent environment including + the paths to compilers and libraries external to cime such as netcdf, + environment variables for use in the running job should also be set here. +
+ + + char + executable name + + + char + redirect for job output + + + + /usr/share/lmod/8.3.1/init/perl + /usr/share/lmod/8.3.1/init/python + /usr/share/lmod/8.3.1/init/sh + /usr/share/lmod/8.3.1/init/csh + /usr/share/lmod/lmod/libexec/lmod perl + /usr/share/lmod/lmod/libexec/lmod python + module + module + + cray-hdf5-parallel + cray-netcdf-hdf5parallel + cray-parallel-netcdf + PrgEnv-gnu + PrgEnv-intel + PrgEnv-nvidia + PrgEnv-cray + PrgEnv-aocc + intel + intel-oneapi + cudatoolkit + craype-accel-nvidia80 + craype-accel-host + perftools-base + perftools + darshan + + + PrgEnv-gnu/8.3.3 + gcc/11.2.0 + cray-libsci/23.02.1.1 + + + PrgEnv-intel/8.3.3 + intel/2023.1.0 + + + PrgEnv-nvidia + nvidia/22.7 + cray-libsci/23.02.1.1 + + + PrgEnv-aocc + aocc/3.2.0 + cray-libsci/23.02.1.1 + + + craype-accel-host + craype/2.7.20 + cray-mpich/8.1.25 + cray-hdf5-parallel/1.12.2.3 + cray-netcdf-hdf5parallel/4.9.0.3 + cray-parallel-netcdf/1.12.3.3 + cmake/3.24.3 + + + + 1 + 1 + 128M + spread + threads + FALSE + /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch + software + MPI_Bcast + + + -1 + + + srun + + --label + -n {{ total_tasks }} -N {{ num_nodes }} + -c $SHELL{echo 256/`./xmlquery --value MAX_MPITASKS_PER_NODE`|bc} + $SHELL{if [ 128 -ge `./xmlquery --value MAX_MPITASKS_PER_NODE` ]; then echo "--cpu_bind=cores"; else echo "--cpu_bind=threads";fi;} + -m plane=$SHELL{echo `./xmlquery --value MAX_MPITASKS_PER_NODE`} + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/Problem.2019 b/scripts/toss/v2.SMYLE-toss/restarts/Problem.2019 new file mode 100644 index 0000000..663d610 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/Problem.2019 @@ -0,0 +1,14 @@ +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/05/v2.LR.historical_daily-cami_0241.eam.i.M05.diff.571.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-05.01/2019-05-01/v21.LR.SMYLE_IC.2019-05.01.eam.i.2019-05-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-05.01/pert.01/v21.LR.SMYLE_IC.pert.eam.i.2019-05-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/restarts + stat: 1 + + errput: ncflint: ERROR file "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/05/v2.LR.historical_daily-cami_0241.eam.i.M05.diff.000.nc" not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file). +ncflint: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command. + + stat: 1 + + errput: ncflint: ERROR file "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/05/v2.LR.historical_daily-cami_0241.eam.i.M05.diff.000.nc" not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file). +ncflint: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command. + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/README.generate.bash b/scripts/toss/v2.SMYLE-toss/restarts/README.generate.bash new file mode 100644 index 0000000..8114825 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/README.generate.bash @@ -0,0 +1,34 @@ +export CYLC_TASK_CYCLE_POINT="1959-11-01" + +### use bash + +### Note: Mar 27, 2024 -- NCO was giving me a library error with this set of commands, +### so I switched to hiccup_env and it works fine. Yay Walter! +### module load e4s +### spack env activate gcc +### spack load nco +### module load cudatoolkit/11.5 +### module load python +### module load python/3.9-anaconda-2021.11 + +bash +module load conda +conda activate hiccup_env + + +## Edit start years to set up +Step 1: ./create-v2.SMYLE.IC_RESTARTS.csh +Step 2: ./run_generate.sh +Step 3: ./fix_xtime.sh (change xtime to xtime.orig in mpaso.rst files) +Step 4: ./fix_ncol_d.sh (change ncol_d to ncol in eam.i files) + + +export CYLC_TASK_CYCLE_POINT=1970-11-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +export CYLC_TASK_CYCLE_POINT=1970-08-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + + +## Don't have to do this anymore, now that I've fixed the code: +# PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py diff --git a/scripts/toss/v2.SMYLE-toss/restarts/README.history b/scripts/toss/v2.SMYLE-toss/restarts/README.history new file mode 100644 index 0000000..ca2d14f --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/README.history @@ -0,0 +1,714 @@ + 1 ls -ld /global/cscratch1/sd/nanr/archive/v2.LR.historical_0* + 2 less ~/README.zstashExtract + 3 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/2015-01-01-00000/*" + 4 mv archive/rest/2015-01-01-00000/v2.LR.historical_0111.* .. + 5 cd ../../case_scripts/ + 6 ./case.submit + 7 ./xmlquery EXEROOT + 8 ls .. + 9 ls ../build/ + 10 ls -ltr ../build/ + 11 qme + 12 exit + 13 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 14 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 15 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 16 pwd + 17 ls + 18 history + 19 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 20 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 21 pwd + 22 ls -l + 23 ls zstash/ + 24 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 25 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 26 exit + 27 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 28 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0241 2>&1 | tee zstash_update_smbb-hist-0241_20230306.log + 29 bg + 30 ls -ltr + 31 jobs + 32 ls -ltr + 33 less zstash_update_smbb-hist-0241_20230306.log + 34 cd .. + 35 mkdir 261 + 36 cd 261/ + 37 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0261 2>&1 | tee zstash_update_smbb-hist-0261_20230306.log & + 38 ls -ltr ~/zstash_create_*261* + 39 less /global/homes/n/nanr/zstash_create_smbb-hist-0261_20230203.log + 40 less /global/homes/n/nanr/zstash_create_smbb-hist-0271_20230203.log + 41 cd ../ + 42 mkdir 271 + 43 cd 271/ + 44 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0271 2>&1 | tee zstash_update_smbb-hist-0271_20230306.log & + 45 ls -ltr + 46 jobs + 47 cd ../ + 48 mkdir 281 + 49 cd 281/ + 50 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0281 2>&1 | tee zstash_update_smbb-hist-0281_20230306.log + 51 bg + 52 cd .. + 53 mkdir 291 + 54 cd 291/ + 55 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0291 2>&1 | tee zstash_update_smbb-hist-0291_20230306.log + 56 bg + 57 cd .. + 58 mkdir 101 + 59 cd 101 + 60 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 2>&1 | tee zstash_update_smbb-hist-0101_20230306.log + 61 bg + 62 jobs + 63 cd ../ + 64 mkdir 151 + 65 cd 151/ + 66 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0151 2>&1 | tee zstash_update_smbb-hist-0151_20230306.log + 67 bg + 68 cd .. + 69 mkdir 201 + 70 cd 201 + 71 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 2>&1 | tee zstash_update_smbb-hist-0201_20230306.log + 72 less ~/zstash_create_*201* + 73 cd .. + 74 mkdir 251 + 75 cd 251/ + 76 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0251 2>&1 | tee zstash_update_smbb-hist-0251_20230306.log + 77 bg + 78 cd ../201 + 79 ls + 80 jobs + 81 cd .. + 82 mkdir 301 + 83 cd 301 + 84 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0301 2>&1 | tee zstash_update_smbb-hist-0301_20230306.log + 85 bg + 86 cd ../../ + 87 cd 210/ + 88 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" + 89 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" | less + 90 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*" | less + 91 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | less + 92 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | wc + 93 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "lnd/hist/*h0*" | wc + 94 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307.log + 95 bg + 96 less zstash_check_210.001-20230307.log + 97 fg + 98 ls + 99 mv zstash zstash-0ld + 100 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 101 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 102 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 103 exit + 104 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 105 less ~/README.zstashInstructions + 106 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/*" + 107 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2021-01-01-00000/*" + 108 ls + 109 cd archive/rest/2021-01-01-00000/ + 110 cp * /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/run/ + 111 scr + 112 exit + 113 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 114 ls -ltr + 115 ls -l archive/rest/ + 116 ls archive/atm/hist/*h0* + 117 b + 118 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 119 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 120 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 121 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201-new --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create-new_smbb-hist-0201_20230313.log + 122 cd .. + 123 b + 124 cd v2.LR.historical-smbb_0201/ + 125 hsi + 126 exit + 127 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 128 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 129 pwd + 130 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 131 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 132 exit + 133 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 134 cd ../../ + 135 cd 210/ + 136 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 137 hsi "ls -l /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001" + 138 ls -ltr + 139 less zstash_check_210.001-20230307b.log + 140 hsi + 141 less ~/README.zstashInstructions + 142 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 143 pwd + 144 less ~/README.zstashExtract + 145 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/rest/2101-01-01-00000/*" + 146 bg + 147 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/run/*" + 148 fg + 149 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 150 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/" + 151 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/*" + 152 exit + 153 screen + 154 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 155 pwd + 156 less ~/zstash_update_320.003_20230413.log + 157 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 158 ls + 159 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 160 pwd + 161 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413.log + 162 history + 163 pwd + 164 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413a.log + 165 exit + 166 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 167 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/archive/rest/*" + 168 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/\archive/rest/*" + 169 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" + 170 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" > doh + 171 less doh + 172 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 173 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000" + 174 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 175 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000/*" + 176 cd ../ + 177 mkdir v2.LR.historical_0121 + 178 cd v2.LR.historical_0121/ + 179 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0121 "archive/rest/1976-01-01-00000/*" + 180 cd .. + 181 mkdir v2.LR.historical_0131 + 182 cd v2.LR.historical_0131/ + 183 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0131 "archive/rest/1976-01-01-00000/*" + 184 cd .. + 185 mkdir v2.LR.historical_0141 + 186 cd v2.LR.historical_0141/ + 187 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0141 "archive/rest/1976-01-01-00000/*" + 188 cd .. + 189 mkdir v2.LR.historical_0161 + 190 cd v2.LR.historical_0161/ + 191 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0161 "archive/rest/1976-01-01-00000/*" + 192 cd .. + 193 mkdir v2.LR.historical_0171 + 194 cd v2.LR.historical_0171/ + 195 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0171 "archive/rest/1976-01-01-00000/*" + 196 cd .. + 197 mkdir v2.LR.historical_0181 + 198 cd v2.LR.historical_0181/ + 199 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0181 "archive/rest/1976-01-01-00000/*" + 200 cd .. + 201 mkdir v2.LR.historical_0191 + 202 cd v2.LR.historical_0191/ + 203 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0191 "archive/rest/1976-01-01-00000/*" + 204 cd .. + 205 mkdir v2.LR.historical_0211 + 206 cd v2.LR.historical_0211/ + 207 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0211 "archive/rest/1976-01-01-00000/*" + 208 cd .. + 209 mkdir v2.LR.historical_0221 + 210 cd v2.LR.historical_0221/ + 211 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0221 "archive/rest/1976-01-01-00000/*" + 212 pwd + 213 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 214 pwd + 215 zstash update --dry-run + 216 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 217 exit + 218 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 219 pwd + 220 zstash update --dry-run + 221 history + 222 bash + 223 exit + 224 pwd + 225 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 226 zstash update --dry-run + 227 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 228 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.111_20230414.log + 229 ls -ltr + 230 cd .. + 231 cd .. + 232 ls -ltr + 233 cd E3SMv2 + 234 ls -ltr + 235 cd v2.LR.SSP370_0281 + 236 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281 + 237 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.281_20230414.log + 238 ../cd v2.LR.SSP370_0211 + 239 ls + 240 pwd + 241 cd ../v2.LR.SSP370_0211/ + 242 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 + 243 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.211_20230414.log + 244 exit + 245 pwd + 246 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 247 vi ~/README.zstashExtract + 248 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0231 "archive/rest/1976-01-01-00000/*" + 249 exit + 250 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 251 vi ~/README.zstashInstructions + 252 zstash check --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 2>&1 | tee zstash_check_0251_20230320.log + 253 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 254 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 255 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230320.log + 256 screen + 257 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 258 pwd + 259 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_320.003_20230321.log + 260 $ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 261 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 262 vi ~/README.zstashInstructions + 263 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 264 pwd + 265 fg + 266 pwd + 267 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230320.log + 268 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 269 cd /global/cfs/cdirs/mp9/archive/ + 270 ls + 271 cd e3smv1 + 272 ls + 273 pwd + 274 ls -ltr + 275 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 276 cd /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0251 + 277 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_0251_20230321.log + 278 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 279 pwd + 280 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/ + 281 ls -ltr + 282 cd v2.LR.historical-smbb_0101 + 283 vi ~/README.zstashInstructions + 284 pwd + 285 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0101_20230203.log + 286 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 287 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 + 288 pwd + 289 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0201_20230314.log + 290 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 291 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 + 292 pwd + 293 ls -ltr ~/zstash* + 294 less /global/homes/n/nanr/zstash_update_smbb-ssp370-0181_20230314.log + 295 pwd + 296 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0191_20230314.log + 297 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 298 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 299 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 300 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" . 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 301 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 302 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 303 hsi ls /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 + 304 hsi ls /home/c/ccsm/E3SMv1-LE/ + 305 hsi ls -l /home/c/ccsm/E3SMv1-LE/ + 306 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 307 pwd + 308 ls + 309 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.003_20230321.log + 310 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 311 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 312 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 313 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 + 314 pwd + 315 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0171_20230314.log + 316 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 317 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 318 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0131_20230217.log + 319 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 320 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 321 pwd + 322 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230314.log + 323 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 324 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 325 pwd + 326 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230314.log + 327 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 328 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 + 329 pwd + 330 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0141_20230314.log + 331 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 332 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 + 333 pwd + 334 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0181_20230314.log + 335 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 336 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 337 pwd + 338 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230217.log + 339 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 340 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 + 341 pwd + 342 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0151_20230314.log + 343 fg + 344 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 345 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 + 346 pwd + 347 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0231_20230314.log + 348 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 349 cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 350 pwd + 351 cfs + 352 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 353 ls -ltr + 354 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230217.log + 355 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 356 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 + 357 pwd + 358 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0161_20230314.log + 359 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 360 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 361 pwd + 362 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0101_20230314.log + 363 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 364 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 365 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 366 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 + 367 pwd + 368 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0211_20230314.log + 369 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 370 pwd + 371 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 372 ls -ltr + 373 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 374 ls -ltr + 375 ls zstash/ + 376 rmdir zstash/ + 377 pwd + 378 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 379 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.002_20230413.log + 380 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 381 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 + 382 pwd + 383 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0251_20230314.log + 384 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 385 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 + 386 pwd + 387 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0301_20230314.log + 388 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 389 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 + 390 pwd + 391 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0261_20230314.log + 392 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 393 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 + 394 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0271_20230314.log + 395 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 396 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 + 397 pwd + 398 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0281_20230314.log + 399 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 400 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 401 pwd + 402 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230314.log + 403 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 404 jobs + 405 cd /global/cscratch1/sd/nanr + 406 cd checkZstash/210/ + 407 ls + 408 pwd + 409 ls + 410 mv zstash zstash-a + 411 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 412 ls -ltr + 413 hsi + 414 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 415 cd zstash + 416 pwd + 417 cd /global/cfs/cdirs/mp9/archive/ + 418 cd e3smv1 + 419 ls + 420 cd *210.001 + 421 ls -ltr + 422 cd zstash/ + 423 ls + 424 pwd + 425 less index.db + 426 sqlite3 index.db + 427 pwd + 428 cd .. + 429 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_210.001-20230307a.log + 430 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 431 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 432 pwd + 433 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230314.log + 434 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 435 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.historical-smbb_0201 + 436 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0201_20230313.log + 437 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 438 vi ~/README.zstashExtract + 439 cd ../ + 440 ls + 441 cd v2.LR.historica_0201 + 442 cd v2.LR.historical_0201 + 443 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1990-01-01-00000/*" + 444 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1970-01-01-00000/*" + 445 qme + 446 exit + 447 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 448 less extract_zstash.sh + 449 hsi ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 450 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 451 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*" + 452 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/" + 453 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/" + 454 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive" + 455 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/" + 456 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 457 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 458 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 459 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" > /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 460 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" >& /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 461 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 462 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 463 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 464 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 465 less ~/doh + 466 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 467 history + 468 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist" + 469 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 470 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm" + 471 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist" + 472 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl >& ~nanr/doh1 + 473 less ~/doh1 + 474 tcsh + 475 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 476 less ~/README.zstashInstructions + 477 hsi ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 478 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 479 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest" + 480 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "rest" + 481 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 | less + 482 cd v2.LR.SSP370_0111 + 483 ls + 484 cd archive/ + 485 ls + 486 mkdir rest + 487 cd rest + 488 ls + 489 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 490 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 491 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/*" + 492 cd ../../../v2.LR.SSP370_0121/archive/ + 493 mkdir rest; cd rest + 494 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0121 "archive/rest/2019-01-01-00000/*" + 495 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 496 less ~/README.zstashInstructions + 497 less ~/README.zstashExtract + 498 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2020-01-01-00000/*" + 499 exit + 500 tcsh + 501 tcsh + 502 exit + 503 tcsh + 504 less /global/cfs/cdirs/ + 505 cd /global/cfs/cdirs/ + 506 cd ccsm1 + 507 ls + 508 cd people + 509 cd nanr/ + 510 cd e3sm_tags + 511 ls + 512 cd E3SMv2.1 + 513 ls + 514 history + 515 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 516 mv E3SM/ E3SM-safe + 517 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 518 hsi /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 519 hsi ls /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 520 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 521 hsi "cd /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/; chmod 744 00005d.tar" + 522 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 523 wk + 524 tcsh + 525 tcsh + 526 ls -ltr + 527 ./create-v2.SMYLE.IC_RESTARTS.csh + 528 less ~/README.nco + 529 module load cray-hdf5 + 530 module load cray-netcdf + 531 module load nco + 532 ./create-v2.SMYLE.IC_RESTARTS.csh + 533 vi ~/README.zstashExtract + 534 vi create-v2.SMYLE.IC_RESTARTS.csh + 535 scp cheyenne.ucar.edu:/glade/work/nanr/cesm_tags/CASE_tools/cesm2-smyle/restarts/generate_cami_ensemble_offline.py . + 536 less ~/.alias.nanr + 537 pwd + 538 vi generate_cami_ensemble_offline.py + 539 pwd + 540 fg + 541 vi create-v2.SMYLE.IC_RESTARTS.csh + 542 vi generate_cami_ensemble_offline.py + 543 vi create-v2.SMYLE.IC_RESTARTS.csh + 544 vi generate_cami_ensemble_offline.py + 545 vi create-v2.SMYLE.IC_RESTARTS.csh + 546 vi generate_cami_ensemble_offline.py + 547 vi create-v2.SMYLE.IC_RESTARTS.csh + 548 ./create-v2.SMYLE.IC_RESTARTS.csh + 549 vi create-v2.SMYLE.IC_RESTARTS.csh + 550 vi generate_cami_ensemble_offline.py + 551 vi create-v2.SMYLE.IC_RESTARTS.csh + 552 grep CESM2_TOOLS_ROOT * + 553 vi create-v2.SMYLE.IC_RESTARTS.csh + 554 ./create-v2.SMYLE.IC_RESTARTS.csh + 555 spack env activate gcc + 556 exit + 557 source ~/.alias.bash + 558 tools + 559 cd e3sm/v2/scripts/v2.SMYLE/restarts/ + 560 ls -ltr + 561 vi generate_cami_ensemble_offline.py + 562 less create-v2.SMYLE.IC_RESTARTS.csh + 563 export CYLC_TASK_CYCLE_POINT=1959-01-01 + 564 ./generate_cami_ensemble_offline.py + 565 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime + 566 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime ./generate_cami_ensemble_offline.py + 567 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME ./generate_cami_ensemble_offline.py + 568 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 569 module avail python + 570 module load python/3.9 + 571 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 572 vi generate_cami_ensemble_offline.py + 573 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 574 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/cases/ + 575 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/ + 576 fg + 577 vi generate_cami_ensemble_offline.py + 578 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 579 ls /global/cfs/cdirs/mp9/ + 580 vi generate_cami_ensemble_offline.py + 581 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 582 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-01.01/ + 583 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 584 vi generate_cami_ensemble_offline.py + 585 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 586 history + 587 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 588 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 589 less ~/README.nco + 590 module load e4s + 591 spack env activate gcc + 592 spack load nco + 593 module load cudatoolkit/11.5 + 594 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 595 history + 596 vi README.generate.bash + 597 vi create-v2.SMYLE.IC_RESTARTS.csh + 598 ./create-v2.SMYLE.IC_RESTARTS.csh + 599 echo $SHERLL + 600 echo $SHELL + 601 vi create-v2.SMYLE.IC_RESTARTS.csh + 602 vi setEnv.scr + 603 chmod u+x setEnv.scr + 604 ./setEnv.scr + 605 echo $CYLC_TASK_CYCLE_POINT + 606 setenv CYLC_TASK_CYCLE_POINT 1960-11-01 + 607 export CYLC_TASK_CYCLE_POINT 1960-11-01 + 608 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 609 echo $CYLC_TASK_CYCLE_POINT + 610 ./generate_cami_ensemble_offline.py + 611 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 612 less README.generate.bash + 613 module load e4s + 614 spack env activate gcc + 615 spack load nco + 616 module load cudatoolkit/11.5 + 617 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 618 ls -l /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 619 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/1960-11-01/ + 620 vi generate_cami_ensemble_offline.py + 621 vi create-v2.SMYLE.IC_RESTARTS.csh + 622 ./create-v2.SMYLE.IC_RESTARTS.csh + 623 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 624 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 625 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/11/ + 626 vi generate_cami_ensemble_offline.py + 627 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 628 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 629 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 630 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 631 ./generate_cami_ensemble_offline.py + 632 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.11/pert.01/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc + 633 vi generate_cami_ensemble_offline.py + 634 ./generate_cami_ensemble_offline.py + 635 history + 636 module load e4s + 637 spack env activate gcc + 638 spack load nco + 639 module load cudatoolkit/11.5 + 640 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 641 grep slat * + 642 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 643 vi generate_cami_ensemble_offline.py + 644 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 645 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 646 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 647 ./create-v2.SMYLE.IC_RESTARTS.csh + 648 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 649 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 650 vi generate_cami_ensemble_offline.py + 651 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 652 vi generate_cami_ensemble_offline.py + 653 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 654 vi ./create-v2.SMYLE.IC_RESTARTS.csh + 655 ./create-v2.SMYLE.IC_RESTARTS.csh + 656 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 657 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 658 history + 659 vi generate_cami_ensemble_offline.py + 660 cp generate_cami_ensemble_offline.py doh.py + 661 vi doh.py + 662 fg + 663 vi doh.py + 664 vi generate_cami_ensemble_offline.py + 665 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 666 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc + 667 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.18/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 668 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.17/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 669 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.14/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 670 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.13/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 671 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.15/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 672 grep eamic * + 673 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 674 jobs + 675 vi generate_cami_ensemble_offline.py + 676 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 677 ./create-v2.SMYLE.IC_RESTARTS.csh + 678 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 679 vi generate_cami_ensemble_offline.py + 680 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 681 fg + 682 vi generate_cami_ensemble_offline.py + 683 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 684 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 685 vi generate_cami_ensemble_offline.py + 686 ./generate_cami_ensemble_offline.py + 687 vi generate_cami_ensemble_offline.py + 688 ./generate_cami_ensemble_offline.py + 689 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/ + 690 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/ + 691 vi generate_cami_ensemble_offline.py + 692 vi create-v2.SMYLE.IC_RESTARTS.csh + 693 ./generate_cami_ensemble_offline.py + 694 ls -l /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.BSMYLE.ne30np4.1960-11.001/run.002/v21.LR.SMYLE_IC.ne30np4.1960-11.01.eam.i.1960-11-01-00000.nc + 695 vi create-v2.SMYLE.IC_RESTARTS.csh + 696 vi generate_cami_ensemble_offline.py + 697 vi create-v2.SMYLE.IC_RESTARTS.csh + 698 ./create-v2.SMYLE.IC_RESTARTS.csh + 699 export CYLC_TASK_CYCLE_POINT=1962-11-01 + 700 ./generate_cami_ensemble_offline.py + 701 vi create-v2.SMYLE.IC_RESTARTS.csh + 702 vi generate_cami_ensemble_offline.py + 703 ./generate_cami_ensemble_offline.py + 704 vi generate_cami_ensemble_offline.py + 705 ./generate_cami_ensemble_offline.py + 706 vi generate_cami_ensemble_offline.py + 707 ./generate_cami_ensemble_offline.py + 708 vi generate_cami_ensemble_offline.py + 709 ./generate_cami_ensemble_offline.py + 710 history + 711 vi README.generate.bash + 712 fg + 713 history + 714 history > doh diff --git a/scripts/toss/v2.SMYLE-toss/restarts/README.paths b/scripts/toss/v2.SMYLE-toss/restarts/README.paths new file mode 100644 index 0000000..1b51379 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/README.paths @@ -0,0 +1,22 @@ +atm: /global/cscratch1/sd/sglanvil/SMYLE_atm + +OCN restarts (2018, and 1958-1965; see README) - Luke Van Roekel: +Glade: /glade/p/datashare/yeager/CATALYST/ocean_ice_ic_JRA_cycle1/ +Cori: /global/cscratch1/sd/sglanvil/SMYLE_ocn +2018 = 0061 +1958 = 0062 +1959 = 0063 +1960 = 0064 +1961 = 0065 +1962 = 0066 +1963 = 0067 +1964 = 0068 +1965 = 0069 +(lets rename at some point) + +PERTURBATIONS (001-999): +Nan Rosenbloom AMIP daily-IC run 1991-2015: /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/CAMI/ +Sasha Glanville make random perts: /global/homes/s/sglanvil/S2S/make_random_diffs.sh +Saved random perts: /global/homes/s/sglanvil/S2S/save_perts +Final pert files:: /global/cscratch1/sd/sglanvil/S2S_perts_DIFF + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/README_perts_sasha b/scripts/toss/v2.SMYLE-toss/restarts/README_perts_sasha new file mode 120000 index 0000000..6d3845e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/README_perts_sasha @@ -0,0 +1 @@ +/global/homes/s/sglanvil/S2S/README_perts \ No newline at end of file diff --git a/scripts/toss/v2.SMYLE-toss/restarts/calcYear.csh b/scripts/toss/v2.SMYLE-toss/restarts/calcYear.csh new file mode 100755 index 0000000..7a839fd --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/calcYear.csh @@ -0,0 +1,48 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + set syr = 2021 + set eyr = 2021 + set mon = 11 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +echo 'Input year: $year == OCN year: ${ocnyr}' + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + + +echo $icefname +echo ls -l $ocndir/$poprfname + +end + +exit + + + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-05.csh b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-05.csh new file mode 100755 index 0000000..4018e7c --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-05.csh @@ -0,0 +1,161 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) + set syr = 1961 + set eyr = 1979 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +#foreach mon ( 11 ) + foreach mon ( 05 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20240326.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/M${mon}/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +#set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ +set lnddir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +# fix xtime offline: fix_xtime.sh (using bash) +# ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} +# rm $icdir/${poprfout}.xtime.nc + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE new file mode 100755 index 0000000..17cfb3b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE @@ -0,0 +1,156 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST + +# Nan needs to update +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-11.csh b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-11.csh new file mode 100755 index 0000000..b93fb48 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS-11.csh @@ -0,0 +1,161 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) + set syr = 2019 + set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) + foreach mon ( 11 ) +#foreach mon ( 05 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/M${mon}/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +#set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ +set lnddir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +# fix xtime offline: fix_xtime.sh (using bash) +# ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} +# rm $icdir/${poprfout}.xtime.nc + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS.csh-OLD b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS.csh-OLD new file mode 100755 index 0000000..125ca2c --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/create-v2.SMYLE.IC_RESTARTS.csh-OLD @@ -0,0 +1,166 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) + set syr = 1970 + set eyr = 1975 +# set syr = 1976 +# set eyr = 1980 +# set syr = 1981 +# set eyr = 2000 +#set syr = 2001 +#set eyr = 2001 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +# foreach mon ( 11 ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +# fix xtime offline: fix_xtime.sh (using bash) +# ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} +# rm $icdir/${poprfout}.xtime.nc + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/fix_ncol_d.sh b/scripts/toss/v2.SMYLE-toss/restarts/fix_ncol_d.sh new file mode 100755 index 0000000..e8f7609 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/fix_ncol_d.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=08 +mon=05 +mon=11 + +for year in {2019..2019}; do +for pert in {01..20}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + eamfile=v21.LR.SMYLE_IC.pert.eam.i.$year-${mon}-01-00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/pert.${pert} + echo $case + echo $icdir + ncrename -d ncol_d,ncol $icdir/${eamfile} +done +done + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/fix_xtime.sh b/scripts/toss/v2.SMYLE-toss/restarts/fix_xtime.sh new file mode 100755 index 0000000..ce7d4c1 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/fix_xtime.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +#mon=08 +mon=05 +mon=11 + +for year in {2019..2019}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + poprfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpaso.rst.$year-${mon}-01_00000.nc + icerfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpassi.rst.$year-${mon}-01_00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 + echo $case + echo $icdir + mv $icdir/${poprfout} $icdir/${poprfout}.xtime.nc + mv $icdir/${icerfout} $icdir/${icerfout}.xtime.nc + ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} + ncrename -v xtime,xtime.orig $icdir/${icerfout}.xtime.nc $icdir/${icerfout} +done + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline-2019.py b/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline-2019.py new file mode 100755 index 0000000..7b109b1 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline-2019.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +#PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools + +if e3smroot is None: + raise SystemExit("ERROR: CESM_ROOT must be defined in environment") + +# This is needed for globus_sdk +#_LIBDIR=os.path.join(os.environ.get("HOME"),".local","lib","python3.6","site-packages") +#sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, random, threading, time, shutil +from standard_script_setup import * +from CIME.utils import run_cmd, safe_copy, expect +from argparse import RawTextHelpFormatter +#from globus_utils import * + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.environ.get("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model,int(args.ensemble_start),int(args.ensemble_end) + +def get_rvals(date, ensemble_start,ensemble_end, model): + random.seed(int(date[0:4])+int(date[5:7])+int(date[8:10])) + rvals = random.sample(range(1001),k=ensemble_end//2) + print("Rvals are {}".format(rvals)) + #rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/","cases","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + if not os.path.isfile(rvals_file): + with open(rvals_file,"w") as fd: + fd.write("{}".format(rvals)) + + return rvals + +def create_eam_ic_perturbed(original, ensemble_start,ensemble_end, date, baserundir, model, outroot="v21.LR.SMYLE_IC.pert.eam.i.", factor=0.15): + rvals = get_rvals(date, ensemble_start,ensemble_end, model) + + outfile = os.path.join(baserundir,outroot+date+"-00000.nc") + # first link the original ic file to the 0th ensemble member + if os.path.exists(outfile): + os.unlink(outfile) + expect(os.path.isfile(original),"ERROR file {} not found".format(original)) + #print("Linking {} to {}".format(original, outfile)) + #rundir = os.path.dirname(outfile) + #print("Linking {} to {}".format(original, os.path.join(rundir,os.path.basename(original)))) + #if os.path.isdir(rundir): + #shutil.rmtree(rundir) + #os.makedirs(rundir) + #os.symlink(original, outfile) + #os.symlink(original, os.path.join(rundir,os.path.basename(original))) + + # for each pair of ensemble members create an ic file with same perturbation opposite sign + month = date[5:7] + year = date[0:3] + + local_path = "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF" + perturb_files = [] + for i in range(ensemble_start,ensemble_end, 2): + print ("HERE rvals[{}] = {}".format((i-1)//2,rvals[(i-1)//2])) + perturb_file = os.path.join("{}".format(month), + "v2.LR.historical_daily-cami_0241.eam.i.M{}.diff.{:03}.nc".format(month,rvals[(i-1)//2])) + dirname = os.path.dirname(os.path.join(local_path,perturb_file)) + if not os.path.isdir(dirname): + print("Creating directory {}".format(dirname)) + os.makedirs(dirname) + perturb_files.append(perturb_file) + + pertroot = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","pert.01") + + for i in range(ensemble_start,ensemble_end, 2): + pfile = os.path.join(local_path, perturb_files.pop(0)) + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-tmp.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-tmp.nc") + print("Creating perturbed init file {}".format(outfile1)) + print("Creating perturbed init file {}".format(outfile2)) + print("Using perturb_file {}".format(pfile)) + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile1, factor)) + t.start() + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile2, -1*factor)) + t.start() + while(threading.active_count() > 1): + time.sleep(1) + + if False: + for i in range(ensemble_start, ensemble_end, 2): + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-00000.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-00000.nc") + outdir1 = baserundir[:-3]+"{:03d}".format(i) + outdir2 = baserundir[:-3]+"{:03d}".format(i+1) + origfile = os.path.basename(original) + print("{} {} ".format(outfile1, os.path.join(outdir1,origfile))) + print("{} {} ".format(outfile2, os.path.join(outdir2,origfile))) + for outdir in (outdir1,outdir2): + if not os.path.isdir(outdir): + os.mkdir(outdir) + print("outdir = {} ".format(outdir)) + if i != 1: + if os.path.isfile(os.path.join(outdir1,origfile)): + os.unlink(os.path.join(outdir1,origfile)) + os.symlink(outfile1, os.path.join(outdir1,origfile)) + print("I made it here = {} ".format(outdir)) + if os.path.isfile(os.path.join(outdir2,origfile)): + os.unlink(os.path.join(outdir2,origfile)) + os.symlink(outfile2, os.path.join(outdir2,origfile)) + + +def create_perturbed_init_file(original, perturb_file, outfile, weight): + ncflint = "ncflint" + if not os.path.isdir(os.path.dirname(outfile)): + os.makedirs(os.path.dirname(outfile)) + pertfile = outfile.replace("-tmp.nc","-00000.nc") + if os.path.isfile(pertfile): + print("Found existing file {}".format(pertfile)) + return # file exists nothing more to do + safe_copy(original, outfile) + if "BWHIST" in original: + cmd = ncflint + " -A -v US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + else: + #cmd = ncflint+" -O -C -v lat,lon,slat,slon,lev,ilev,hyai,hybi,hyam,hybm,US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + cmd = ncflint+" -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + run_cmd(cmd, verbose=True) + if os.path.isfile(outfile): + os.rename(outfile, outfile.replace("-tmp.nc","-00000.nc")) + else: + print("Rename of {} failed".format(outfile)) + +def _main_func(description): + date, model,ensemble_start,ensemble_end = parse_command_line(sys.argv, description) + + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","{}".format(date)) + user = os.getenv("USER") + baserundir = os.path.join("/pscratch/sd/n/{}/".format(user),"v21.SMYLE","v21.LR.BSMYLE."+date[0:7]+".001","run.{:03d}".format(ensemble_start)) + eaminame = os.path.join(sdrestdir,"v21.LR.SMYLE_IC.{}.01.eam.i.{date}-00000.nc".format(date[:7],date=date)) + outroot = "v21.LR.SMYLE_IC.pert.eam.i." + + create_eam_ic_perturbed(eaminame,ensemble_start,ensemble_end, date,baserundir, model, outroot=outroot) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline.py b/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline.py new file mode 100755 index 0000000..7b109b1 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/generate_cami_ensemble_offline.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +#PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools + +if e3smroot is None: + raise SystemExit("ERROR: CESM_ROOT must be defined in environment") + +# This is needed for globus_sdk +#_LIBDIR=os.path.join(os.environ.get("HOME"),".local","lib","python3.6","site-packages") +#sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, random, threading, time, shutil +from standard_script_setup import * +from CIME.utils import run_cmd, safe_copy, expect +from argparse import RawTextHelpFormatter +#from globus_utils import * + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.environ.get("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model,int(args.ensemble_start),int(args.ensemble_end) + +def get_rvals(date, ensemble_start,ensemble_end, model): + random.seed(int(date[0:4])+int(date[5:7])+int(date[8:10])) + rvals = random.sample(range(1001),k=ensemble_end//2) + print("Rvals are {}".format(rvals)) + #rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/","cases","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + if not os.path.isfile(rvals_file): + with open(rvals_file,"w") as fd: + fd.write("{}".format(rvals)) + + return rvals + +def create_eam_ic_perturbed(original, ensemble_start,ensemble_end, date, baserundir, model, outroot="v21.LR.SMYLE_IC.pert.eam.i.", factor=0.15): + rvals = get_rvals(date, ensemble_start,ensemble_end, model) + + outfile = os.path.join(baserundir,outroot+date+"-00000.nc") + # first link the original ic file to the 0th ensemble member + if os.path.exists(outfile): + os.unlink(outfile) + expect(os.path.isfile(original),"ERROR file {} not found".format(original)) + #print("Linking {} to {}".format(original, outfile)) + #rundir = os.path.dirname(outfile) + #print("Linking {} to {}".format(original, os.path.join(rundir,os.path.basename(original)))) + #if os.path.isdir(rundir): + #shutil.rmtree(rundir) + #os.makedirs(rundir) + #os.symlink(original, outfile) + #os.symlink(original, os.path.join(rundir,os.path.basename(original))) + + # for each pair of ensemble members create an ic file with same perturbation opposite sign + month = date[5:7] + year = date[0:3] + + local_path = "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF" + perturb_files = [] + for i in range(ensemble_start,ensemble_end, 2): + print ("HERE rvals[{}] = {}".format((i-1)//2,rvals[(i-1)//2])) + perturb_file = os.path.join("{}".format(month), + "v2.LR.historical_daily-cami_0241.eam.i.M{}.diff.{:03}.nc".format(month,rvals[(i-1)//2])) + dirname = os.path.dirname(os.path.join(local_path,perturb_file)) + if not os.path.isdir(dirname): + print("Creating directory {}".format(dirname)) + os.makedirs(dirname) + perturb_files.append(perturb_file) + + pertroot = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","pert.01") + + for i in range(ensemble_start,ensemble_end, 2): + pfile = os.path.join(local_path, perturb_files.pop(0)) + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-tmp.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-tmp.nc") + print("Creating perturbed init file {}".format(outfile1)) + print("Creating perturbed init file {}".format(outfile2)) + print("Using perturb_file {}".format(pfile)) + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile1, factor)) + t.start() + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile2, -1*factor)) + t.start() + while(threading.active_count() > 1): + time.sleep(1) + + if False: + for i in range(ensemble_start, ensemble_end, 2): + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-00000.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-00000.nc") + outdir1 = baserundir[:-3]+"{:03d}".format(i) + outdir2 = baserundir[:-3]+"{:03d}".format(i+1) + origfile = os.path.basename(original) + print("{} {} ".format(outfile1, os.path.join(outdir1,origfile))) + print("{} {} ".format(outfile2, os.path.join(outdir2,origfile))) + for outdir in (outdir1,outdir2): + if not os.path.isdir(outdir): + os.mkdir(outdir) + print("outdir = {} ".format(outdir)) + if i != 1: + if os.path.isfile(os.path.join(outdir1,origfile)): + os.unlink(os.path.join(outdir1,origfile)) + os.symlink(outfile1, os.path.join(outdir1,origfile)) + print("I made it here = {} ".format(outdir)) + if os.path.isfile(os.path.join(outdir2,origfile)): + os.unlink(os.path.join(outdir2,origfile)) + os.symlink(outfile2, os.path.join(outdir2,origfile)) + + +def create_perturbed_init_file(original, perturb_file, outfile, weight): + ncflint = "ncflint" + if not os.path.isdir(os.path.dirname(outfile)): + os.makedirs(os.path.dirname(outfile)) + pertfile = outfile.replace("-tmp.nc","-00000.nc") + if os.path.isfile(pertfile): + print("Found existing file {}".format(pertfile)) + return # file exists nothing more to do + safe_copy(original, outfile) + if "BWHIST" in original: + cmd = ncflint + " -A -v US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + else: + #cmd = ncflint+" -O -C -v lat,lon,slat,slon,lev,ilev,hyai,hybi,hyam,hybm,US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + cmd = ncflint+" -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + run_cmd(cmd, verbose=True) + if os.path.isfile(outfile): + os.rename(outfile, outfile.replace("-tmp.nc","-00000.nc")) + else: + print("Rename of {} failed".format(outfile)) + +def _main_func(description): + date, model,ensemble_start,ensemble_end = parse_command_line(sys.argv, description) + + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","{}".format(date)) + user = os.getenv("USER") + baserundir = os.path.join("/pscratch/sd/n/{}/".format(user),"v21.SMYLE","v21.LR.BSMYLE."+date[0:7]+".001","run.{:03d}".format(ensemble_start)) + eaminame = os.path.join(sdrestdir,"v21.LR.SMYLE_IC.{}.01.eam.i.{date}-00000.nc".format(date[:7],date=date)) + outroot = "v21.LR.SMYLE_IC.pert.eam.i." + + create_eam_ic_perturbed(eaminame,ensemble_start,ensemble_end, date,baserundir, model, outroot=outroot) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/toss/v2.SMYLE-toss/restarts/run_generate-2019.sh b/scripts/toss/v2.SMYLE-toss/restarts/run_generate-2019.sh new file mode 100755 index 0000000..b1ef87e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/run_generate-2019.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=11 +mon=05 +#for year in {1970..2019}; do +for year in {2019..2019}; do + export CYLC_TASK_CYCLE_POINT=$year-${mon}-01 + PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline-2019.py +done + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/run_generate.sh b/scripts/toss/v2.SMYLE-toss/restarts/run_generate.sh new file mode 100755 index 0000000..f6415ff --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/run_generate.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=05 +#for year in {1970..2019}; do +for year in {1959..1969}; do + export CYLC_TASK_CYCLE_POINT=$year-${mon}-01 + PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py +done + diff --git a/scripts/toss/v2.SMYLE-toss/restarts/setEnv.scr b/scripts/toss/v2.SMYLE-toss/restarts/setEnv.scr new file mode 100755 index 0000000..3e3284a --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/restarts/setEnv.scr @@ -0,0 +1,4 @@ +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/change-user_nl_eam-2013-05_smbb.csh b/scripts/toss/v2.SMYLE-toss/scripts/M05/change-user_nl_eam-2013-05_smbb.csh new file mode 100755 index 0000000..0a2637b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/change-user_nl_eam-2013-05_smbb.csh @@ -0,0 +1,54 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb +setenv TOOLSROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 05 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + ./xmlchange STOP_N=8 + ./xmlchange REST_N=8 + + mv user_nl_eam user_nl_eam-historical + cp $TOOLSROOT/user_nl_files/smbb/user_nl_eam-SSP370smbb ./user_nl_eam + #cp $TOOLSROOT/env_mach/env_mach_specific.xml . + ./xmlchange CONTINUE_RUN=TRUE + ./xmlchange DOUT_S=TRUE + ./case.submit + + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/cp-histfiles-05.csh b/scripts/toss/v2.SMYLE-toss/scripts/M05/cp-histfiles-05.csh new file mode 100755 index 0000000..d435f68 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/cp-histfiles-05.csh @@ -0,0 +1,55 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT_OLD /global/cfs/cdirs/mp9/archive/v21.LR.SMYLEsmbb +setenv DOUT_NEW /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 05 ) + +# case name counter +set smbr = 2 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) + +set mbr_padZeros = `printf %03d $mbr` +set ARCHDIR = archive.${mbr_padZeros} +set CASEDIR = case_scripts.${mbr_padZeros} +set RUNDIR = run.${mbr_padZeros} + +#echo "================================== " +#echo $CASE +if (-d $DOUT_NEW/$CASE) then + cd $DOUT_NEW/$CASE/$RUNDIR + echo " copyin files === " $DOUT_NEW/$CASE/$ARCHDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/lnd/hist/v21.LR.BSMYLEsmbb.2013-${mon}.$mbr_padZeros.elm.h1.2014-${mon}-01-00000.nc $DOUT_NEW/$CASE/$RUNDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/atm/hist/v21.LR.BSMYLEsmbb.2013-${mon}.$mbr_padZeros.eam.h1.2014-${mon}-01-00000.nc $DOUT_NEW/$CASE/$RUNDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/rof/hist/v21.LR.BSMYLEsmbb.2013-${mon}.$mbr_padZeros.mosart.h1.2014-${mon}-02-00000.nc $DOUT_NEW/$CASE/$RUNDIR + +else + echo " missing === " $DOUT_NEW/$CASE/$ARCHDIR +endif + +cd $DOUT_NEW/$CASE/$CASEDIR +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-05.sh b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-05.sh new file mode 100755 index 0000000..795f010 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-05.sh @@ -0,0 +1,562 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +## Rerunning: 1989-2018 +useyear=2015 +usemonth=05 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-2013-05.sh b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-2013-05.sh new file mode 100755 index 0000000..3657818 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-2013-05.sh @@ -0,0 +1,553 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2013 +usemonth=05 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + REST_OPTION="nmonths" + # 2013-11 through 2014-12 = 14 months; HIST portion has 2015 in it; + # then switch to SSP370 (which has 2014 in as well) + STOP_N="20" # How often to stop the model, should be a multiple of REST_N + REST_N="20" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370-2014-05.sh b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370-2014-05.sh new file mode 100755 index 0000000..27f3809 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370-2014-05.sh @@ -0,0 +1,566 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2014 +usemonth=05 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +COMPSET="WCYCLSSP370" # SSP370 transient +#COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + +!! See README file: /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/README.GHG_CMIP_SSP370 +bndtvghg = '/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_0000-2500_c20210509.nc' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370.sh b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370.sh new file mode 100755 index 0000000..3b5dd5e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/run.v21.LR.BSMYLEsmbb-SSP370.sh @@ -0,0 +1,563 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2018 +usemonth=05 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +COMPSET="WCYCLSSP370" # SSP370 transient +#COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M05/submitRunsSMBB-2013.csh b/scripts/toss/v2.SMYLE-toss/scripts/M05/submitRunsSMBB-2013.csh new file mode 100755 index 0000000..d83ce4a --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M05/submitRunsSMBB-2013.csh @@ -0,0 +1,52 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb +setenv TOOLSROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 05 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + #cp /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb/v21.LR.BSMYLEsmbb.2014-11.001/case_scripts.001/user_nl_eam . + ./xmlchange STOP_N=6 + ./xmlchange REST_N=6 + + #diff user_nl_eam ../case_scripts.001/user_nl_eam + #./preview_namelists + ./case.submit + + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/change-STOP_N-2013-11.csh b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-STOP_N-2013-11.csh new file mode 100755 index 0000000..29eed57 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-STOP_N-2013-11.csh @@ -0,0 +1,46 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /global/cfs/cdirs/mp9/archive/v21.LR.SMYLE +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLE + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLE.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + ./xmlchange STOP_N=14 + ./xmlchange REST_N=14 + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11.csh b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11.csh new file mode 100755 index 0000000..434d90b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11.csh @@ -0,0 +1,53 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLE +setenv TOOLSROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 2 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLE.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + #./xmlchange STOP_N=14 + #./xmlchange REST_N=14 + + #mv user_nl_eam user_nl_eam-historical + #cp $TOOLSROOT/user_nl_files/user_nl_eam-SSP370 ./user_nl_eam + cp $TOOLSROOT/env_mach/env_mach_specific.xml . + ./xmlchange CONTINUE_RUN=TRUE + ./case.submit + + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11_smbb.csh b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11_smbb.csh new file mode 100755 index 0000000..919d5c1 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/change-user_nl_eam-2013-11_smbb.csh @@ -0,0 +1,53 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb +setenv TOOLSROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 2 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + ./xmlchange STOP_N=14 + ./xmlchange REST_N=14 + + mv user_nl_eam user_nl_eam-historical + cp $TOOLSROOT/user_nl_files/smbb/user_nl_eam-SSP370smbb ./user_nl_eam + #cp $TOOLSROOT/env_mach/env_mach_specific.xml . + ./xmlchange CONTINUE_RUN=TRUE + ./case.submit + + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2012-11.sh-notUsing b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2012-11.sh-notUsing new file mode 100755 index 0000000..5bf155b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2012-11.sh-notUsing @@ -0,0 +1,552 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2012 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +#CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + REST_OPTION="nmonths" + # 2012-11 through 2015-02 = 28 months; HIST portion has 2015 in it; + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2013-11.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2013-11.sh new file mode 100755 index 0000000..878eb32 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-2013-11.sh @@ -0,0 +1,553 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2013 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +#CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + REST_OPTION="nmonths" + # 2013-11 through 2014-12 = 14 months; HIST portion has 2015 in it; + # then switch to SSP370 (which has 2014 in as well) + STOP_N="14" # How often to stop the model, should be a multiple of REST_N + REST_N="14" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-SSP370.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-SSP370.sh new file mode 100755 index 0000000..9e5a554 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE-SSP370.sh @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +COMPSET="WCYCLSSP370" # SSP370 transient +#COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE.sh new file mode 100755 index 0000000..ce0a785 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLE.sh @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1983 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + #./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-2013-11.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-2013-11.sh new file mode 100755 index 0000000..a2af601 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-2013-11.sh @@ -0,0 +1,553 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2013 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + REST_OPTION="nmonths" + # 2013-11 through 2014-12 = 14 months; HIST portion has 2015 in it; + # then switch to SSP370 (which has 2014 in as well) + STOP_N="14" # How often to stop the model, should be a multiple of REST_N + REST_N="14" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370-2014.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370-2014.sh new file mode 100755 index 0000000..f9b7625 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370-2014.sh @@ -0,0 +1,566 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2014 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +COMPSET="WCYCLSSP370" # SSP370 transient +#COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + +!! See README file: /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/README.GHG_CMIP_SSP370 +bndtvghg = '/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_0000-2500_c20210509.nc' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370.sh new file mode 100755 index 0000000..96ac50b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb-SSP370.sh @@ -0,0 +1,563 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +COMPSET="WCYCLSSP370" # SSP370 transient +#COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb.sh b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb.sh new file mode 100755 index 0000000..af2b6e1 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/M11/run.v21.LR.BSMYLEsmbb.sh @@ -0,0 +1,562 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +## Rerunning: 1989-2018 +useyear=2016 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLEsmbb.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLEsmbb/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +#CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="16:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/README.testcases b/scripts/toss/v2.SMYLE-toss/scripts/README.testcases new file mode 100644 index 0000000..3d8b51e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/README.testcases @@ -0,0 +1,4 @@ +./create_newcase --case $SCRATCH/testBcase.20TR --compset=WCYCL20TR --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu + +./create_newcase --case $SCRATCH/testBcase --compset=WCYCLSSP370 --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu +./create_newcase --case $SCRATCH/testBcase2 --compset=WCYCLSSP370 --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu diff --git a/scripts/toss/v2.SMYLE-toss/scripts/audit.CFS.csh b/scripts/toss/v2.SMYLE-toss/scripts/audit.CFS.csh new file mode 100755 index 0000000..a8b550f --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/audit.CFS.csh @@ -0,0 +1,54 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv TSERIES2 /global/cfs/cdirs/mp9/archive/v21.LR.SMYLE + +# ... +set syr = 1980 +set eyr = 1998 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLE.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set ARCHDIR = archive.00${mbr} +else + set ARCHDIR = archive.0${mbr} +endif + + +#echo "================================== " +#echo $CASE +if (-d $TSERIES2/$CASE) then + cd $TSERIES2/$CASE + set t2 = `ls -lR $TSERIES2/$CASE/$ARCHDIR | wc -l` + set s2 = `du . -sh` + if ($t2 < 588 ) then + echo $CASE/$ARCHDIR " ============== " $t2 $s2 + else + echo $CASE/$ARCHDIR " === " $t2 $s2 + endif +else + echo " missing === " $CASE/$ARCHDIR +endif + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/audit.PSCRATCH.csh b/scripts/toss/v2.SMYLE-toss/scripts/audit.PSCRATCH.csh new file mode 100755 index 0000000..93d08ec --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/audit.PSCRATCH.csh @@ -0,0 +1,55 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +#setenv TSERIES2 /global/cfs/cdirs/mp9/archive/v21.LR.SMYLE +setenv TSERIES2 /pscratch/sd/n/nanr/v21.LR.SMYLE + +# ... +set syr = 1970 +set eyr = 2018 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLE.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set ARCHDIR = archive.00${mbr} +else + set ARCHDIR = archive.0${mbr} +endif + + +#echo "================================== " +#echo $CASE +if (-d $TSERIES2/$CASE) then + cd $TSERIES2/$CASE + set t2 = `ls -lR $TSERIES2/$CASE/$ARCHDIR | wc -l` + set s2 = `du . -sh` + if ($t2 < 588 ) then + echo $CASE/$ARCHDIR " ============== " $t2 $s2 + else + echo $CASE/$ARCHDIR " === " $t2 $s2 + endif +else + echo " missing === " $CASE/$ARCHDIR +endif + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py b/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py new file mode 100755 index 0000000..be28b2b --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +#s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +s2sfcstroot = "/global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/" + +# Machine and project +MACHINE="pm-cpu" +PROJECT="mp9" + +if e3smroot is None: + raise SystemExit("ERROR: E3SM_ROOT must be defined in environment") + +_LIBDIR = os.path.join(e3smroot,"cime","scripts","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts","lib") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, glob, shutil +# import CIME.build as build +from standard_script_setup import * +from CIME.case import Case +from CIME.utils import safe_copy +from argparse import RawTextHelpFormatter +from CIME.locked_files import lock_file, unlock_file + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.getenv("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model, int(args.ensemble_start), int(args.ensemble_end) + +def stage_refcase(rundir, refdir, date, basecasename): + if not os.path.isdir(rundir): + os.makedirs(rundir) + nfname = "v21.LR.SMYLE_IC.ne30np4" + #else: + # nfname = "b.e21.f09_g17" + print ("refdir="+refdir) + print ("rundir="+rundir) + for reffile in glob.iglob(refdir+"/*"): + print ("theNEWFILE="+reffile) + if os.path.basename(reffile).startswith("rpointer"): + safe_copy(reffile, rundir) + else: + newfile = os.path.basename(reffile) + newfile = os.path.join(rundir,newfile) + if "cam.i" in newfile: + if not "001" in newfile: + os.symlink(reffile,newfile.replace(".nc","-original.nc")) + else: + if os.path.lexists(newfile): + os.unlink(newfile) + os.symlink(reffile, newfile) + +def per_run_case_updates(case, date, sdrestdir): + caseroot = case.get_value("CASEROOT") + basecasename = os.path.basename(caseroot)[:-4] + member = os.path.basename(caseroot)[-3:] + + unlock_file("env_case.xml",caseroot=caseroot) + casename = basecasename+"."+member + case.set_value("CASE",casename) + case.flush() + lock_file("env_case.xml",caseroot=caseroot) + + case.set_value("CONTINUE_RUN",False) + case.set_value("BUILD_COMPLETE",True) + case.set_value("RUN_REFDATE",date) + case.set_value("RUN_STARTDATE",date) + case.set_value("RUN_REFDIR",sdrestdir) + + case.case_setup() + + stage_refcase(rundir, sdrestdir, date, basecasename) + case.set_value("BATCH_SYSTEM", "none") + case.set_value("RUNDIR", rundir+member) + safe_copy(os.path.join(caseroot,"env_batch.xml"),os.path.join(caseroot,"LockedFiles","env_batch.xml")) + + +#def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + #sdrestdir, machine, project): +def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + sdrestdir, machine): + caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start),"case_scripts") + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + with Case(caseroot, read_only=False) as case: + if not os.path.isdir(caseroot): + #case.create(os.path.basename(caseroot), e3smroot, compset, res, + # run_unsupported=True, answer="r",walltime="12:00:00", + # user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + #case.create(os.path.basename(caseroot), e3smroot, compset, res, run_unsupported=True, answer="r",walltime="12:00:00",machine="pm-cpu",project="mp9") + case.create(os.path.basename(caseroot), e3smroot, compset, res, walltime="12:00:00",machine_name="pm-cpu",driver="mct",user_mods_dirs=[user_mods_dir]) + # make sure that changing the casename will not affect these variables + user = os.getenv("USER") + cimeroot = os.path.join("/pscratch/sd/n/nanr/".format(user),"v21.LR.SMYLE") + exeroot = os.path.join("/pscratch/sd/n/nanr/".format(user),"v21.LR.SMYLE/exerootdir/bld") + case.set_value("CIME_OUTPUT_ROOT",cimeroot) + if exeroot and os.path.exists(os.path.join(exeroot,"e3sm.exe")): + case.set_value("EXEROOT",exeroot) + else: + case.set_value("EXEROOT",case.get_value("EXEROOT", resolved=True)) + + case.set_value("RUNDIR",case.get_value("RUNDIR",resolved=True)+".001") + + case.set_value("RUN_TYPE","hybrid") + # case.set_value("JOB_QUEUE","economy") + case.set_value("GET_REFCASE",False) + case.set_value("RUN_REFDIR",sdrestdir) + case.set_value("RUN_REFCASE", "v21.SMYLE_IC.ne30np4.{}.01".format(date[:7])) + case.set_value("NTHRDS", 1) + + case.set_value("STOP_OPTION","nmonths") + case.set_value("STOP_N", 24) + #case.set_value("REST_OPTION","nmonths") + #case.set_value("REST_N", 24) + + case.set_value("CCSM_BGC","CO2A") + case.set_value("EXTERNAL_WORKFLOW",True) + + nint = 3 + n2nt = 11 + rundir = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start),"run") + #rundir = case.get_value("RUNDIR") + #print("rundir 1 = {}".format(rundir)) + #rundir = rundir[:-n2nt]+"001/run.{:03d}".format(ensemble_start) + #print("rundir 2 = {}".format(rundir)) + #rundir = rundir[:-nint]+"{:03d}".format(ensemble_start) + #print("rundir 3 = {}".format(rundir)) + #case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir) + #if not exeroot: + #build.case_build(caseroot, case=case) + + return caseroot + +def clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite): + + nint=3 + cloneroot = caseroot + for i in range(ensemble_start+1, ensemble_end+1): + member_string = '{{0:0{0:d}d}}'.format(nint).format(i) + if ensemble_end > ensemble_start: + caseroot = caseroot[:-nint] + member_string + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + if not os.path.isdir(caseroot): + with Case(cloneroot, read_only=False) as clone: + clone.create_clone(caseroot, keepexe=True, + user_mods_dir=user_mods_dir) + with Case(caseroot, read_only=True) as case: + # rundir is initially 00 reset to current member + rundir = case.get_value("RUNDIR") + rundir = rundir[:-nint]+member_string + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir) + +def _main_func(description): + #date, basecasename = parse_command_line(sys.argv, description) + date, model, ensemble_start, ensemble_end = parse_command_line(sys.argv, description) + basecasename = "v21.LR.BSMYLE.ne30np4" + + # TODO make these input vars + + basemonth = int(date[5:7]) + baseyear = int(date[0:4]) + baseroot = "/pscratch/sd/n/nanr/v21.LR.SMYLE/" + res = "ne30pg2_EC30to60E2r2" + waccm = False + #if model == "e3smsmyle": + #compset = "WCYCL20TR" + compset = "WCYCL20TR" + + print ("baseyear is {} basemonth is {}".format(baseyear,basemonth)) + + overwrite = True + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC.ne30np4."+date[0:7]+".01","{}".format(date)) + machine = "pm-cpu" + project = "mp9" + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + + # END TODO + print("basemonth = {}".format(basemonth)) + #caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + #compset, overwrite, sdrestdir, machine, project) + caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + compset, overwrite, sdrestdir,machine) + clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, overwrite) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py-safe b/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py-safe new file mode 100755 index 0000000..ee5ac7c --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/buildcase-v2.SMYLE.py-safe @@ -0,0 +1,234 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +s2sfcstroot = "/global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/" + +if e3smroot is None: + raise SystemExit("ERROR: E3SM_ROOT must be defined in environment") + +_LIBDIR = os.path.join(e3smroot,"cime","scripts","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts","lib") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, glob, shutil +# import CIME.build as build +from standard_script_setup import * +from CIME.case import Case +from CIME.utils import safe_copy +from argparse import RawTextHelpFormatter +from CIME.locked_files import lock_file, unlock_file + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.getenv("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model, int(args.ensemble_start), int(args.ensemble_end) + +def stage_refcase(rundir, refdir, date, basecasename): + if not os.path.isdir(rundir): + os.makedirs(rundir) + nfname = "v21.LR.SMYLE_IC.ne30np4" + #else: + # nfname = "b.e21.f09_g17" + print ("refdir="+refdir) + print ("rundir="+rundir) + for reffile in glob.iglob(refdir+"/*"): + print ("theNEWFILE="+reffile) + if os.path.basename(reffile).startswith("rpointer"): + safe_copy(reffile, rundir) + else: + newfile = os.path.basename(reffile) + #newfile = "{}.cice.r.{}-00000.nc".format(nfname,date) + newfile = os.path.join(rundir,newfile) + if "cam.i" in newfile: + if not "001" in newfile: + os.symlink(reffile,newfile.replace(".nc","-original.nc")) + else: + if os.path.lexists(newfile): + os.unlink(newfile) + os.symlink(reffile, newfile) + +def per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir): + caseroot = case.get_value("CASEROOT") + basecasename = os.path.basename(caseroot)[:-4] + member = os.path.basename(caseroot)[-3:] + + unlock_file("env_case.xml",caseroot=caseroot) + casename = basecasename+"."+member + case.set_value("CASE",casename) + case.flush() + lock_file("env_case.xml",caseroot=caseroot) + + case.set_value("CONTINUE_RUN",False) + case.set_value("BUILD_COMPLETE",True) + case.set_value("RUN_REFDATE",date) + case.set_value("RUN_STARTDATE",date) + case.set_value("RUN_REFDIR",sdrestdir) + #case.set_value("PROJECT","NCGD0047") +# case.set_value("PROJECT","mp9") +# case.set_value("OCN_TRACER_MODULES","iage cfc ecosys") +# dout_s_root = case.get_value("DOUT_S_ROOT") +# dout_s_root = os.path.join(os.path.dirname(dout_s_root),casename) +# if dout_s_root.startswith("/glade/scratch"): +# dout_s_root = dout_s_root.replace("/glade/scratch/","/glade/p/nsc/ncgd0042/") +# case.set_value("DOUT_S_ROOT",dout_s_root) + # restage user_nl files for each run + for usermod in glob.iglob(user_mods_dir+"/user*"): + safe_copy(usermod, caseroot) + + case.case_setup() + + stage_refcase(rundir, sdrestdir, date, basecasename) + case.set_value("BATCH_SYSTEM", "none") + safe_copy(os.path.join(caseroot,"env_batch.xml"),os.path.join(caseroot,"LockedFiles","env_batch.xml")) +# lock_file("env_batch.xml",caseroot=caseroot) + + +def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + sdrestdir, user_mods_dir, pecount=None): + caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start)) + #caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".001") + #caseroot = os.path.join(baseroot,basecasename+".{}".format(date[:7])+".00") + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + + with Case(caseroot, read_only=False) as case: + if not os.path.isdir(caseroot): + #case.create(os.path.basename(caseroot), e3smroot, compset, res, + # run_unsupported=True, answer="r",walltime="12:00:00", + # user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + case.create(os.path.basename(caseroot), e3smroot, compset, res, + run_unsupported=True, answer="r",walltime="12:00:00", + user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + # make sure that changing the casename will not affect these variables + user = os.getenv("USER") + cimeroot = os.path.join("/glade/scratch/{}/".format(user),"SMYLE") + exeroot = os.path.join("/glade/scratch/{}/".format(user),"SMYLE/exerootdir/bld") + case.set_value("CIME_OUTPUT_ROOT",cimeroot) + #case.set_value("CIME_OUTPUT_ROOT","/glade/scratch/nanr/SMYLE") + if exeroot and os.path.exists(os.path.join(exeroot,"e3sm.exe")): + case.set_value("EXEROOT",exeroot) + else: + case.set_value("EXEROOT",case.get_value("EXEROOT", resolved=True)) + + case.set_value("RUNDIR",case.get_value("RUNDIR",resolved=True)+".001") + case.set_value("CAM_CONFIG_OPTS",case.get_value("CAM_CONFIG_OPTS")+" -cosp ") + + case.set_value("RUN_TYPE","hybrid") + #case.set_value("JOB_QUEUE","economy",subgroup="case.run") + case.set_value("JOB_QUEUE","economy") + case.set_value("GET_REFCASE",False) + case.set_value("RUN_REFDIR",sdrestdir) + case.set_value("RUN_REFCASE", "b.e21.SMYLE_IC.f09_g17.{}.01".format(date[:7])) + case.set_value("OCN_TRACER_MODULES","") + case.set_value("OCN_TRACER_MODULES","iage") + case.set_value("OCN_CHL_TYPE","diagnostic") + case.set_value("NTHRDS", 1) + + case.set_value("STOP_OPTION","nmonths") + case.set_value("STOP_N", 24) + #case.set_value("REST_OPTION","nmonths") + #case.set_value("REST_N", 24) + + case.set_value("CCSM_BGC","CO2A") + case.set_value("EXTERNAL_WORKFLOW",True) + case.set_value("CLM_NAMELIST_OPTS", "use_init_interp=.true.") + + nint = 3 + n2nt = 11 + rundir = case.get_value("RUNDIR") + print("rundir 1 = {}".format(rundir)) + #rundir = os.path.join("/glade/scratch/nanr/","SMYLE","b.e21.BSMYLE.f09_g17."+date[0:7]+".001", + #rundir = os.path.join(baseroot,basecasename+"."+date[:7]+".001") + rundir = rundir[:-n2nt]+"001/run.{:03d}".format(ensemble_start) + print("rundir 2 = {}".format(rundir)) + rundir = rundir[:-nint]+"{:03d}".format(ensemble_start) + print("rundir 3 = {}".format(rundir)) + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir) + #if not exeroot: + #build.case_build(caseroot, case=case) + + return caseroot + +def clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite): + + nint=3 + cloneroot = caseroot + for i in range(ensemble_start+1, ensemble_end+1): + member_string = '{{0:0{0:d}d}}'.format(nint).format(i) + if ensemble_end > ensemble_start: + caseroot = caseroot[:-nint] + member_string + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + if not os.path.isdir(caseroot): + with Case(cloneroot, read_only=False) as clone: + clone.create_clone(caseroot, keepexe=True, + user_mods_dir=user_mods_dir) + with Case(caseroot, read_only=True) as case: + # rundir is initially 00 reset to current member + rundir = case.get_value("RUNDIR") + rundir = rundir[:-nint]+member_string + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir) + +def _main_func(description): + #date, basecasename = parse_command_line(sys.argv, description) + date, model, ensemble_start, ensemble_end = parse_command_line(sys.argv, description) + basecasename = "v21.BSMYLE.ne30np4" + + # TODO make these input vars + + basemonth = int(date[5:7]) + baseyear = int(date[0:4]) + baseroot = "/pscratch/sd/n/nanr/v21.SMYLE/" + res = "ne30pg2_EC30to60E2r2" + waccm = False + if model == "e3smsmyle": + compset = "WCYCL20TR" + + print ("baseyear is {} basemonth is {}".format(baseyear,basemonth)) + + overwrite = True + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.SMYLE_IC.ne30np4."+date[0:7]+".01","{}".format(date)) + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + + # END TODO + print("basemonth = {}".format(basemonth)) + caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + compset, overwrite, sdrestdir, user_mods_dir+'.base', pecount="S") + clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/toss/v2.SMYLE-toss/scripts/change-DOUT_S_ROOT.csh b/scripts/toss/v2.SMYLE-toss/scripts/change-DOUT_S_ROOT.csh new file mode 100755 index 0000000..30c6cc3 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/change-DOUT_S_ROOT.csh @@ -0,0 +1,54 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT_OLD /global/cfs/cdirs/mp9/archive/v21.LR.SMYLEsmbb +setenv DOUT_NEW /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set ARCHDIR = archive.00${mbr} + set CASEDIR = case_scripts.00${mbr} +else + set ARCHDIR = archive.0${mbr} + set CASEDIR = case_scripts.0${mbr} +endif + + +#echo "================================== " +#echo $CASE +if (-d $DOUT_NEW/$CASE) then + cd $DOUT_NEW/$CASE/$CASEDIR + echo " changing === " $DOUT_NEW/$CASE/$ARCHDIR + echo " changing === " $DOUT_NEW/$CASE/$ARCHDIR + ./xmlquery DOUT_S_ROOT + ./xmlchange DOUT_S_ROOT=$DOUT_NEW/$CASE/$ARCHDIR +else + echo " missing === " $DOUT_NEW/$CASE/$ARCHDIR +endif + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/change-SMBB.csh b/scripts/toss/v2.SMYLE-toss/scripts/change-SMBB.csh new file mode 100755 index 0000000..fa755ee --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/change-SMBB.csh @@ -0,0 +1,47 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /global/cfs/cdirs/mp9/archive/v21.LR.SMYLE +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb + +# ... +set syr = 1989 +set eyr = 1989 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} + set RUNDIR = run.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} + set RUNDIR = run.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + ./xmlchange DOUT_S_ROOT=/pscratch/sd/n/nanr/v21.LR.SMYLEsmbb/v21.LR.BSMYLEsmbb.1989-11.001/$RUNDIR/ + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/checkSuccess.csh b/scripts/toss/v2.SMYLE-toss/scripts/checkSuccess.csh new file mode 100755 index 0000000..045ce42 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/checkSuccess.csh @@ -0,0 +1,55 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv TSERIES1 /pscratch/sd/n/nanr/v21.LR.SMYLE +setenv TSERIES2 /pscratch/sd/s/sglanvil/v21.LR.SMYLE/ +setenv TSERIES3 /global/cfs/cdirs/mp9/archive/v21.LR.SMYLE + +# ... +set syr = 1980 +set eyr = 2014 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLE.${year}-11.001 + +#echo "================================== " +#echo $CASE +if (-d $TSERIES1/$CASE) then + cd $TSERIES1/$CASE + set t2 = `ls -ld $TSERIES1/$CASE/archive.* | wc -l` + echo " NSCR $CASE ========= " $t2 +else + if (-d $TSERIES2/$CASE) then + cd $TSERIES2/$CASE + set t2 = `ls -ld $TSERIES2/$CASE/archive.* | wc -l` + echo " SSCR $CASE ========= " $t2 + else + if (-d $TSERIES3/$CASE) then + cd $TSERIES3/$CASE + set t2 = `ls -ld $TSERIES3/$CASE/archive.* | wc -l` + echo " mCFS $CASE ========= " $t2 + else + echo " missing === " $CASE/ + endif + endif +endif + +#end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/cp-histfiles.csh b/scripts/toss/v2.SMYLE-toss/scripts/cp-histfiles.csh new file mode 100755 index 0000000..de51122 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/cp-histfiles.csh @@ -0,0 +1,65 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT_OLD /global/cfs/cdirs/mp9/archive/v21.LR.SMYLEsmbb +setenv DOUT_NEW /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb + +# ... +set syr = 2013 +set eyr = 2013 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 3 +set embr = 19 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) + +set mbr_padZeros = `printf %03d $mbr` +set ARCHDIR = archive.${mbr_padZeros} +set CASEDIR = case_scripts.${mbr_padZeros} +set RUNDIR = run.${mbr_padZeros} +#if ($mbr < 10) then + #set ARCHDIR = archive.00${mbr} + #set CASEDIR = case_scripts.00${mbr} + #set RUNDIR = run.00${mbr} +#else + #set ARCHDIR = archive.0${mbr} + #set CASEDIR = case_scripts.0${mbr} + #set RUNDIR = run.0${mbr} +#endif + + +#echo "================================== " +#echo $CASE +if (-d $DOUT_NEW/$CASE) then + cd $DOUT_NEW/$CASE/$RUNDIR + echo " copyin files === " $DOUT_NEW/$CASE/$ARCHDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/lnd/hist/v21.LR.BSMYLEsmbb.2013-11.$mbr_padZeros.elm.h1.2014-11-01-00000.nc $DOUT_NEW/$CASE/$RUNDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/atm/hist/v21.LR.BSMYLEsmbb.2013-11.$mbr_padZeros.eam.h1.2014-11-01-00000.nc $DOUT_NEW/$CASE/$RUNDIR + cp $DOUT_NEW/$CASE/$ARCHDIR/rof/hist/v21.LR.BSMYLEsmbb.2013-11.$mbr_padZeros.mosart.h1.2014-11-02-00000.nc $DOUT_NEW/$CASE/$RUNDIR + +else + echo " missing === " $DOUT_NEW/$CASE/$ARCHDIR +endif + +cd $DOUT_NEW/$CASE/$CASEDIR +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase new file mode 100755 index 0000000..3da9c9a --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase @@ -0,0 +1,468 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="I20TRCRUELMBGC" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.${COMPSET}_test" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +#MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +#START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +#GET_REFCASE=TRUE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +#RUN_REFCASE="v2.LR.historical_${iyr}" +#RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + PELAYOUT="L" + WALLTIME="30:00:00" + STOP_OPTION="nyears" + STOP_N="30" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM-daily.i.Files.sh b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM-daily.i.Files.sh new file mode 100755 index 0000000..5aaead5 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM-daily.i.Files.sh @@ -0,0 +1,454 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="20TR_DATM%CRU_ELM%SP_SICE_SOCN_MOSART_SGLC_SWAV_SIAC_SESP" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.I20TRELM_CRUNCEP-daily" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1958-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=true +RUN_REFDIR="/pscratch/sd/n/nanr/archive/s2sLandSpinupSE_perl_ICRUELM/rest/0401-01-01-00000" +RUN_REFCASE="s2sLandSpinupSE_perl_ICRUELM" +RUN_REFDATE="0401-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="ndays" + REST_N="1" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + ./xmlchange DATM_CLMNCEP_YR_ALIGN=1958 + ./xmlchange DATM_CLMNCEP_YR_START=1958 + ./xmlchange DATM_CLMNCEP_YR_END=2010 + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM.sh b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM.sh new file mode 100755 index 0000000..9c858dd --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/elm-Spinup/run.v21.LR.I20TRELM.sh @@ -0,0 +1,447 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="20TR_DATM%CRU_ELM%SP_SICE_SOCN_MOSART_SGLC_SWAV_SIAC_SESP" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.I20TRELM_CRUNCEP" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1958-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=true +RUN_REFDIR="/pscratch/sd/n/nanr/archive/s2sLandSpinupSE_perl_ICRUELM/rest/0401-01-01-00000" +RUN_REFCASE="s2sLandSpinupSE_perl_ICRUELM" +RUN_REFDATE="0401-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + ./xmlchange DATM_CLMNCEP_YR_ALIGN=1958 + ./xmlchange DATM_CLMNCEP_YR_START=1958 + ./xmlchange DATM_CLMNCEP_YR_END=2010 + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/fix.test b/scripts/toss/v2.SMYLE-toss/scripts/fix.test new file mode 100644 index 0000000..7d1d665 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/fix.test @@ -0,0 +1,12 @@ +./xmlchange RUN_REFCASE=v21.LR.SMYLE_IC.ne30np4.1972-11.01 +./xmlquery RUN_REFDATE +./xmlchange RUN_REFDATE=1972-11-01 +./xmlchange RUN_STARTDATE=1972-11-01 +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/v2* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase.20TR/run/ +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/rpointer.* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase.20TR/run/ + +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/v2* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/rpointer.* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +cd /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +rm v21.LR.BSMYLE.1972-11.001.mpas* + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/run-st.archive.sh b/scripts/toss/v2.SMYLE-toss/scripts/run-st.archive.sh new file mode 100755 index 0000000..61a185e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/run-st.archive.sh @@ -0,0 +1,128 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1987 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +#CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} +CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + +# Make directories created by this script world-readable +umask 022 + +# st_archive +case_archive + +# All done +echo $'\n----- All done -----\n' + +} + + +#----------------------------------------------------- +case_archive() { + + echo $'\n----- Starting case_archive -----\n' + echo ${CASE_SCRIPTS_DIR} + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.st_archive + + popd +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/submitArchiver.csh b/scripts/toss/v2.SMYLE-toss/scripts/submitArchiver.csh new file mode 100755 index 0000000..a301054 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/submitArchiver.csh @@ -0,0 +1,44 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb + +# ... +set syr = 1989 +set eyr = 1989 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 2 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + ./case.st_archive + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/submitRunsSMBB.csh b/scripts/toss/v2.SMYLE-toss/scripts/submitRunsSMBB.csh new file mode 100755 index 0000000..6925cb4 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/submitRunsSMBB.csh @@ -0,0 +1,52 @@ +#!/bin/csh +### set env variables +### module load ncl nco + +setenv DOUT /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb +setenv TOOLSROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ + +# ... +set syr = 2014 +set eyr = 2014 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 11 ) + +# case name counter +set smbr = 3 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set CASE = v21.LR.BSMYLEsmbb.${year}-11.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASEDIR = case_scripts.00${mbr} +else + set CASEDIR = case_scripts.0${mbr} +endif + + +echo "================================== " +#echo $CASE + cd $DOUT/$CASE/$CASEDIR + cp /pscratch/sd/n/nanr/v21.LR.SMYLEsmbb/v21.LR.BSMYLEsmbb.2014-11.001/case_scripts.001/user_nl_eam . + #./xmlchange STOP_N=14 + #./xmlchange REST_N=14 + + #diff user_nl_eam ../case_scripts.001/user_nl_eam + #./preview_namelists + ./case.submit + + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE-TESTME.sh-NotUsing b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE-TESTME.sh-NotUsing new file mode 100755 index 0000000..fc5cc99 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE-TESTME.sh-NotUsing @@ -0,0 +1,506 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1977 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +#CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=false +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +#user_nl() { + + +#cat << EOF >> user_nl_elm + +#! Override +#check_finidat_fsurdat_consistency = .false. +#check_dynpft_consistency = .false. + +#EOF + + +#} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + #${CODE_ROOT}/cime/scripts/create_newcase \ + #--case ${CASE_NAME} \ + #--case-group ${CASE_GROUP} \ + #--output-root ${CASE_ROOT} \ + #--script-root ${CASE_SCRIPTS_DIR} \ + #--handle-preexisting-dirs u \ + #--compset ${COMPSET} \ + #--res ${RESOLUTION} \ + #--machine ${MACHINE} \ + #--project ${PROJECT} \ + #--walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + echo "${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/" + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + # rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + # ln -s ${ics}/pert.${shortmbr}/${eamic} $MAIN_CASE_ROOT/run.${mbr}/ + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + # ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + #if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ## ./xmlchange DEBUG=${DEBUG_COMPILE^^} + #fi + + # Run CIME case.build + # ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + # ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + # ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + # ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + # ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + # ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + # ./xmlchange BUDGETS=TRUE + + # Set resubmissions + #if (( RESUBMIT > 0 )); then + # ./xmlchange RESUBMIT=${RESUBMIT} + #fi + + # Run type + # Start from default of user-specified initial conditions + #if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + # ./xmlchange RUN_TYPE="startup" + # ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + #elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + # ./xmlchange CONTINUE_RUN="TRUE" + + #elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + # ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + # ./xmlchange GET_REFCASE=${GET_REFCASE} + # ./xmlchange RUN_REFDIR=${RUN_REFDIR} + # ./xmlchange RUN_REFCASE=${RUN_REFCASE} + # ./xmlchange RUN_REFDATE=${RUN_REFDATE} + #echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + #echo '$RUN_REFDIR = '${RUN_REFDIR} + #echo '$RUN_REFCASE = '${RUN_REFCASE} + #echo '$RUN_REFDATE = '${START_DATE} + + #else + #echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + #exit 380 + #fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + # ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + # cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh new file mode 100755 index 0000000..793bd69 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +#useyear=1980 +#useyear=1981 +#useyear=1982 +useyear=1980 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking new file mode 100755 index 0000000..181786e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking @@ -0,0 +1,540 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { +# years=( 1958 1959 ) +useyear=1960 + +#array=( 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 ) +array=( 001 ) +for mbr in "${array[@]}" +do + +# For debugging, uncomment libe below +#set -x + + +# Year array YYYY: + +echo ${mbr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${mbr}" +#if [[ ${mbr} -lt 10 ]] +#then +# CASE_NAME="v21.LR.BSMYLE.00${mbr}" +#else +# CASE_NAME="v21.LR.BSMYLE.0${mbr}" +#fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-11-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv21-SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="24" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="${useyear}1101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="12" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv21-SMYLE/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/trop_mozart_aero/emis/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016-c230601.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + #if + # Build with COSP, except for a data atmosphere (datm) + # if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + # echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + # echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # copy restarts + if ${mbr}==1 then + USE_REFDIR="${USE_RUN_REFDIR}/${useyear}-11/" + elif + if ${mbr}<10 then + PERT_REFDIR="${USE_REFDIR}/pert.0${mbr}/" + else + USE_RUN_REFDIR="${USE_REFDIR}/pert.${mbr}/" + fi + fi + cp $USE_REFDIR/rpointer* $RUNDIR/ + ln -s $USE_REFDIR/v21.* $RUNDIR/ + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking2 b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking2 new file mode 100755 index 0000000..8a1d68e --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-notWorking2 @@ -0,0 +1,541 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { +# years=( 1958 1959 ) +useyear=1960 + +#array=( 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 ) +#array=( 001 ) +#for mbr in "${array[@]}" +for (( mbr=1; i<10; i++)) +do + +# For debugging, uncomment libe below +#set -x + + +# Year array YYYY: + +echo ${mbr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${mbr}" +#if [[ ${mbr} -lt 10 ]] +#then +# CASE_NAME="v21.LR.BSMYLE.00${mbr}" +#else +# CASE_NAME="v21.LR.BSMYLE.0${mbr}" +#fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-11-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv21-SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="24" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="${useyear}1101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="12" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv21-SMYLE/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/trop_mozart_aero/emis/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016-c230601.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + #if + # Build with COSP, except for a data atmosphere (datm) + # if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + # echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + # echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # copy restarts + if ${mbr}==1 then + USE_REFDIR="${USE_RUN_REFDIR}/${useyear}-11/" + elif + if ${mbr}<10 then + PERT_REFDIR="${USE_REFDIR}/pert.0${mbr}/" + else + USE_RUN_REFDIR="${USE_REFDIR}/pert.${mbr}/" + fi + fi + cp $USE_REFDIR/rpointer* $RUNDIR/ + ln -s $USE_REFDIR/v21.* $RUNDIR/ + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe1 b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe1 new file mode 100755 index 0000000..5601196 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe1 @@ -0,0 +1,458 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 001 ) +for mbr in "${array[@]}" +do + +main() { + +echo ${mbr} + +# For debugging, uncomment libe below +#set -x + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.1960-11.${mbr}" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1958-11-01" + +useyear=1960 +GET_REFCASE=true +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + ./xmlchange DATM_CLMNCEP_YR_ALIGN=1958 + ./xmlchange DATM_CLMNCEP_YR_START=1958 + ./xmlchange DATM_CLMNCEP_YR_END=2010 + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe2 b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe2 new file mode 100755 index 0000000..969c377 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.BSMYLE.sh-safe2 @@ -0,0 +1,456 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 001 002 003 004 005 006 007 008 009 010 ) +for mbr in "${array[@]}" +do + +main() { + +echo ${mbr} + +# For debugging, uncomment libe below +#set -x + +useyear=1959 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1959-11-01" + +GET_REFCASE=true +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.FROM-AUFIRE.sh b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.FROM-AUFIRE.sh new file mode 100755 index 0000000..27ce9a2 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.FROM-AUFIRE.sh @@ -0,0 +1,563 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SMYLE-TEST_jul13.${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1972-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +#RUN_REFDIR="/global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_${iyr}/archive/rest/archive/rest/2019-01-01-00000" +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/" +RUN_REFCASE="v2.LR.SSP370_${iyr}" +RUN_REFDATE="2019-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="18" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20190701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/v2.AUFIRE/forcingFiles/cmip6_ssp370_mam4_smoothed_bc_a4_elev_2018-202007_c230612.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # cp ${RUN_REFDIR}/* ${CASE_RUN_DIR} + mkdir -p ${CASE_RUN_DIR} + # cp ${RUN_REFDIR}/rpointer.atm ${CASE_RUN_DIR} + # ln -s ${RUN_REFDIR}/v2.* ${CASE_RUN_DIR} + + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.atm ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.drv ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ice ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.lnd ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ocn ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.rof ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.cpl.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h1.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h2.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h3.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h4.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h5.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h6.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.i.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.rs.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpaso.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpassi.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.ICRUELM.sh b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.ICRUELM.sh new file mode 100755 index 0000000..b836deb --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/scripts/v20/run.v2.LR.ICRUELM.sh @@ -0,0 +1,468 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="ICRUELM" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.${COMPSET}_test" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +#MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +#START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +#GET_REFCASE=TRUE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +#RUN_REFCASE="v2.LR.historical_${iyr}" +#RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + PELAYOUT="L" + WALLTIME="30:00:00" + STOP_OPTION="nyears" + STOP_N="30" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/README.atm-forcing b/scripts/toss/v2.SMYLE-toss/user_nl_files/README.atm-forcing new file mode 100644 index 0000000..73f8c3f --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/README.atm-forcing @@ -0,0 +1,60 @@ + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', +! ---------- + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', +! ---------- + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', +! ---------- + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' +! ---------- + ext_frc_type = 'INTERP_MISSING_MONTHS' +! ---------- + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_surf_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_surf_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_surf_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_surf_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_surf_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_surf_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_surf_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_surf_2015-2100_c210216.nc' + +! ---------- + + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_1.9x2.5_L26_1850-2015_c20181106.nc' + + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_SSP370_1.9x2.5_L70_2015-2100_c20211006.nc' + +! ---------- + +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP-1-2-0_Annual_Global_0000-2014_c20180105.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ + + +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_2015-2500_c20210509.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ + diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_elm b/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_elm new file mode 100644 index 0000000..f1fa0b2 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_elm @@ -0,0 +1,9 @@ + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_year_consistency = .true. + diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_mosart b/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_mosart new file mode 100644 index 0000000..ab66396 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/notUsed/user_nl_mosart @@ -0,0 +1,5 @@ + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 + diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/smbb/user_nl_eam-SSP370smbb b/scripts/toss/v2.SMYLE-toss/user_nl_files/smbb/user_nl_eam-SSP370smbb new file mode 100644 index 0000000..9587aed --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/smbb/user_nl_eam-SSP370smbb @@ -0,0 +1,68 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +&chem_inparm + chlorine_loading_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/Linoz_Chlorine_Loading_CMIP6_Hist_SSP370_0003-2503_c20210202.nc' + chlorine_loading_type = 'SERIAL' + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + linoz_data_file = 'linoz_1850-2500_CMIP6_Hist_SSP370_10deg_58km_c20210202.nc' + linoz_data_type = 'INTERP_MISSING_MONTHS' + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_surf_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_surf_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_surf_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_surf_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_surf_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_surf_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_surf_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_surf_2015-2100_c210216.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_SSP370_1.9x2.5_L70_2015-2100_c20211006.nc' +/ +&chem_surfvals_nl + !! See README file: /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/README.GHG_CMIP_SSP370 + bndtvghg = '/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_0000-2500_c20210509.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ + +&solar_inparm + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850-2299_input4MIPS_c20181106.nc' + solar_data_type = 'SERIAL' + solar_htng_spctrl_scl = .true. +/ + +&prescribed_volcaero_nl + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc' + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_filetype = 'VOLC_CMIP6' + prescribed_volcaero_type = 'CYCLICAL' +/ diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam b/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam new file mode 100644 index 0000000..0918300 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam @@ -0,0 +1,17 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + diff --git a/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam-SSP370 b/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam-SSP370 new file mode 100644 index 0000000..bf7d606 --- /dev/null +++ b/scripts/toss/v2.SMYLE-toss/user_nl_files/user_nl_eam-SSP370 @@ -0,0 +1,67 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +&chem_inparm + chlorine_loading_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/Linoz_Chlorine_Loading_CMIP6_Hist_SSP370_0003-2503_c20210202.nc' + chlorine_loading_type = 'SERIAL' + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_elev_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + linoz_data_file = 'linoz_1850-2500_CMIP6_Hist_SSP370_10deg_58km_c20210202.nc' + linoz_data_type = 'INTERP_MISSING_MONTHS' + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_surf_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_surf_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_surf_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_surf_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_surf_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_surf_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_surf_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_surf_2015-2100_c210216.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_SSP370_1.9x2.5_L70_2015-2100_c20211006.nc' +/ +&chem_surfvals_nl + !! See README file: /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/README.GHG_CMIP_SSP370 + bndtvghg = '/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_0000-2500_c20210509.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ + +&solar_inparm + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850-2299_input4MIPS_c20181106.nc' + solar_data_type = 'SERIAL' + solar_htng_spctrl_scl = .true. +/ + +&prescribed_volcaero_nl + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc' + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_filetype = 'VOLC_CMIP6' + prescribed_volcaero_type = 'CYCLICAL' +/ diff --git a/scripts/v2.LR.AMIP/README.notes b/scripts/v2.LR.AMIP/README.notes new file mode 100644 index 0000000..79bb9a0 --- /dev/null +++ b/scripts/v2.LR.AMIP/README.notes @@ -0,0 +1,6 @@ +F20TR : 20TR_EAM%CMIP6_ELM%SPBC_MPASSI%PRES_DOCN%DOM_MOSART_SGLC_SWAV +RESOLUTION="ne30pg2_EC30to60E2r2"k + +# 20th century transient +WCYCL20TR : 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +RESOLUTION="ne30pg2_EC30to60E2r2"k diff --git a/scripts/v2.LR.AMIP/change_CONT_RUN.scr b/scripts/v2.LR.AMIP/change_CONT_RUN.scr new file mode 100755 index 0000000..27b357e --- /dev/null +++ b/scripts/v2.LR.AMIP/change_CONT_RUN.scr @@ -0,0 +1,15 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange CONTINUE_RUN=FALSE +./xmlchange DOUT_S=FALSE + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.AMIP/change_DOUT.scr b/scripts/v2.LR.AMIP/change_DOUT.scr new file mode 100755 index 0000000..87958e7 --- /dev/null +++ b/scripts/v2.LR.AMIP/change_DOUT.scr @@ -0,0 +1,14 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP20TR_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange DOUT_S=FALSE + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.AMIP/change_JOB-QUEUE-EXEROOT.csh b/scripts/v2.LR.AMIP/change_JOB-QUEUE-EXEROOT.csh new file mode 100755 index 0000000..4025507 --- /dev/null +++ b/scripts/v2.LR.AMIP/change_JOB-QUEUE-EXEROOT.csh @@ -0,0 +1,25 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP20TR_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP20TR_${mbr}/run +set REFROOT = /global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${mbr}/archive/rest/1976-01-01-00000/ + +cd $CASEROOT/ +./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2-AMIP/EXEROOT/build/ +#./xmlchange JOB_QUEUE=debug --subgroup case.st_archive +#./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive + +#mv case.st_archive tmp.st_archive +#cat tmp.st_archive | sed 's/101/1/' > case.st_archive + +cd $RUNROOT/ +cp $REFROOT/rpointer* . +ln -s $REFROOT/v2.*.nc . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.AMIP/run.v2.LR.AMIP_YYYY.sh b/scripts/v2.LR.AMIP/run.v2.LR.AMIP_YYYY.sh new file mode 100755 index 0000000..ae5f12b --- /dev/null +++ b/scripts/v2.LR.AMIP/run.v2.LR.AMIP_YYYY.sh @@ -0,0 +1,511 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="F20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.AMIP20TR_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-AMIP/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AMIP/run.v2.LR.amip_0101-test.sh-notUsed b/scripts/v2.LR.AMIP/run.v2.LR.amip_0101-test.sh-notUsed new file mode 100755 index 0000000..bf1aacb --- /dev/null +++ b/scripts/v2.LR.AMIP/run.v2.LR.amip_0101-test.sh-notUsed @@ -0,0 +1,512 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="F20TR" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.amip_0101-TEST" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.piControl/archive/rest/0501-01-01-00000/" +readonly RUN_REFCASE="v2.LR.shi" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/TEST/${CASE_NAME}" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="/lcrc/group/e3sm/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="30:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="30" # How often to stop the model, should be a multiple of REST_N + readonly REST_OPTION="nyears" + readonly REST_N="5" # How often to write a restart file + readonly RESUBMIT="4" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Custom PE layout +custom_pelayout + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + +# ===================================================== +# Custom PE layout: custom-N where N is number of nodes +# ===================================================== + +custom_pelayout() { + +if [[ ${PELAYOUT} == custom-* ]]; +then + echo $'\n CUSTOMIZE PROCESSOR CONFIGURATION:' + + # Number of cores per node (machine specific) + if [ "${MACHINE}" == "chrysalis" ]; then + ncore=64 + elif [ "${MACHINE}" == "compy" ]; then + ncore=40 + else + echo 'ERROR: MACHINE = '${MACHINE}' is not supported for custom PE layout.' + exit 400 + fi + + # Extract number of nodes + tmp=($(echo ${PELAYOUT} | tr "-" " ")) + nnodes=${tmp[1]} + + # Customize + pushd ${CASE_SCRIPTS_DIR} + ./xmlchange NTASKS=$(( $nnodes * $ncore )) + ./xmlchange NTHRDS=1 + ./xmlchange MAX_MPITASKS_PER_NODE=$ncore + ./xmlchange MAX_TASKS_PER_NODE=$ncore + popd + +fi + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + if [[ ${PELAYOUT} == custom-* ]]; + then + layout="M" # temporary placeholder for create_newcase + else + layout=${PELAYOUT} + + fi + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${layout} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.AMIP/run_amip.20TR.scr b/scripts/v2.LR.AMIP/run_amip.20TR.scr new file mode 100755 index 0000000..6a5f3ba --- /dev/null +++ b/scripts/v2.LR.AMIP/run_amip.20TR.scr @@ -0,0 +1,477 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +for iyr in "${array[@]}" +do + +main() { + +# Year array YYYY: + +echo ${iyr} + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" + +# Simulation +COMPSET="F20TR" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.AMIP20TR_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-AMIP/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + PELAYOUT="L" + WALLTIME="30:00:00" + STOP_OPTION="nyears" + STOP_N="30" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AMIP/run_amip.20TR.scr-fromGit b/scripts/v2.LR.AMIP/run_amip.20TR.scr-fromGit new file mode 100644 index 0000000..edfa986 --- /dev/null +++ b/scripts/v2.LR.AMIP/run_amip.20TR.scr-fromGit @@ -0,0 +1,509 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=chrysalis +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="F20TR" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.amip_0101" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20210929" +readonly BRANCH="1e754ed78cdae71b8a82bfdc7a2ec6a89d3d94af" # v2 tag +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1870-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/lcrc/group/e3sm/ac.forsyth2/E3SMv2/v2.LR.amip_0101/init" +readonly RUN_REFCASE="v2.LR.historical_0101" +readonly RUN_REFDATE="1870-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +readonly CASE_ROOT="/lcrc/group/e3sm/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="custom-30" + readonly WALLTIME="30:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="30" # How often to stop the model, should be a multiple of REST_N + readonly REST_OPTION="nyears" + readonly REST_N="5" # How often to write a restart file + readonly RESUBMIT="4" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Custom PE layout +custom_pelayout + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + +# ===================================================== +# Custom PE layout: custom-N where N is number of nodes +# ===================================================== + +custom_pelayout() { + +if [[ ${PELAYOUT} == custom-* ]]; +then + echo $'\n CUSTOMIZE PROCESSOR CONFIGURATION:' + + # Number of cores per node (machine specific) + if [ "${MACHINE}" == "chrysalis" ]; then + ncore=64 + elif [ "${MACHINE}" == "compy" ]; then + ncore=40 + else + echo 'ERROR: MACHINE = '${MACHINE}' is not supported for custom PE layout.' + exit 400 + fi + + # Extract number of nodes + tmp=($(echo ${PELAYOUT} | tr "-" " ")) + nnodes=${tmp[1]} + + # Customize + pushd ${CASE_SCRIPTS_DIR} + ./xmlchange NTASKS=$(( $nnodes * $ncore )) + ./xmlchange NTHRDS=1 + ./xmlchange MAX_MPITASKS_PER_NODE=$ncore + ./xmlchange MAX_TASKS_PER_NODE=$ncore + popd + +fi + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + if [[ ${PELAYOUT} == custom-* ]]; + then + layout="M" # temporary placeholder for create_newcase + else + layout=${PELAYOUT} + + fi + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${layout} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.AMIP/v2.LR.AMIP_bundle.sh b/scripts/v2.LR.AMIP/v2.LR.AMIP_bundle.sh new file mode 100755 index 0000000..0295f5c --- /dev/null +++ b/scripts/v2.LR.AMIP/v2.LR.AMIP_bundle.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.AMIP +#SBATCH --account=mp9 +#SBATCH --nodes=1111 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.AMIP_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP20TR_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/AMIP + . ./v2.LR.AMIP_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/AMIP +. ./v2.LR.AMIP_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.AMIP_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.AMIP/v2.LR.AMIP_extra.sh b/scripts/v2.LR.AMIP/v2.LR.AMIP_extra.sh new file mode 100644 index 0000000..0329d34 --- /dev/null +++ b/scripts/v2.LR.AMIP/v2.LR.AMIP_extra.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +#echo ======================================================== +#n="0141" +#echo Attempting to restart ${n} +#date +#cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts +#./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " +#./xmlchange STOP_N="5" +#####./xmlchange STOP_N="3" +#./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & +#PID=$! +#echo $PID +#echo ======================================================== + diff --git a/scripts/v2.LR.AMIP/v2.LR.AMIP_post.sh b/scripts/v2.LR.AMIP/v2.LR.AMIP_post.sh new file mode 100644 index 0000000..c6dbf62 --- /dev/null +++ b/scripts/v2.LR.AMIP/v2.LR.AMIP_post.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + cd /global/cscratch1/sd/nanr/E3SMv2-AMIP/v2.LR.AMIP20TR_${n}/case_scripts + ./xmlchange CONTINUE_RUN="TRUE" +done + +echo === Submitting continuation job === +date +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/AMIP +sbatch v2.LR.AMIP_bundle.sh +echo =================================== diff --git a/scripts/v2.LR.AUFIRE/README.HICCUP b/scripts/v2.LR.AUFIRE/README.HICCUP new file mode 100644 index 0000000..288e4f7 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/README.HICCUP @@ -0,0 +1,80 @@ +What I've done: + +git clone https://github.com/E3SM-Project/HICCUP ./HICCUP + +## This zip file was attached to Walter's August 21, 2023 email: + scp cheyenne.ucar.edu:/glade/scratch/nanr/catalyst_hiccup.zip . + +bash + +From README.MD +~~ -------------------------- +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.7 +module load python + +conda activate hiccup_env3 +./setup.py install + +~~ -------------------------- +# First, need to get API for data download: +# Look for api on website: ECMWF +https://cds.climate.copernicus.eu/api-how-to + +# set up this file to download ERA5 data +/global/homes/n/nanr/.cdsapirc + +# Get Data (edit time periods and month!) +./get_hindcast_data.ERA5.py + +# process Data (edit time periods and month!) +sbatch 2023-CATALYST.batch.py +~~ -------------------------- +# pick up data Walter shared for me +cd /pscratch/sd/n/nanr/data_IC/data_ERA5 +take -u whannah +take -u whannah -a +~~ -------------------------- + +# From Walter Hannah: + + + + +~~ -------------------------- +conda create --name hiccup_env3 -c conda-forge xarray dask +(nersc-python) nanr@perlmutter:login11:~> conda activate hiccup_env3 +conda install -c conda-forge pandas numpy scipy netcdf4 pynio hdf5 +conda install -c conda-forge cdsapi tempest-remap nco +~~ -------------------------- + +/global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/HICCUP/catalyst_hiccup +2023-CATALYST.batch.py + +Got help from JimEdwards: Added script from 2023-CATALYST.batch.sh to the top of 2023-CATALYST.batch.py and changed from --partition to -qos regular. +Then submitted: + +sbatch 2023-CATALYST.batch.py + + + +======================================= +From Walter Hannah: August 21, 2023 +The first step is to grab the HICCUP package from here => https://github.com/E3SM-Project/HICCUP +After cloning that you need to follow the set up instructions to create a new conda env and run “./setup.py install” + +Afterwards you can extract the scripts in the attached zip archive, which contains these files: + +2023-CATALYST.batch.py +2023-CATALYST.batch.sh +2023-CATALYST.create_IC_from_obs.py +get_hindcast_data.ERA5.py + +get_hindcast_data.ERA5.py is for aquiring the ERA5 data that will be turned into the initial condition (IC) files. Each IC needs both vertically resolved and surface data, which end up coming in separate files. +2023-CATALYST.create_IC_from_obs.py is where all the action happens, but for various reasons it can only process/create a single IC file at a time, so… +I created 2023-CATALYST.batch.py to loop through all the hindcast dates and call the previous script. +However, You might not want to run this on a login node, so if that’s the case then you can use 2023-CATALYST.batch.sh to run it as a batch job. +Make sure you go through all the scripts and adjust the various paths before calling them. You’ll also likely need to adjust the batch job parameters to change the project account and whatnot. +Feel free to hit me with any questions. diff --git a/scripts/v2.LR.AUFIRE/restarts/README.generate-ICs.bash b/scripts/v2.LR.AUFIRE/restarts/README.generate-ICs.bash new file mode 100644 index 0000000..ee4be95 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/README.generate-ICs.bash @@ -0,0 +1,35 @@ +export CYLC_TASK_CYCLE_POINT="1959-11-01" + +### use bash + +# not needed anymore: module load cpe/23.03 + +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/12.2 +module load python/3.9-anaconda-2021.11 + +## error: module load cudatoolkit/11.5 + + +module load python + + +## Edit start years to set up +Step 1: ./create-v2.SMYLE.IC_RESTARTS.csh Or create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE +Step 2: ./run_generate.sh +Step 3: ./fix_xtime.sh (change xtime to xtime.orig in mpaso.rst files) +Step 4: ./fix_ncol_d.sh (change ncol_d to ncol in eam.i files) + + +export CYLC_TASK_CYCLE_POINT=1970-11-01 +export CYLC_TASK_CYCLE_POINT=2019-08-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +export CYLC_TASK_CYCLE_POINT=1970-08-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + + +## Don't have to do this anymore, now that I've fixed the code: +# PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py diff --git a/scripts/v2.LR.AUFIRE/restarts/README.history b/scripts/v2.LR.AUFIRE/restarts/README.history new file mode 100644 index 0000000..ca2d14f --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/README.history @@ -0,0 +1,714 @@ + 1 ls -ld /global/cscratch1/sd/nanr/archive/v2.LR.historical_0* + 2 less ~/README.zstashExtract + 3 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/2015-01-01-00000/*" + 4 mv archive/rest/2015-01-01-00000/v2.LR.historical_0111.* .. + 5 cd ../../case_scripts/ + 6 ./case.submit + 7 ./xmlquery EXEROOT + 8 ls .. + 9 ls ../build/ + 10 ls -ltr ../build/ + 11 qme + 12 exit + 13 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 14 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 15 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 16 pwd + 17 ls + 18 history + 19 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 20 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 21 pwd + 22 ls -l + 23 ls zstash/ + 24 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 25 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 26 exit + 27 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 28 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0241 2>&1 | tee zstash_update_smbb-hist-0241_20230306.log + 29 bg + 30 ls -ltr + 31 jobs + 32 ls -ltr + 33 less zstash_update_smbb-hist-0241_20230306.log + 34 cd .. + 35 mkdir 261 + 36 cd 261/ + 37 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0261 2>&1 | tee zstash_update_smbb-hist-0261_20230306.log & + 38 ls -ltr ~/zstash_create_*261* + 39 less /global/homes/n/nanr/zstash_create_smbb-hist-0261_20230203.log + 40 less /global/homes/n/nanr/zstash_create_smbb-hist-0271_20230203.log + 41 cd ../ + 42 mkdir 271 + 43 cd 271/ + 44 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0271 2>&1 | tee zstash_update_smbb-hist-0271_20230306.log & + 45 ls -ltr + 46 jobs + 47 cd ../ + 48 mkdir 281 + 49 cd 281/ + 50 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0281 2>&1 | tee zstash_update_smbb-hist-0281_20230306.log + 51 bg + 52 cd .. + 53 mkdir 291 + 54 cd 291/ + 55 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0291 2>&1 | tee zstash_update_smbb-hist-0291_20230306.log + 56 bg + 57 cd .. + 58 mkdir 101 + 59 cd 101 + 60 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 2>&1 | tee zstash_update_smbb-hist-0101_20230306.log + 61 bg + 62 jobs + 63 cd ../ + 64 mkdir 151 + 65 cd 151/ + 66 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0151 2>&1 | tee zstash_update_smbb-hist-0151_20230306.log + 67 bg + 68 cd .. + 69 mkdir 201 + 70 cd 201 + 71 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 2>&1 | tee zstash_update_smbb-hist-0201_20230306.log + 72 less ~/zstash_create_*201* + 73 cd .. + 74 mkdir 251 + 75 cd 251/ + 76 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0251 2>&1 | tee zstash_update_smbb-hist-0251_20230306.log + 77 bg + 78 cd ../201 + 79 ls + 80 jobs + 81 cd .. + 82 mkdir 301 + 83 cd 301 + 84 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0301 2>&1 | tee zstash_update_smbb-hist-0301_20230306.log + 85 bg + 86 cd ../../ + 87 cd 210/ + 88 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" + 89 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" | less + 90 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*" | less + 91 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | less + 92 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | wc + 93 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "lnd/hist/*h0*" | wc + 94 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307.log + 95 bg + 96 less zstash_check_210.001-20230307.log + 97 fg + 98 ls + 99 mv zstash zstash-0ld + 100 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 101 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 102 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 103 exit + 104 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 105 less ~/README.zstashInstructions + 106 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/*" + 107 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2021-01-01-00000/*" + 108 ls + 109 cd archive/rest/2021-01-01-00000/ + 110 cp * /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/run/ + 111 scr + 112 exit + 113 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 114 ls -ltr + 115 ls -l archive/rest/ + 116 ls archive/atm/hist/*h0* + 117 b + 118 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 119 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 120 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 121 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201-new --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create-new_smbb-hist-0201_20230313.log + 122 cd .. + 123 b + 124 cd v2.LR.historical-smbb_0201/ + 125 hsi + 126 exit + 127 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 128 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 129 pwd + 130 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 131 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 132 exit + 133 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 134 cd ../../ + 135 cd 210/ + 136 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 137 hsi "ls -l /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001" + 138 ls -ltr + 139 less zstash_check_210.001-20230307b.log + 140 hsi + 141 less ~/README.zstashInstructions + 142 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 143 pwd + 144 less ~/README.zstashExtract + 145 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/rest/2101-01-01-00000/*" + 146 bg + 147 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/run/*" + 148 fg + 149 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 150 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/" + 151 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/*" + 152 exit + 153 screen + 154 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 155 pwd + 156 less ~/zstash_update_320.003_20230413.log + 157 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 158 ls + 159 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 160 pwd + 161 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413.log + 162 history + 163 pwd + 164 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413a.log + 165 exit + 166 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 167 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/archive/rest/*" + 168 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/\archive/rest/*" + 169 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" + 170 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" > doh + 171 less doh + 172 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 173 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000" + 174 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 175 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000/*" + 176 cd ../ + 177 mkdir v2.LR.historical_0121 + 178 cd v2.LR.historical_0121/ + 179 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0121 "archive/rest/1976-01-01-00000/*" + 180 cd .. + 181 mkdir v2.LR.historical_0131 + 182 cd v2.LR.historical_0131/ + 183 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0131 "archive/rest/1976-01-01-00000/*" + 184 cd .. + 185 mkdir v2.LR.historical_0141 + 186 cd v2.LR.historical_0141/ + 187 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0141 "archive/rest/1976-01-01-00000/*" + 188 cd .. + 189 mkdir v2.LR.historical_0161 + 190 cd v2.LR.historical_0161/ + 191 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0161 "archive/rest/1976-01-01-00000/*" + 192 cd .. + 193 mkdir v2.LR.historical_0171 + 194 cd v2.LR.historical_0171/ + 195 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0171 "archive/rest/1976-01-01-00000/*" + 196 cd .. + 197 mkdir v2.LR.historical_0181 + 198 cd v2.LR.historical_0181/ + 199 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0181 "archive/rest/1976-01-01-00000/*" + 200 cd .. + 201 mkdir v2.LR.historical_0191 + 202 cd v2.LR.historical_0191/ + 203 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0191 "archive/rest/1976-01-01-00000/*" + 204 cd .. + 205 mkdir v2.LR.historical_0211 + 206 cd v2.LR.historical_0211/ + 207 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0211 "archive/rest/1976-01-01-00000/*" + 208 cd .. + 209 mkdir v2.LR.historical_0221 + 210 cd v2.LR.historical_0221/ + 211 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0221 "archive/rest/1976-01-01-00000/*" + 212 pwd + 213 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 214 pwd + 215 zstash update --dry-run + 216 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 217 exit + 218 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 219 pwd + 220 zstash update --dry-run + 221 history + 222 bash + 223 exit + 224 pwd + 225 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 226 zstash update --dry-run + 227 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 228 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.111_20230414.log + 229 ls -ltr + 230 cd .. + 231 cd .. + 232 ls -ltr + 233 cd E3SMv2 + 234 ls -ltr + 235 cd v2.LR.SSP370_0281 + 236 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281 + 237 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.281_20230414.log + 238 ../cd v2.LR.SSP370_0211 + 239 ls + 240 pwd + 241 cd ../v2.LR.SSP370_0211/ + 242 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 + 243 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.211_20230414.log + 244 exit + 245 pwd + 246 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 247 vi ~/README.zstashExtract + 248 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0231 "archive/rest/1976-01-01-00000/*" + 249 exit + 250 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 251 vi ~/README.zstashInstructions + 252 zstash check --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 2>&1 | tee zstash_check_0251_20230320.log + 253 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 254 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 255 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230320.log + 256 screen + 257 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 258 pwd + 259 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_320.003_20230321.log + 260 $ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 261 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 262 vi ~/README.zstashInstructions + 263 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 264 pwd + 265 fg + 266 pwd + 267 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230320.log + 268 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 269 cd /global/cfs/cdirs/mp9/archive/ + 270 ls + 271 cd e3smv1 + 272 ls + 273 pwd + 274 ls -ltr + 275 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 276 cd /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0251 + 277 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_0251_20230321.log + 278 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 279 pwd + 280 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/ + 281 ls -ltr + 282 cd v2.LR.historical-smbb_0101 + 283 vi ~/README.zstashInstructions + 284 pwd + 285 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0101_20230203.log + 286 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 287 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 + 288 pwd + 289 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0201_20230314.log + 290 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 291 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 + 292 pwd + 293 ls -ltr ~/zstash* + 294 less /global/homes/n/nanr/zstash_update_smbb-ssp370-0181_20230314.log + 295 pwd + 296 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0191_20230314.log + 297 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 298 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 299 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 300 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" . 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 301 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 302 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 303 hsi ls /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 + 304 hsi ls /home/c/ccsm/E3SMv1-LE/ + 305 hsi ls -l /home/c/ccsm/E3SMv1-LE/ + 306 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 307 pwd + 308 ls + 309 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.003_20230321.log + 310 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 311 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 312 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 313 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 + 314 pwd + 315 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0171_20230314.log + 316 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 317 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 318 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0131_20230217.log + 319 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 320 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 321 pwd + 322 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230314.log + 323 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 324 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 325 pwd + 326 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230314.log + 327 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 328 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 + 329 pwd + 330 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0141_20230314.log + 331 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 332 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 + 333 pwd + 334 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0181_20230314.log + 335 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 336 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 337 pwd + 338 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230217.log + 339 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 340 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 + 341 pwd + 342 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0151_20230314.log + 343 fg + 344 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 345 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 + 346 pwd + 347 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0231_20230314.log + 348 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 349 cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 350 pwd + 351 cfs + 352 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 353 ls -ltr + 354 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230217.log + 355 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 356 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 + 357 pwd + 358 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0161_20230314.log + 359 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 360 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 361 pwd + 362 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0101_20230314.log + 363 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 364 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 365 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 366 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 + 367 pwd + 368 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0211_20230314.log + 369 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 370 pwd + 371 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 372 ls -ltr + 373 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 374 ls -ltr + 375 ls zstash/ + 376 rmdir zstash/ + 377 pwd + 378 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 379 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.002_20230413.log + 380 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 381 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 + 382 pwd + 383 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0251_20230314.log + 384 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 385 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 + 386 pwd + 387 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0301_20230314.log + 388 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 389 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 + 390 pwd + 391 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0261_20230314.log + 392 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 393 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 + 394 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0271_20230314.log + 395 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 396 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 + 397 pwd + 398 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0281_20230314.log + 399 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 400 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 401 pwd + 402 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230314.log + 403 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 404 jobs + 405 cd /global/cscratch1/sd/nanr + 406 cd checkZstash/210/ + 407 ls + 408 pwd + 409 ls + 410 mv zstash zstash-a + 411 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 412 ls -ltr + 413 hsi + 414 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 415 cd zstash + 416 pwd + 417 cd /global/cfs/cdirs/mp9/archive/ + 418 cd e3smv1 + 419 ls + 420 cd *210.001 + 421 ls -ltr + 422 cd zstash/ + 423 ls + 424 pwd + 425 less index.db + 426 sqlite3 index.db + 427 pwd + 428 cd .. + 429 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_210.001-20230307a.log + 430 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 431 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 432 pwd + 433 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230314.log + 434 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 435 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.historical-smbb_0201 + 436 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0201_20230313.log + 437 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 438 vi ~/README.zstashExtract + 439 cd ../ + 440 ls + 441 cd v2.LR.historica_0201 + 442 cd v2.LR.historical_0201 + 443 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1990-01-01-00000/*" + 444 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1970-01-01-00000/*" + 445 qme + 446 exit + 447 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 448 less extract_zstash.sh + 449 hsi ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 450 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 451 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*" + 452 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/" + 453 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/" + 454 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive" + 455 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/" + 456 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 457 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 458 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 459 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" > /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 460 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" >& /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 461 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 462 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 463 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 464 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 465 less ~/doh + 466 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 467 history + 468 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist" + 469 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 470 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm" + 471 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist" + 472 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl >& ~nanr/doh1 + 473 less ~/doh1 + 474 tcsh + 475 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 476 less ~/README.zstashInstructions + 477 hsi ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 478 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 479 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest" + 480 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "rest" + 481 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 | less + 482 cd v2.LR.SSP370_0111 + 483 ls + 484 cd archive/ + 485 ls + 486 mkdir rest + 487 cd rest + 488 ls + 489 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 490 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 491 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/*" + 492 cd ../../../v2.LR.SSP370_0121/archive/ + 493 mkdir rest; cd rest + 494 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0121 "archive/rest/2019-01-01-00000/*" + 495 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 496 less ~/README.zstashInstructions + 497 less ~/README.zstashExtract + 498 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2020-01-01-00000/*" + 499 exit + 500 tcsh + 501 tcsh + 502 exit + 503 tcsh + 504 less /global/cfs/cdirs/ + 505 cd /global/cfs/cdirs/ + 506 cd ccsm1 + 507 ls + 508 cd people + 509 cd nanr/ + 510 cd e3sm_tags + 511 ls + 512 cd E3SMv2.1 + 513 ls + 514 history + 515 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 516 mv E3SM/ E3SM-safe + 517 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 518 hsi /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 519 hsi ls /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 520 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 521 hsi "cd /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/; chmod 744 00005d.tar" + 522 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 523 wk + 524 tcsh + 525 tcsh + 526 ls -ltr + 527 ./create-v2.SMYLE.IC_RESTARTS.csh + 528 less ~/README.nco + 529 module load cray-hdf5 + 530 module load cray-netcdf + 531 module load nco + 532 ./create-v2.SMYLE.IC_RESTARTS.csh + 533 vi ~/README.zstashExtract + 534 vi create-v2.SMYLE.IC_RESTARTS.csh + 535 scp cheyenne.ucar.edu:/glade/work/nanr/cesm_tags/CASE_tools/cesm2-smyle/restarts/generate_cami_ensemble_offline.py . + 536 less ~/.alias.nanr + 537 pwd + 538 vi generate_cami_ensemble_offline.py + 539 pwd + 540 fg + 541 vi create-v2.SMYLE.IC_RESTARTS.csh + 542 vi generate_cami_ensemble_offline.py + 543 vi create-v2.SMYLE.IC_RESTARTS.csh + 544 vi generate_cami_ensemble_offline.py + 545 vi create-v2.SMYLE.IC_RESTARTS.csh + 546 vi generate_cami_ensemble_offline.py + 547 vi create-v2.SMYLE.IC_RESTARTS.csh + 548 ./create-v2.SMYLE.IC_RESTARTS.csh + 549 vi create-v2.SMYLE.IC_RESTARTS.csh + 550 vi generate_cami_ensemble_offline.py + 551 vi create-v2.SMYLE.IC_RESTARTS.csh + 552 grep CESM2_TOOLS_ROOT * + 553 vi create-v2.SMYLE.IC_RESTARTS.csh + 554 ./create-v2.SMYLE.IC_RESTARTS.csh + 555 spack env activate gcc + 556 exit + 557 source ~/.alias.bash + 558 tools + 559 cd e3sm/v2/scripts/v2.SMYLE/restarts/ + 560 ls -ltr + 561 vi generate_cami_ensemble_offline.py + 562 less create-v2.SMYLE.IC_RESTARTS.csh + 563 export CYLC_TASK_CYCLE_POINT=1959-01-01 + 564 ./generate_cami_ensemble_offline.py + 565 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime + 566 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime ./generate_cami_ensemble_offline.py + 567 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME ./generate_cami_ensemble_offline.py + 568 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 569 module avail python + 570 module load python/3.9 + 571 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 572 vi generate_cami_ensemble_offline.py + 573 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 574 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/cases/ + 575 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/ + 576 fg + 577 vi generate_cami_ensemble_offline.py + 578 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 579 ls /global/cfs/cdirs/mp9/ + 580 vi generate_cami_ensemble_offline.py + 581 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 582 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-01.01/ + 583 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 584 vi generate_cami_ensemble_offline.py + 585 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 586 history + 587 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 588 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 589 less ~/README.nco + 590 module load e4s + 591 spack env activate gcc + 592 spack load nco + 593 module load cudatoolkit/11.5 + 594 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 595 history + 596 vi README.generate.bash + 597 vi create-v2.SMYLE.IC_RESTARTS.csh + 598 ./create-v2.SMYLE.IC_RESTARTS.csh + 599 echo $SHERLL + 600 echo $SHELL + 601 vi create-v2.SMYLE.IC_RESTARTS.csh + 602 vi setEnv.scr + 603 chmod u+x setEnv.scr + 604 ./setEnv.scr + 605 echo $CYLC_TASK_CYCLE_POINT + 606 setenv CYLC_TASK_CYCLE_POINT 1960-11-01 + 607 export CYLC_TASK_CYCLE_POINT 1960-11-01 + 608 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 609 echo $CYLC_TASK_CYCLE_POINT + 610 ./generate_cami_ensemble_offline.py + 611 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 612 less README.generate.bash + 613 module load e4s + 614 spack env activate gcc + 615 spack load nco + 616 module load cudatoolkit/11.5 + 617 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 618 ls -l /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 619 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/1960-11-01/ + 620 vi generate_cami_ensemble_offline.py + 621 vi create-v2.SMYLE.IC_RESTARTS.csh + 622 ./create-v2.SMYLE.IC_RESTARTS.csh + 623 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 624 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 625 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/11/ + 626 vi generate_cami_ensemble_offline.py + 627 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 628 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 629 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 630 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 631 ./generate_cami_ensemble_offline.py + 632 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.11/pert.01/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc + 633 vi generate_cami_ensemble_offline.py + 634 ./generate_cami_ensemble_offline.py + 635 history + 636 module load e4s + 637 spack env activate gcc + 638 spack load nco + 639 module load cudatoolkit/11.5 + 640 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 641 grep slat * + 642 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 643 vi generate_cami_ensemble_offline.py + 644 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 645 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 646 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 647 ./create-v2.SMYLE.IC_RESTARTS.csh + 648 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 649 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 650 vi generate_cami_ensemble_offline.py + 651 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 652 vi generate_cami_ensemble_offline.py + 653 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 654 vi ./create-v2.SMYLE.IC_RESTARTS.csh + 655 ./create-v2.SMYLE.IC_RESTARTS.csh + 656 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 657 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 658 history + 659 vi generate_cami_ensemble_offline.py + 660 cp generate_cami_ensemble_offline.py doh.py + 661 vi doh.py + 662 fg + 663 vi doh.py + 664 vi generate_cami_ensemble_offline.py + 665 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 666 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc + 667 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.18/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 668 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.17/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 669 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.14/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 670 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.13/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 671 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.15/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 672 grep eamic * + 673 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 674 jobs + 675 vi generate_cami_ensemble_offline.py + 676 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 677 ./create-v2.SMYLE.IC_RESTARTS.csh + 678 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 679 vi generate_cami_ensemble_offline.py + 680 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 681 fg + 682 vi generate_cami_ensemble_offline.py + 683 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 684 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 685 vi generate_cami_ensemble_offline.py + 686 ./generate_cami_ensemble_offline.py + 687 vi generate_cami_ensemble_offline.py + 688 ./generate_cami_ensemble_offline.py + 689 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/ + 690 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/ + 691 vi generate_cami_ensemble_offline.py + 692 vi create-v2.SMYLE.IC_RESTARTS.csh + 693 ./generate_cami_ensemble_offline.py + 694 ls -l /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.BSMYLE.ne30np4.1960-11.001/run.002/v21.LR.SMYLE_IC.ne30np4.1960-11.01.eam.i.1960-11-01-00000.nc + 695 vi create-v2.SMYLE.IC_RESTARTS.csh + 696 vi generate_cami_ensemble_offline.py + 697 vi create-v2.SMYLE.IC_RESTARTS.csh + 698 ./create-v2.SMYLE.IC_RESTARTS.csh + 699 export CYLC_TASK_CYCLE_POINT=1962-11-01 + 700 ./generate_cami_ensemble_offline.py + 701 vi create-v2.SMYLE.IC_RESTARTS.csh + 702 vi generate_cami_ensemble_offline.py + 703 ./generate_cami_ensemble_offline.py + 704 vi generate_cami_ensemble_offline.py + 705 ./generate_cami_ensemble_offline.py + 706 vi generate_cami_ensemble_offline.py + 707 ./generate_cami_ensemble_offline.py + 708 vi generate_cami_ensemble_offline.py + 709 ./generate_cami_ensemble_offline.py + 710 history + 711 vi README.generate.bash + 712 fg + 713 history + 714 history > doh diff --git a/scripts/v2.LR.AUFIRE/restarts/README.paths b/scripts/v2.LR.AUFIRE/restarts/README.paths new file mode 100644 index 0000000..1b51379 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/README.paths @@ -0,0 +1,22 @@ +atm: /global/cscratch1/sd/sglanvil/SMYLE_atm + +OCN restarts (2018, and 1958-1965; see README) - Luke Van Roekel: +Glade: /glade/p/datashare/yeager/CATALYST/ocean_ice_ic_JRA_cycle1/ +Cori: /global/cscratch1/sd/sglanvil/SMYLE_ocn +2018 = 0061 +1958 = 0062 +1959 = 0063 +1960 = 0064 +1961 = 0065 +1962 = 0066 +1963 = 0067 +1964 = 0068 +1965 = 0069 +(lets rename at some point) + +PERTURBATIONS (001-999): +Nan Rosenbloom AMIP daily-IC run 1991-2015: /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/CAMI/ +Sasha Glanville make random perts: /global/homes/s/sglanvil/S2S/make_random_diffs.sh +Saved random perts: /global/homes/s/sglanvil/S2S/save_perts +Final pert files:: /global/cscratch1/sd/sglanvil/S2S_perts_DIFF + diff --git a/scripts/v2.LR.AUFIRE/restarts/README_perts_sasha b/scripts/v2.LR.AUFIRE/restarts/README_perts_sasha new file mode 120000 index 0000000..6d3845e --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/README_perts_sasha @@ -0,0 +1 @@ +/global/homes/s/sglanvil/S2S/README_perts \ No newline at end of file diff --git a/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE b/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE new file mode 100755 index 0000000..911f460 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE @@ -0,0 +1,160 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST + +# Nan needs to update +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ +set lnddir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +# Original +# set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ +# copied here: +set ocndir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS.csh b/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS.csh new file mode 100755 index 0000000..125ca2c --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/create-v2.SMYLE.IC_RESTARTS.csh @@ -0,0 +1,166 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) + set syr = 1970 + set eyr = 1975 +# set syr = 1976 +# set eyr = 1980 +# set syr = 1981 +# set eyr = 2000 +#set syr = 2001 +#set eyr = 2001 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +# foreach mon ( 11 ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +# fix xtime offline: fix_xtime.sh (using bash) +# ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} +# rm $icdir/${poprfout}.xtime.nc + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/v2.LR.AUFIRE/restarts/fix_ncol_d.sh b/scripts/v2.LR.AUFIRE/restarts/fix_ncol_d.sh new file mode 100755 index 0000000..5dc2714 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/fix_ncol_d.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=11 +mon=08 + +for year in {2019..2019}; do +#for pert in {01..20}; do +for pert in {21..40}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + eamfile=v21.LR.SMYLE_IC.pert.eam.i.$year-${mon}-01-00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/pert.${pert} + echo $case + echo $icdir + ncrename -d ncol_d,ncol $icdir/${eamfile} +done +done + diff --git a/scripts/v2.LR.AUFIRE/restarts/fix_xtime.sh b/scripts/v2.LR.AUFIRE/restarts/fix_xtime.sh new file mode 100755 index 0000000..bf5ec07 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/fix_xtime.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=08 + +for year in {2019..2019}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + poprfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpaso.rst.$year-${mon}-01_00000.nc + icerfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpassi.rst.$year-${mon}-01_00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 + echo $case + echo $icdir + mv $icdir/${poprfout} $icdir/${poprfout}.xtime.nc + mv $icdir/${icerfout} $icdir/${icerfout}.xtime.nc + ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} + ncrename -v xtime,xtime.orig $icdir/${icerfout}.xtime.nc $icdir/${icerfout} +done + diff --git a/scripts/v2.LR.AUFIRE/restarts/generate_cami_ensemble_offline.py b/scripts/v2.LR.AUFIRE/restarts/generate_cami_ensemble_offline.py new file mode 100755 index 0000000..c78d61f --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/generate_cami_ensemble_offline.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +#PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools + +if e3smroot is None: + raise SystemExit("ERROR: CESM_ROOT must be defined in environment") + +# This is needed for globus_sdk +#_LIBDIR=os.path.join(os.environ.get("HOME"),".local","lib","python3.6","site-packages") +#sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, random, threading, time, shutil +from standard_script_setup import * +from CIME.utils import run_cmd, safe_copy, expect +from argparse import RawTextHelpFormatter +#from globus_utils import * + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=40, + help="Specify the last ensemble member") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.environ.get("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model,int(args.ensemble_start),int(args.ensemble_end) + +def get_rvals(date, ensemble_start,ensemble_end, model): + random.seed(int(date[0:4])+int(date[5:7])+int(date[8:10])) + rvals = random.sample(range(1001),k=ensemble_end//2) + print("Rvals are {}".format(rvals)) + #rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/","cases","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + if not os.path.isfile(rvals_file): + with open(rvals_file,"w") as fd: + fd.write("{}".format(rvals)) + + return rvals + +def create_eam_ic_perturbed(original, ensemble_start,ensemble_end, date, baserundir, model, outroot="v21.LR.SMYLE_IC.pert.eam.i.", factor=0.15): + rvals = get_rvals(date, ensemble_start,ensemble_end, model) + + outfile = os.path.join(baserundir,outroot+date+"-00000.nc") + # first link the original ic file to the 0th ensemble member + if os.path.exists(outfile): + os.unlink(outfile) + expect(os.path.isfile(original),"ERROR file {} not found".format(original)) + #print("Linking {} to {}".format(original, outfile)) + #rundir = os.path.dirname(outfile) + #print("Linking {} to {}".format(original, os.path.join(rundir,os.path.basename(original)))) + #if os.path.isdir(rundir): + #shutil.rmtree(rundir) + #os.makedirs(rundir) + #os.symlink(original, outfile) + #os.symlink(original, os.path.join(rundir,os.path.basename(original))) + + # for each pair of ensemble members create an ic file with same perturbation opposite sign + month = date[5:7] + year = date[0:3] + + local_path = "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF" + perturb_files = [] + for i in range(ensemble_start,ensemble_end, 2): + print ("HERE rvals[{}] = {}".format((i-1)//2,rvals[(i-1)//2])) + perturb_file = os.path.join("{}".format(month), + "v2.LR.historical_daily-cami_0241.eam.i.M{}.diff.{:03}.nc".format(month,rvals[(i-1)//2])) + dirname = os.path.dirname(os.path.join(local_path,perturb_file)) + if not os.path.isdir(dirname): + print("Creating directory {}".format(dirname)) + os.makedirs(dirname) + perturb_files.append(perturb_file) + + pertroot = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","pert.01") + + for i in range(ensemble_start,ensemble_end, 2): + pfile = os.path.join(local_path, perturb_files.pop(0)) + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-tmp.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-tmp.nc") + print("Creating perturbed init file {}".format(outfile1)) + print("Creating perturbed init file {}".format(outfile2)) + print("Using perturb_file {}".format(pfile)) + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile1, factor)) + t.start() + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile2, -1*factor)) + t.start() + while(threading.active_count() > 1): + time.sleep(1) + + if False: + for i in range(ensemble_start, ensemble_end, 2): + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-00000.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-00000.nc") + outdir1 = baserundir[:-3]+"{:03d}".format(i) + outdir2 = baserundir[:-3]+"{:03d}".format(i+1) + origfile = os.path.basename(original) + print("{} {} ".format(outfile1, os.path.join(outdir1,origfile))) + print("{} {} ".format(outfile2, os.path.join(outdir2,origfile))) + for outdir in (outdir1,outdir2): + if not os.path.isdir(outdir): + os.mkdir(outdir) + print("outdir = {} ".format(outdir)) + if i != 1: + if os.path.isfile(os.path.join(outdir1,origfile)): + os.unlink(os.path.join(outdir1,origfile)) + os.symlink(outfile1, os.path.join(outdir1,origfile)) + print("I made it here = {} ".format(outdir)) + if os.path.isfile(os.path.join(outdir2,origfile)): + os.unlink(os.path.join(outdir2,origfile)) + os.symlink(outfile2, os.path.join(outdir2,origfile)) + + +def create_perturbed_init_file(original, perturb_file, outfile, weight): + ncflint = "ncflint" + if not os.path.isdir(os.path.dirname(outfile)): + os.makedirs(os.path.dirname(outfile)) + pertfile = outfile.replace("-tmp.nc","-00000.nc") + if os.path.isfile(pertfile): + print("Found existing file {}".format(pertfile)) + return # file exists nothing more to do + safe_copy(original, outfile) + if "BWHIST" in original: + cmd = ncflint + " -A -v US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + else: + #cmd = ncflint+" -O -C -v lat,lon,slat,slon,lev,ilev,hyai,hybi,hyam,hybm,US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + cmd = ncflint+" -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + run_cmd(cmd, verbose=True) + if os.path.isfile(outfile): + os.rename(outfile, outfile.replace("-tmp.nc","-00000.nc")) + else: + print("Rename of {} failed".format(outfile)) + +def _main_func(description): + date, model,ensemble_start,ensemble_end = parse_command_line(sys.argv, description) + + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","{}".format(date)) + user = os.getenv("USER") + baserundir = os.path.join("/pscratch/sd/n/{}/".format(user),"v21.SMYLE","v21.LR.BSMYLE."+date[0:7]+".001","run.{:03d}".format(ensemble_start)) + eaminame = os.path.join(sdrestdir,"v21.LR.SMYLE_IC.{}.01.eam.i.{date}-00000.nc".format(date[:7],date=date)) + outroot = "v21.LR.SMYLE_IC.pert.eam.i." + + create_eam_ic_perturbed(eaminame,ensemble_start,ensemble_end, date,baserundir, model, outroot=outroot) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/v2.LR.AUFIRE/restarts/run_generate.sh b/scripts/v2.LR.AUFIRE/restarts/run_generate.sh new file mode 100755 index 0000000..156b506 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/run_generate.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=08 +for year in {2019..2019}; do + export CYLC_TASK_CYCLE_POINT=$year-${mon}-01 + PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py +done + diff --git a/scripts/v2.LR.AUFIRE/restarts/setEnv.scr b/scripts/v2.LR.AUFIRE/restarts/setEnv.scr new file mode 100755 index 0000000..3e3284a --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/setEnv.scr @@ -0,0 +1,4 @@ +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 diff --git a/scripts/v2.LR.AUFIRE/restarts/xfrlist b/scripts/v2.LR.AUFIRE/restarts/xfrlist new file mode 100644 index 0000000..307eda9 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/restarts/xfrlist @@ -0,0 +1,221 @@ +Rvals are [363, 848, 183, 723, 28, 381, 370, 268, 985, 166, 672, 11, 192, 316, 637, 626, 364, 539, 0, 582] +HERE rvals[0] = 363 +HERE rvals[1] = 848 +HERE rvals[2] = 183 +HERE rvals[3] = 723 +HERE rvals[4] = 28 +HERE rvals[5] = 381 +HERE rvals[6] = 370 +HERE rvals[7] = 268 +HERE rvals[8] = 985 +HERE rvals[9] = 166 +HERE rvals[10] = 672 +HERE rvals[11] = 11 +HERE rvals[12] = 192 +HERE rvals[13] = 316 +HERE rvals[14] = 637 +HERE rvals[15] = 626 +HERE rvals[16] = 364 +HERE rvals[17] = 539 +HERE rvals[18] = 0 +HERE rvals[19] = 582 +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.01/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.02/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.363.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.03/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.04/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.848.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.05/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.06/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.183.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.07/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.08/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.723.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.09/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.10/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.028.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.11/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.12/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.381.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.13/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.14/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.370.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.15/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.16/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.268.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.17/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.18/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.985.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.14/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.19/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.20/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.166.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.21/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.22/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.672.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.18/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.23/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.24/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.011.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.25/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.26/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.192.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.27/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.28/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.316.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.29/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.30/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.637.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.31/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.32/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.626.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.33/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.34/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.364.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.35/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.36/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.539.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.37/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.38/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.000.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.39/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Creating perturbed init file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.40/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +Using perturb_file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.582.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.15/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.17/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.20/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.13/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.19/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.05/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.16/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.08/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.07/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.12/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.11/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.03/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.02/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.01/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.06/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.10/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.04/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +Found existing file /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.09/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-00000.nc +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.626.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.31/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.364.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.34/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.582.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.40/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.539.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.35/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.316.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.27/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.364.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.33/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.672.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.21/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.637.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.30/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.192.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.25/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.011.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.23/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.539.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.36/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.626.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.32/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.637.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.29/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.192.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.26/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.316.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.28/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.37/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.38/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.582.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.39/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.011.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.24/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts + stat: 127 + + stat: 127 + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + +RUN: ncflint -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w -0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/08/v2.LR.historical_daily-cami_0241.eam.i.M08.diff.672.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/2019-08-01/v21.LR.SMYLE_IC.2019-08.01.eam.i.2019-08-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.2019-08.01/pert.22/v21.LR.SMYLE_IC.pert.eam.i.2019-08-01-tmp.nc +FROM: /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/restarts + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + + stat: 127 + + errput: ncflint: error while loading shared libraries: libcupti.so.12: cannot open shared object file: No such file or directory + diff --git a/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-08-5year.scr b/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-08-5year.scr new file mode 100755 index 0000000..9db3f44 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-08-5year.scr @@ -0,0 +1,46 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 2 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +./xmlchange STOP_DATE=20250101 +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-cntl-08-5year.scr b/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-cntl-08-5year.scr new file mode 100755 index 0000000..4f6b82d --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/change_STOP_DATE-AUFIRE-cntl-08-5year.scr @@ -0,0 +1,50 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 21 +set embr = 30 + +@ mb = $smbr +@ me = $embr + +#set MAIN_CASE = v2.LR.SSP370_AUFIRE-cntl.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +set MAIN_CASE = v2.LR.SSP370_AUFIRE-cntl.${year}-${mon}.0${mbr} +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +./xmlchange STOP_DATE=20250101 +./xmlchange STOP_N=18 +./xmlchange REST_N=18 +./xmlchange CONTINUE_RUN=TRUE +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/cpMosartFile.scr b/scripts/v2.LR.AUFIRE/scripts/cpMosartFile.scr new file mode 100755 index 0000000..3ad3f42 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/cpMosartFile.scr @@ -0,0 +1,48 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 4 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set RUN_DIR = run.00${mbr} + set ARCH_DIR = archive.00${mbr} +else + set RUN_DIR = run.0${mbr} + set ARCH_DIR = archive.0${mbr} +endif + +set RUNROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${RUN_DIR} +set ARCHROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${ARCH_DIR} + +cd $RUNROOT +echo $RUNROOT +cp $ARCH_DIR/rof/hist/*.mosart.h1.2019-08-02-00000.nc . + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.BSMYLE.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.BSMYLE.sh new file mode 100755 index 0000000..793bd69 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.BSMYLE.sh @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +#useyear=1980 +#useyear=1981 +#useyear=1982 +useyear=1980 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-TEST.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-TEST.sh new file mode 100755 index 0000000..ce1cab1 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-TEST.sh @@ -0,0 +1,567 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE_aug4_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2019-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +#RUN_REFDIR="/global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_${iyr}/archive/rest/archive/rest/2019-01-01-00000" +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/" +RUN_REFCASE="v2.LR.SSP370_${iyr}" +RUN_REFDATE="2019-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="18" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20190701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_withGFED_2018-202007_c230803.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_withGFED_2018-202007_c230803.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_withGFED_2018-202007_c230803.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_withGFED_2018-202007_c230803.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_withGFED_2018-202007_c230803.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_withGFED_2018-202007_c230803.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_withGFEDscale25_2018-202007_c230804.nc', + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # cp ${RUN_REFDIR}/* ${CASE_RUN_DIR} + mkdir -p ${CASE_RUN_DIR} + # cp ${RUN_REFDIR}/rpointer.atm ${CASE_RUN_DIR} + # ln -s ${RUN_REFDIR}/v2.* ${CASE_RUN_DIR} + + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.atm ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.drv ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ice ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.lnd ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ocn ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.rof ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.cpl.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h1.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h2.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h3.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h4.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h5.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h6.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.i.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.rs.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpaso.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpassi.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.21-30.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.21-30.sh new file mode 100755 index 0000000..6a744cd --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.21-30.sh @@ -0,0 +1,549 @@ +#!/bin/bash +####!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +array=( 21 22 23 24 25 26 27 28 29 30 ) +#array=( 1 2 ) + +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=08 + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE-cntl.${useyear}-${usemonth}.${mbr}" +CASE_GROUP="v2.LR" + +#if [[ ${imbr} -eq "1" ]] +#then + MAIN_CASE_NAME="v2.LR.SSP370_AUFIRE-cntl.${useyear}-${usemonth}.001" +#fi +CASE_GROUP="v2.LR" + + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + #PELAYOUT=${layout} + WALLTIME="12:00:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="36:00:00" + STOP_OPTION="nmonths" + STOP_N="60" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20250101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="60" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +##OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true + +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.sh new file mode 100755 index 0000000..7e65d13 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-cntl.sh @@ -0,0 +1,548 @@ +#!/bin/bash +####!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +#array=( 1 2 ) + +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=08 + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE-cntl.${useyear}-${usemonth}.${mbr}" +CASE_GROUP="v2.LR" + +if [[ ${imbr} -eq "1" ]] +then + MAIN_CASE_NAME="v2.LR.SSP370_AUFIRE-cntl.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + #PELAYOUT=${layout} + WALLTIME="12:00:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nmonths" + STOP_N="48" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20230801" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="24" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +##OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true + +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-old.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-old.sh new file mode 100755 index 0000000..5be6a15 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE-old.sh @@ -0,0 +1,528 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE-Full_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2020-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFDIR="/global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_${iyr}/archive/rest/archive/rest/2019-01-01-00000" +RUN_REFCASE="v2.LR.SSP370_${iyr}" +RUN_REFDATE="2019-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nmonths" + STOP_N="18" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20190701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/trop_mozart_aero/emis/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016-c230601.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.21-30.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.21-30.sh new file mode 100755 index 0000000..333fce4 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.21-30.sh @@ -0,0 +1,560 @@ +#!/bin/bash +####!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +array=( 21 22 23 24 25 26 27 28 29 30 ) +#array=( 1 2 ) + +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=08 + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.${mbr}" +CASE_GROUP="v2.LR" + +#if [[ ${imbr} -eq "1" ]] +#then + MAIN_CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.001" +#fi +CASE_GROUP="v2.LR" + + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + #PELAYOUT=${layout} + WALLTIME="12:00:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="11" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20200701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="11" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +##OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true + +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_withGFED_2018-202007_c230803.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_withGFED_2018-202007_c230803.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_withGFED_2018-202007_c230803.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_withGFED_2018-202007_c230803.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_withGFED_2018-202007_c230803.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_withGFED_2018-202007_c230803.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_withGFEDscale25_2018-202007_c230804.nc', + ext_frc_type = 'INTERP_MISSING_MONTHS' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh new file mode 100755 index 0000000..4e39977 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh @@ -0,0 +1,559 @@ +#!/bin/bash +####!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +#array=( 1 2 ) + +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=08 + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.${mbr}" +CASE_GROUP="v2.LR" + +if [[ ${imbr} -eq "1" ]] +then + MAIN_CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.AUFIRE/${MAIN_CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + #PELAYOUT=${layout} + WALLTIME="12:00:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="11" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20200701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="11" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +##OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true + +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_withGFED_2018-202007_c230803.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_withGFED_2018-202007_c230803.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_withGFED_2018-202007_c230803.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_withGFED_2018-202007_c230803.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_withGFED_2018-202007_c230803.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_withGFED_2018-202007_c230803.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_withGFEDscale25_2018-202007_c230804.nc', + ext_frc_type = 'INTERP_MISSING_MONTHS' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh-SAFE b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh-SAFE new file mode 100755 index 0000000..835546d --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/run.v2.LR.SSP370_YYYY-AUFIRE.sh-SAFE @@ -0,0 +1,544 @@ +#!/bin/bash +####!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) + +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + + +# For debugging, uncomment libe below +#set -x + +useyear=2019 +usemonth=08 + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.${mbr}" +CASE_GROUP="v2.LR" + +if [[ ${imbr} -eq "1" ]] +then + MAIN_CASE_NAME="v2.LR.SSP370_AUFIRE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="${SCRATCH}/v2.LR.AUFIRE/${MAIN_CASE_NAME}" +CASE_ROOT="${SCRATCH}/v2.LR.AUFIRE/${MAIN_CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + #PELAYOUT=${layout} + WALLTIME="12:00:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="11" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20200701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="11" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.AUFIRE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true + +if [[ ${imbr} -eq "1" ]] +then + do_case_build=true +else + do_case_build=false +fi +do_case_submit=false +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_withGFED_2018-202007_c230803.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_withGFED_2018-202007_c230803.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_withGFED_2018-202007_c230803.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_withGFED_2018-202007_c230803.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_withGFED_2018-202007_c230803.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_withGFED_2018-202007_c230803.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_withGFEDscale25_2018-202007_c230804.nc', + ext_frc_type = 'INTERP_MISSING_MONTHS' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + ./xmlchange RUN_STARTDATE=${RUN_REFDATE} + + # cp ${RUN_REFDIR}/* ${CASE_RUN_DIR} + mkdir -p ${CASE_RUN_DIR} + cp ${RUN_REFDIR}/rpointer.atm ${CASE_RUN_DIR} + ln -s ${RUN_REFDIR}/v2.* ${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.AUFIRE/scripts/submit-cntl-last4mons.scr b/scripts/v2.LR.AUFIRE/scripts/submit-cntl-last4mons.scr new file mode 100755 index 0000000..a216591 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/submit-cntl-last4mons.scr @@ -0,0 +1,47 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 21 +set embr = 29 + +@ mb = $smbr +@ me = $embr + + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} + set MAIN_CASE = v2.LR.SSP370_AUFIRE-cntl.${year}-${mon}.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} + set MAIN_CASE = v2.LR.SSP370_AUFIRE-cntl.${year}-${mon}.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +./xmlchange STOP_N=17,REST_N=17 +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/submitCases.scr b/scripts/v2.LR.AUFIRE/scripts/submitCases.scr new file mode 100755 index 0000000..32bd916 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/submitCases.scr @@ -0,0 +1,45 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 22 +set embr = 30 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/turnOff-AUFIRE-08-5year.scr b/scripts/v2.LR.AUFIRE/scripts/turnOff-AUFIRE-08-5year.scr new file mode 100755 index 0000000..4098525 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/turnOff-AUFIRE-08-5year.scr @@ -0,0 +1,59 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 21 +set embr = 30 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +mv $CASEROOT/user_nl_eam $CASEROOT/user_nl_eam-aufiresON +cp $E3SM_TOOLS_ROOT/user_nl_files/user_nl_eam-aufiresOFF $CASEROOT/user_nl_eam +./xmlchange CONTINUE_RUN=TRUE +# end in July 2021 +#./xmlchange STOP_N=13 + +# end in July 2022 +#./xmlchange STOP_N=25 + +# end in December 2024 (54 months) +./xmlchange STOP_DATE=20250101 +./xmlchange STOP_N=27 +./xmlchange REST_N=27 +./xmlchange RESUBMIT=1 +./case.submit + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/scripts/turnOff_DOUT_S.scr b/scripts/v2.LR.AUFIRE/scripts/turnOff_DOUT_S.scr new file mode 100755 index 0000000..8fd6680 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/scripts/turnOff_DOUT_S.scr @@ -0,0 +1,45 @@ +#!/bin/csh +### set env variables +module load ncl nco + +setenv E3SM_TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.AUFIRE/ + +# ... +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) +#foreach mon ( 02 05 08 11 ) + +# case name counter +set smbr = 1 +set embr = 20 + +@ mb = $smbr +@ me = $embr + +set MAIN_CASE = v2.LR.SSP370_AUFIRE-cntl.${year}-${mon}.001 + +foreach mbr ( `seq $mb $me` ) +if ($mbr < 10) then + set CASE_DIR = case_scripts.00${mbr} +else + set CASE_DIR = case_scripts.0${mbr} +endif + +set CASEROOT = $SCRATCH/v2.LR.AUFIRE/${MAIN_CASE}/${CASE_DIR} + +cd $CASEROOT +echo $CASEROOT +./xmlchange DOUT_S=FALSE + +end # member loop +end # member loop +end # member loop + +exit + diff --git a/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresOFF b/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresOFF new file mode 100644 index 0000000..0400767 --- /dev/null +++ b/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresOFF @@ -0,0 +1,30 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + diff --git a/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresON b/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresON new file mode 100644 index 0000000..a2e2d9e --- /dev/null +++ b/scripts/v2.LR.AUFIRE/user_nl_files/user_nl_eam-aufiresON @@ -0,0 +1,28 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_withGFED_2018-202007_c230803.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_withGFED_2018-202007_c230803.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_withGFED_2018-202007_c230803.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_withGFED_2018-202007_c230803.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_withGFED_2018-202007_c230803.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_withGFED_2018-202007_c230803.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/emis/emissions_ssp370_smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_withGFEDscale25_2018-202007_c230804.nc', + ext_frc_type = 'INTERP_MISSING_MONTHS' + diff --git a/scripts/v2.LR.LE-rerun/README.extractKai.Restarts b/scripts/v2.LR.LE-rerun/README.extractKai.Restarts new file mode 100644 index 0000000..447e2dc --- /dev/null +++ b/scripts/v2.LR.LE-rerun/README.extractKai.Restarts @@ -0,0 +1,11 @@ + 551 mkdir v2.LR.historical_0251; cd v2.LR.historical_0251; + 557 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0251 "archive/rest/1975-01-01-00000/*" + 561 mkdir v2.LR.historical_0211; cd *211 + 564 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0211 "archive/rest/*" | less + 565 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0211 "archive/rest/1976-01-01-00000/*" + 569 mkdir v2.LR.historical_0191; cd *191 + 571 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0191 "archive/rest/1976-01-01-00000/*" & + 573 mkdir v2.LR.historical_0151; cd *151 + 577 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0151 "archive/rest/1975-01-01-00000/*" + 580 mkdir v2.LR.historical_0121; cd *121 + 582 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0121 "archive/rest/1976-01-01-00000/*" & diff --git a/scripts/v2.LR.LE-rerun/env_mach/env_mach_specific.xml b/scripts/v2.LR.LE-rerun/env_mach/env_mach_specific.xml new file mode 100644 index 0000000..32320b7 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/env_mach/env_mach_specific.xml @@ -0,0 +1,87 @@ + + +
+ These variables control the machine dependent environment including + the paths to compilers and libraries external to cime such as netcdf, + environment variables for use in the running job should also be set here. +
+ + + char + executable name + + + char + redirect for job output + + + + /usr/share/lmod/8.3.1/init/perl + /usr/share/lmod/8.3.1/init/python + /usr/share/lmod/8.3.1/init/sh + /usr/share/lmod/8.3.1/init/csh + /usr/share/lmod/lmod/libexec/lmod perl + /usr/share/lmod/lmod/libexec/lmod python + module + module + + cray-hdf5-parallel + cray-netcdf-hdf5parallel + cray-parallel-netcdf + PrgEnv-gnu + PrgEnv-nvidia + cudatoolkit + craype-accel-nvidia80 + craype-accel-host + perftools-base + perftools + darshan + + + PrgEnv-gnu/8.3.3 + gcc/11.2.0 + + + PrgEnv-nvidia + nvidia/22.5 + + + PrgEnv-aocc + aocc/3.2.0 + + + craype-accel-host + cray-libsci + craype + cray-mpich/8.1.27 + cray-hdf5-parallel/1.12.2.7 + cray-netcdf-hdf5parallel/4.9.0.7 + cray-parallel-netcdf/1.12.3.7 + cmake/3.22.0 + + + + 1 + 1 + 128M + spread + threads + FALSE + /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch + software + MPI_Bcast + + + -1 + + + srun + + --label + -n {{ total_tasks }} -N {{ num_nodes }} + -c $SHELL{echo 256/`./xmlquery --value MAX_MPITASKS_PER_NODE`|bc} + $SHELL{if [ 128 -ge `./xmlquery --value MAX_MPITASKS_PER_NODE` ]; then echo "--cpu_bind=cores"; else echo "--cpu_bind=threads";fi;} + -m plane=$SHELL{echo `./xmlquery --value MAX_MPITASKS_PER_NODE`} + + + diff --git a/scripts/v2.LR.LE-rerun/old/IC/atm-IC/run.v2.LR.cami.sh b/scripts/v2.LR.LE-rerun/old/IC/atm-IC/run.v2.LR.cami.sh new file mode 100755 index 0000000..058b9ec --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/IC/atm-IC/run.v2.LR.cami.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + + array=( 0241 ) +refarray=( 1991 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${refarray[$ctr]} +echo ${refyear} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="m4195" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical_daily-cami_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-cami/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam +inithist = 'DAILY' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + ctr=$((ctr+1)) + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.LE-rerun/old/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh b/scripts/v2.LR.LE-rerun/old/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh new file mode 100755 index 0000000..4213896 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/IC/monthlyRestarts/run.v2.LR.mkMonthlyRestarts.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + + array=( 0201 ) +refarray=( 1970 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${refarray[$ctr]} +echo ${refyear} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical_monthly-restarts_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-monthlyRestarts/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="3" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam +inithist = 'DAILY' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + ctr=$((ctr+1)) + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.LE-rerun/old/README.levels b/scripts/v2.LR.LE-rerun/old/README.levels new file mode 100644 index 0000000..9b22167 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/README.levels @@ -0,0 +1,46 @@ +bash +module load cray-hdf5 +module load cray-netcdf +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 +module load python + + +ncks -v lat_vertices,lon_vertices,time_bnds v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000.nc eam.vars.nc + + double time_bnds(time, nbnd) ; + time_bnds:_FillValue = NaN ; + time_bnds:long_name = "time interval endpoints" ; + double lat_vertices(ncol, nv) ; + lat_vertices:_FillValue = NaN ; + lat_vertices:long_name = "Gridcell latitude vertices" ; + lat_vertices:coordinates = "lon lat" ; + double lon_vertices(ncol, nv) ; + lon_vertices:_FillValue = NaN ; + lon_vertices:long_name = "Gridcell longitude vertices" ; + lon_vertices:coordinates = "lon lat" ; + +Original: +netcdf v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000 { +dimensions: + time = 1 ; + ncol = 48602 ; + lev = 72 ; + ilev = 73 ; + nv = 1 ; + nbnd = 2 ; + +Perturbed +netcdf v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000 { +dimensions: + time = UNLIMITED ; // (1 currently) + ncol_d = 48602 ; + lev = 72 ; + ilev = 73 ; + + :history = "Thu Jul 13 10:44:31 2023: ncflint -O -C -v lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/11/v2.LR.historical_daily-cami_0241.eam.i.M11.diff.762.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1971-11.01/1971-11-01/v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1971-11.01/pert.03/v21.LR.SMYLE_IC.pert.eam.i.1971-11-01-tmp.nc\n", + "Wed May 24 11:27:41 2023: ncdiff /global/cfs/cdirs/mp9/SMYLE/S2S_perts/11//v2.LR.historical_daily-cami_0241.eam.i.1992-11-11-00000.nc /global/cfs/cdirs/mp9/SMYLE/S2S_perts/11//v2.LR.historical_daily-cami_0241.eam.i.2006-11-07-00000.nc /global/cfs/cdirs/mp9/SMYLE/S2S_perts_DIFF/11//v2.LR.historical_daily-cami_0241.eam.i.M11.diff.762.nc\n", + "created on 02/19/23 15:32:29" ; + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/README.generate.bash b/scripts/v2.LR.LE-rerun/old/restarts/README.generate.bash new file mode 100644 index 0000000..626d932 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/README.generate.bash @@ -0,0 +1,28 @@ +export CYLC_TASK_CYCLE_POINT="1959-11-01" + +### use bash + +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 +module load python + +module load python/3.9-anaconda-2021.11 + +## Edit start years to set up +Step 1: ./create-v2.SMYLE.IC_RESTARTS.csh +Step 2: ./run_generate.sh +Step 3: ./fix_xtime.sh (change xtime to xtime.orig in mpaso.rst files) +Step 4: ./fix_ncol_d.sh (change ncol_d to ncol in eam.i files) + + +export CYLC_TASK_CYCLE_POINT=1970-11-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +export CYLC_TASK_CYCLE_POINT=1970-08-01 +PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + + +## Don't have to do this anymore, now that I've fixed the code: +# PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py diff --git a/scripts/v2.LR.LE-rerun/old/restarts/README.history b/scripts/v2.LR.LE-rerun/old/restarts/README.history new file mode 100644 index 0000000..ca2d14f --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/README.history @@ -0,0 +1,714 @@ + 1 ls -ld /global/cscratch1/sd/nanr/archive/v2.LR.historical_0* + 2 less ~/README.zstashExtract + 3 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/2015-01-01-00000/*" + 4 mv archive/rest/2015-01-01-00000/v2.LR.historical_0111.* .. + 5 cd ../../case_scripts/ + 6 ./case.submit + 7 ./xmlquery EXEROOT + 8 ls .. + 9 ls ../build/ + 10 ls -ltr ../build/ + 11 qme + 12 exit + 13 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 14 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 15 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 16 pwd + 17 ls + 18 history + 19 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 20 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 21 pwd + 22 ls -l + 23 ls zstash/ + 24 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 25 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230306.log + 26 exit + 27 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 28 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0241 2>&1 | tee zstash_update_smbb-hist-0241_20230306.log + 29 bg + 30 ls -ltr + 31 jobs + 32 ls -ltr + 33 less zstash_update_smbb-hist-0241_20230306.log + 34 cd .. + 35 mkdir 261 + 36 cd 261/ + 37 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0261 2>&1 | tee zstash_update_smbb-hist-0261_20230306.log & + 38 ls -ltr ~/zstash_create_*261* + 39 less /global/homes/n/nanr/zstash_create_smbb-hist-0261_20230203.log + 40 less /global/homes/n/nanr/zstash_create_smbb-hist-0271_20230203.log + 41 cd ../ + 42 mkdir 271 + 43 cd 271/ + 44 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0271 2>&1 | tee zstash_update_smbb-hist-0271_20230306.log & + 45 ls -ltr + 46 jobs + 47 cd ../ + 48 mkdir 281 + 49 cd 281/ + 50 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0281 2>&1 | tee zstash_update_smbb-hist-0281_20230306.log + 51 bg + 52 cd .. + 53 mkdir 291 + 54 cd 291/ + 55 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0291 2>&1 | tee zstash_update_smbb-hist-0291_20230306.log + 56 bg + 57 cd .. + 58 mkdir 101 + 59 cd 101 + 60 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 2>&1 | tee zstash_update_smbb-hist-0101_20230306.log + 61 bg + 62 jobs + 63 cd ../ + 64 mkdir 151 + 65 cd 151/ + 66 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0151 2>&1 | tee zstash_update_smbb-hist-0151_20230306.log + 67 bg + 68 cd .. + 69 mkdir 201 + 70 cd 201 + 71 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 2>&1 | tee zstash_update_smbb-hist-0201_20230306.log + 72 less ~/zstash_create_*201* + 73 cd .. + 74 mkdir 251 + 75 cd 251/ + 76 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0251 2>&1 | tee zstash_update_smbb-hist-0251_20230306.log + 77 bg + 78 cd ../201 + 79 ls + 80 jobs + 81 cd .. + 82 mkdir 301 + 83 cd 301 + 84 zstash check --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0301 2>&1 | tee zstash_update_smbb-hist-0301_20230306.log + 85 bg + 86 cd ../../ + 87 cd 210/ + 88 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" + 89 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "rest/*" | less + 90 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*" | less + 91 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | less + 92 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "atm/hist/*h0*" | wc + 93 zstash ls --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 "lnd/hist/*h0*" | wc + 94 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307.log + 95 bg + 96 less zstash_check_210.001-20230307.log + 97 fg + 98 ls + 99 mv zstash zstash-0ld + 100 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 101 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 102 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 103 exit + 104 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 105 less ~/README.zstashInstructions + 106 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/*" + 107 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2021-01-01-00000/*" + 108 ls + 109 cd archive/rest/2021-01-01-00000/ + 110 cp * /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/run/ + 111 scr + 112 exit + 113 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 114 ls -ltr + 115 ls -l archive/rest/ + 116 ls archive/atm/hist/*h0* + 117 b + 118 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 119 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 120 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-hist-0201_20230313.log + 121 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201-new --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create-new_smbb-hist-0201_20230313.log + 122 cd .. + 123 b + 124 cd v2.LR.historical-smbb_0201/ + 125 hsi + 126 exit + 127 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 128 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 129 pwd + 130 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 131 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0131_20230313.log + 132 exit + 133 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 134 cd ../../ + 135 cd 210/ + 136 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307a.log + 137 hsi "ls -l /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001" + 138 ls -ltr + 139 less zstash_check_210.001-20230307b.log + 140 hsi + 141 less ~/README.zstashInstructions + 142 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 143 pwd + 144 less ~/README.zstashExtract + 145 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/rest/2101-01-01-00000/*" + 146 bg + 147 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 "archive/run/*" + 148 fg + 149 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 150 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/" + 151 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ "archive/ocn/*" + 152 exit + 153 screen + 154 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 155 pwd + 156 less ~/zstash_update_320.003_20230413.log + 157 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 158 ls + 159 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 160 pwd + 161 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413.log + 162 history + 163 pwd + 164 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_480.002_20230413a.log + 165 exit + 166 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 167 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/archive/rest/*" + 168 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "/\archive/rest/*" + 169 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" + 170 zstash ls --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/*" > doh + 171 less doh + 172 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 173 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000" + 174 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000 + 175 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0111 "archive/rest/1976-01-01-00000/*" + 176 cd ../ + 177 mkdir v2.LR.historical_0121 + 178 cd v2.LR.historical_0121/ + 179 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0121 "archive/rest/1976-01-01-00000/*" + 180 cd .. + 181 mkdir v2.LR.historical_0131 + 182 cd v2.LR.historical_0131/ + 183 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0131 "archive/rest/1976-01-01-00000/*" + 184 cd .. + 185 mkdir v2.LR.historical_0141 + 186 cd v2.LR.historical_0141/ + 187 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0141 "archive/rest/1976-01-01-00000/*" + 188 cd .. + 189 mkdir v2.LR.historical_0161 + 190 cd v2.LR.historical_0161/ + 191 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0161 "archive/rest/1976-01-01-00000/*" + 192 cd .. + 193 mkdir v2.LR.historical_0171 + 194 cd v2.LR.historical_0171/ + 195 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0171 "archive/rest/1976-01-01-00000/*" + 196 cd .. + 197 mkdir v2.LR.historical_0181 + 198 cd v2.LR.historical_0181/ + 199 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0181 "archive/rest/1976-01-01-00000/*" + 200 cd .. + 201 mkdir v2.LR.historical_0191 + 202 cd v2.LR.historical_0191/ + 203 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0191 "archive/rest/1976-01-01-00000/*" + 204 cd .. + 205 mkdir v2.LR.historical_0211 + 206 cd v2.LR.historical_0211/ + 207 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0211 "archive/rest/1976-01-01-00000/*" + 208 cd .. + 209 mkdir v2.LR.historical_0221 + 210 cd v2.LR.historical_0221/ + 211 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0221 "archive/rest/1976-01-01-00000/*" + 212 pwd + 213 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 214 pwd + 215 zstash update --dry-run + 216 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 217 exit + 218 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 219 pwd + 220 zstash update --dry-run + 221 history + 222 bash + 223 exit + 224 pwd + 225 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 226 zstash update --dry-run + 227 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 228 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.111_20230414.log + 229 ls -ltr + 230 cd .. + 231 cd .. + 232 ls -ltr + 233 cd E3SMv2 + 234 ls -ltr + 235 cd v2.LR.SSP370_0281 + 236 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281 + 237 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0281/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.281_20230414.log + 238 ../cd v2.LR.SSP370_0211 + 239 ls + 240 pwd + 241 cd ../v2.LR.SSP370_0211/ + 242 zstash update --dry-run --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211 + 243 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0211/ 2>&1 | tee ~nanr/zstash_update_v2.ssp370.211_20230414.log + 244 exit + 245 pwd + 246 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 247 vi ~/README.zstashExtract + 248 zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0231 "archive/rest/1976-01-01-00000/*" + 249 exit + 250 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 251 vi ~/README.zstashInstructions + 252 zstash check --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 2>&1 | tee zstash_check_0251_20230320.log + 253 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 254 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 255 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230320.log + 256 screen + 257 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 258 pwd + 259 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_320.003_20230321.log + 260 $ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 261 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 262 vi ~/README.zstashInstructions + 263 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 264 pwd + 265 fg + 266 pwd + 267 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230320.log + 268 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 269 cd /global/cfs/cdirs/mp9/archive/ + 270 ls + 271 cd e3smv1 + 272 ls + 273 pwd + 274 ls -ltr + 275 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 276 cd /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0251 + 277 zstash update --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0251 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_0251_20230321.log + 278 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 279 pwd + 280 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/ + 281 ls -ltr + 282 cd v2.LR.historical-smbb_0101 + 283 vi ~/README.zstashInstructions + 284 pwd + 285 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0101_20230203.log + 286 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 287 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 + 288 pwd + 289 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0201_20230314.log + 290 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 291 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 + 292 pwd + 293 ls -ltr ~/zstash* + 294 less /global/homes/n/nanr/zstash_update_smbb-ssp370-0181_20230314.log + 295 pwd + 296 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0191 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0191_20230314.log + 297 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 298 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 299 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 300 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" . 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 301 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_320.003_20230413.log + 302 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 303 hsi ls /home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 + 304 hsi ls /home/c/ccsm/E3SMv1-LE/ + 305 hsi ls -l /home/c/ccsm/E3SMv1-LE/ + 306 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 307 pwd + 308 ls + 309 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.003_20230321.log + 310 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 311 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 312 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 313 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 + 314 pwd + 315 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0171 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0171_20230314.log + 316 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 317 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 + 318 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0131 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0131_20230217.log + 319 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 320 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 + 321 pwd + 322 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0241 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0241_20230314.log + 323 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 324 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 325 pwd + 326 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230314.log + 327 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 328 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 + 329 pwd + 330 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0141 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0141_20230314.log + 331 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 332 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 + 333 pwd + 334 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0181 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0181_20230314.log + 335 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 336 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 + 337 pwd + 338 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0121 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0121_20230217.log + 339 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 340 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 + 341 pwd + 342 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0151 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0151_20230314.log + 343 fg + 344 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 345 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 + 346 pwd + 347 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0231 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0231_20230314.log + 348 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 349 cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 350 pwd + 351 cfs + 352 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 353 ls -ltr + 354 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230217.log + 355 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 356 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 + 357 pwd + 358 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0161 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0161_20230314.log + 359 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 360 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 361 pwd + 362 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0101_20230314.log + 363 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 364 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0101 + 365 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 366 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 + 367 pwd + 368 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0211 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0211_20230314.log + 369 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 370 pwd + 371 cd /global/cfs/cdirs/mp9/archive/e3smv1 + 372 ls -ltr + 373 cd e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + 374 ls -ltr + 375 ls zstash/ + 376 rmdir zstash/ + 377 pwd + 378 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 379 zstash create --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_480.002_20230413.log + 380 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 381 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 + 382 pwd + 383 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0251 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0251_20230314.log + 384 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 385 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 + 386 pwd + 387 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0301 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0301_20230314.log + 388 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 389 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 + 390 pwd + 391 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0261 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0261_20230314.log + 392 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 393 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 + 394 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0271 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0271_20230314.log + 395 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 396 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 + 397 pwd + 398 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0281 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0281_20230314.log + 399 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 400 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 + 401 pwd + 402 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0111 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_create_smbb-ssp370-0111_20230314.log + 403 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 404 jobs + 405 cd /global/cscratch1/sd/nanr + 406 cd checkZstash/210/ + 407 ls + 408 pwd + 409 ls + 410 mv zstash zstash-a + 411 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 412 ls -ltr + 413 hsi + 414 zstash check --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 2>&1 | tee zstash_check_210.001-20230307b.log + 415 cd zstash + 416 pwd + 417 cd /global/cfs/cdirs/mp9/archive/ + 418 cd e3smv1 + 419 ls + 420 cd *210.001 + 421 ls -ltr + 422 cd zstash/ + 423 ls + 424 pwd + 425 less index.db + 426 sqlite3 index.db + 427 pwd + 428 cd .. + 429 zstash update --hpss=/home/c/ccsm/E3SMv1-LE/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_210.001-20230307a.log + 430 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 431 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 + 432 pwd + 433 zstash update --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.SSP370-smbb_0221 --exclude="rest/???[!05]-*/" 2>&1 | tee ~nanr/zstash_update_smbb-ssp370-0221_20230314.log + 434 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 435 cd /global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.historical-smbb_0201 + 436 zstash create --hpss=/home/c/ccsm/E3SMv2-SMBB/v2.LR.historical-smbb_0201 --exclude="rest/???[!05]-*/" --maxsize 128 . 2>&1 | tee ~nanr/zstash_create_smbb-hist-0201_20230313.log + 437 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 438 vi ~/README.zstashExtract + 439 cd ../ + 440 ls + 441 cd v2.LR.historica_0201 + 442 cd v2.LR.historical_0201 + 443 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1990-01-01-00000/*" + 444 zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1970-01-01-00000/*" + 445 qme + 446 exit + 447 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 448 less extract_zstash.sh + 449 hsi ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 450 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 451 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*" + 452 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/" + 453 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/" + 454 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive" + 455 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/" + 456 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 457 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 458 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 459 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" > /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 460 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" >& /global/cscratch1/sd/nanr/hpss.archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 461 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*h0*.nc" + 462 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist/*.nc" + 463 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/*" + 464 zstash ls -l --hpss=/home/s/slgs/20200116.en1.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist/ + 465 less ~/doh + 466 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 467 history + 468 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl/atm/hist" + 469 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl + 470 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm" + 471 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl "archive/20191024.en1.LE_ens_1_ssp370.ne30_oECv3_ICG.compy/atm/hist" + 472 zstash ls -l --hpss=/home/s/slgs/20200119.en3.LE_ens_ssp370.ne30_oECv3_ICG.cori-knl >& ~nanr/doh1 + 473 less ~/doh1 + 474 tcsh + 475 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 476 less ~/README.zstashInstructions + 477 hsi ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 478 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 + 479 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest" + 480 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "rest" + 481 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 | less + 482 cd v2.LR.SSP370_0111 + 483 ls + 484 cd archive/ + 485 ls + 486 mkdir rest + 487 cd rest + 488 ls + 489 zstash ls --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 490 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/" + 491 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2019-01-01-00000/*" + 492 cd ../../../v2.LR.SSP370_0121/archive/ + 493 mkdir rest; cd rest + 494 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0121 "archive/rest/2019-01-01-00000/*" + 495 source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh + 496 less ~/README.zstashInstructions + 497 less ~/README.zstashExtract + 498 zstash extract --hpss=/home/c/ccsm/E3SMv2/v2.LR.SSP370_0111 "archive/rest/2020-01-01-00000/*" + 499 exit + 500 tcsh + 501 tcsh + 502 exit + 503 tcsh + 504 less /global/cfs/cdirs/ + 505 cd /global/cfs/cdirs/ + 506 cd ccsm1 + 507 ls + 508 cd people + 509 cd nanr/ + 510 cd e3sm_tags + 511 ls + 512 cd E3SMv2.1 + 513 ls + 514 history + 515 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 516 mv E3SM/ E3SM-safe + 517 git clone -b maint-2.1 --recursive https://github.com/E3SM-Project/E3SM.git + 518 hsi /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 519 hsi ls /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 520 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 521 hsi "cd /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/; chmod 744 00005d.tar" + 522 hsi ls -l /home/c/ccsm/E3SMv2/v2.LR.SSP370_0111/00005d.tar + 523 wk + 524 tcsh + 525 tcsh + 526 ls -ltr + 527 ./create-v2.SMYLE.IC_RESTARTS.csh + 528 less ~/README.nco + 529 module load cray-hdf5 + 530 module load cray-netcdf + 531 module load nco + 532 ./create-v2.SMYLE.IC_RESTARTS.csh + 533 vi ~/README.zstashExtract + 534 vi create-v2.SMYLE.IC_RESTARTS.csh + 535 scp cheyenne.ucar.edu:/glade/work/nanr/cesm_tags/CASE_tools/cesm2-smyle/restarts/generate_cami_ensemble_offline.py . + 536 less ~/.alias.nanr + 537 pwd + 538 vi generate_cami_ensemble_offline.py + 539 pwd + 540 fg + 541 vi create-v2.SMYLE.IC_RESTARTS.csh + 542 vi generate_cami_ensemble_offline.py + 543 vi create-v2.SMYLE.IC_RESTARTS.csh + 544 vi generate_cami_ensemble_offline.py + 545 vi create-v2.SMYLE.IC_RESTARTS.csh + 546 vi generate_cami_ensemble_offline.py + 547 vi create-v2.SMYLE.IC_RESTARTS.csh + 548 ./create-v2.SMYLE.IC_RESTARTS.csh + 549 vi create-v2.SMYLE.IC_RESTARTS.csh + 550 vi generate_cami_ensemble_offline.py + 551 vi create-v2.SMYLE.IC_RESTARTS.csh + 552 grep CESM2_TOOLS_ROOT * + 553 vi create-v2.SMYLE.IC_RESTARTS.csh + 554 ./create-v2.SMYLE.IC_RESTARTS.csh + 555 spack env activate gcc + 556 exit + 557 source ~/.alias.bash + 558 tools + 559 cd e3sm/v2/scripts/v2.SMYLE/restarts/ + 560 ls -ltr + 561 vi generate_cami_ensemble_offline.py + 562 less create-v2.SMYLE.IC_RESTARTS.csh + 563 export CYLC_TASK_CYCLE_POINT=1959-01-01 + 564 ./generate_cami_ensemble_offline.py + 565 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime + 566 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime ./generate_cami_ensemble_offline.py + 567 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME ./generate_cami_ensemble_offline.py + 568 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 569 module avail python + 570 module load python/3.9 + 571 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 572 vi generate_cami_ensemble_offline.py + 573 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 574 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/cases/ + 575 ls /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/cases/ + 576 fg + 577 vi generate_cami_ensemble_offline.py + 578 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 579 ls /global/cfs/cdirs/mp9/ + 580 vi generate_cami_ensemble_offline.py + 581 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 582 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-01.01/ + 583 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 584 vi generate_cami_ensemble_offline.py + 585 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 586 history + 587 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 588 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 589 less ~/README.nco + 590 module load e4s + 591 spack env activate gcc + 592 spack load nco + 593 module load cudatoolkit/11.5 + 594 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 595 history + 596 vi README.generate.bash + 597 vi create-v2.SMYLE.IC_RESTARTS.csh + 598 ./create-v2.SMYLE.IC_RESTARTS.csh + 599 echo $SHERLL + 600 echo $SHELL + 601 vi create-v2.SMYLE.IC_RESTARTS.csh + 602 vi setEnv.scr + 603 chmod u+x setEnv.scr + 604 ./setEnv.scr + 605 echo $CYLC_TASK_CYCLE_POINT + 606 setenv CYLC_TASK_CYCLE_POINT 1960-11-01 + 607 export CYLC_TASK_CYCLE_POINT 1960-11-01 + 608 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 609 echo $CYLC_TASK_CYCLE_POINT + 610 ./generate_cami_ensemble_offline.py + 611 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 612 less README.generate.bash + 613 module load e4s + 614 spack env activate gcc + 615 spack load nco + 616 module load cudatoolkit/11.5 + 617 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 618 ls -l /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/ + 619 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/1960-11-01/ + 620 vi generate_cami_ensemble_offline.py + 621 vi create-v2.SMYLE.IC_RESTARTS.csh + 622 ./create-v2.SMYLE.IC_RESTARTS.csh + 623 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 624 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 625 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/11/ + 626 vi generate_cami_ensemble_offline.py + 627 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 628 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 629 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 630 export CYLC_TASK_CYCLE_POINT=1959-11-01 + 631 ./generate_cami_ensemble_offline.py + 632 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.11/pert.01/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc + 633 vi generate_cami_ensemble_offline.py + 634 ./generate_cami_ensemble_offline.py + 635 history + 636 module load e4s + 637 spack env activate gcc + 638 spack load nco + 639 module load cudatoolkit/11.5 + 640 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 641 grep slat * + 642 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 643 vi generate_cami_ensemble_offline.py + 644 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 645 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 646 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 647 ./create-v2.SMYLE.IC_RESTARTS.csh + 648 export CYLC_TASK_CYCLE_POINT=1961-11-01 + 649 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 650 vi generate_cami_ensemble_offline.py + 651 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1959-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1959-11-01-00000.nc | less + 652 vi generate_cami_ensemble_offline.py + 653 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 654 vi ./create-v2.SMYLE.IC_RESTARTS.csh + 655 ./create-v2.SMYLE.IC_RESTARTS.csh + 656 export CYLC_TASK_CYCLE_POINT=1960-11-01 + 657 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 658 history + 659 vi generate_cami_ensemble_offline.py + 660 cp generate_cami_ensemble_offline.py doh.py + 661 vi doh.py + 662 fg + 663 vi doh.py + 664 vi generate_cami_ensemble_offline.py + 665 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 666 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.19/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc + 667 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.18/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 668 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.17/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 669 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.14/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 670 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.13/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 671 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.15/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 672 grep eamic * + 673 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 674 jobs + 675 vi generate_cami_ensemble_offline.py + 676 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 677 ./create-v2.SMYLE.IC_RESTARTS.csh + 678 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 679 vi generate_cami_ensemble_offline.py + 680 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 681 fg + 682 vi generate_cami_ensemble_offline.py + 683 PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + 684 ncdump -h /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e2sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/pert.16/v21.LR.SMYLE_IC.pert.ne30np4.eam.i.1960-11-01-00000.nc | less + 685 vi generate_cami_ensemble_offline.py + 686 ./generate_cami_ensemble_offline.py + 687 vi generate_cami_ensemble_offline.py + 688 ./generate_cami_ensemble_offline.py + 689 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.1960-11.01/ + 690 ls /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/ + 691 vi generate_cami_ensemble_offline.py + 692 vi create-v2.SMYLE.IC_RESTARTS.csh + 693 ./generate_cami_ensemble_offline.py + 694 ls -l /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.BSMYLE.ne30np4.1960-11.001/run.002/v21.LR.SMYLE_IC.ne30np4.1960-11.01.eam.i.1960-11-01-00000.nc + 695 vi create-v2.SMYLE.IC_RESTARTS.csh + 696 vi generate_cami_ensemble_offline.py + 697 vi create-v2.SMYLE.IC_RESTARTS.csh + 698 ./create-v2.SMYLE.IC_RESTARTS.csh + 699 export CYLC_TASK_CYCLE_POINT=1962-11-01 + 700 ./generate_cami_ensemble_offline.py + 701 vi create-v2.SMYLE.IC_RESTARTS.csh + 702 vi generate_cami_ensemble_offline.py + 703 ./generate_cami_ensemble_offline.py + 704 vi generate_cami_ensemble_offline.py + 705 ./generate_cami_ensemble_offline.py + 706 vi generate_cami_ensemble_offline.py + 707 ./generate_cami_ensemble_offline.py + 708 vi generate_cami_ensemble_offline.py + 709 ./generate_cami_ensemble_offline.py + 710 history + 711 vi README.generate.bash + 712 fg + 713 history + 714 history > doh diff --git a/scripts/v2.LR.LE-rerun/old/restarts/README.paths b/scripts/v2.LR.LE-rerun/old/restarts/README.paths new file mode 100644 index 0000000..1b51379 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/README.paths @@ -0,0 +1,22 @@ +atm: /global/cscratch1/sd/sglanvil/SMYLE_atm + +OCN restarts (2018, and 1958-1965; see README) - Luke Van Roekel: +Glade: /glade/p/datashare/yeager/CATALYST/ocean_ice_ic_JRA_cycle1/ +Cori: /global/cscratch1/sd/sglanvil/SMYLE_ocn +2018 = 0061 +1958 = 0062 +1959 = 0063 +1960 = 0064 +1961 = 0065 +1962 = 0066 +1963 = 0067 +1964 = 0068 +1965 = 0069 +(lets rename at some point) + +PERTURBATIONS (001-999): +Nan Rosenbloom AMIP daily-IC run 1991-2015: /global/cfs/cdirs/mp9/archive/E3SMv2-SMYLE/initial_conditions/CAMI/ +Sasha Glanville make random perts: /global/homes/s/sglanvil/S2S/make_random_diffs.sh +Saved random perts: /global/homes/s/sglanvil/S2S/save_perts +Final pert files:: /global/cscratch1/sd/sglanvil/S2S_perts_DIFF + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/README_perts_sasha b/scripts/v2.LR.LE-rerun/old/restarts/README_perts_sasha new file mode 120000 index 0000000..6d3845e --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/README_perts_sasha @@ -0,0 +1 @@ +/global/homes/s/sglanvil/S2S/README_perts \ No newline at end of file diff --git a/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE b/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE new file mode 100755 index 0000000..17cfb3b --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS-08.csh-AuFIRE @@ -0,0 +1,156 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) +set syr = 2019 +set eyr = 2019 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST + +# Nan needs to update +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS.csh b/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS.csh new file mode 100755 index 0000000..125ca2c --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/create-v2.SMYLE.IC_RESTARTS.csh @@ -0,0 +1,166 @@ +#! /bin/csh -fxv + +setenv TOOLS_ROOT /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/ +# module load e4s +# spack env activate gcc +# spack load nco +# module load cudatoolkit/11.5 + +## FOSI alignment +# 1958 = 0306 +# ... + +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpaso.rst.0306-11-01_00000.nc +# 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil.mpassi.rst.0306-11-01_00000.nc +# /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + + +#Remove xtime: +#mv v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpaso.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpaso.rst.0501-01-01_00000.nc +# +#mv v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc +#ncrename -v xtime,xtime.orig v2.LR.piControl.mpassi.rst.0501-01-01_00000.orig.nc v2.LR.piControl.mpassi.rst.0501-01-01_00000.nc + +#foreach year ( 1954 1964 1974 1984 1994 2004 ) + set syr = 1970 + set eyr = 1975 +# set syr = 1976 +# set eyr = 1980 +# set syr = 1981 +# set eyr = 2000 +#set syr = 2001 +#set eyr = 2001 + +@ ib = $syr +@ ie = $eyr + +foreach year ( `seq $ib $ie` ) +# foreach mon ( 11 ) +foreach mon ( 08 ) + +set case = v21.LR.SMYLE_IC.${year}-${mon}.01 + +set Picdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/ +set icdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 +if (! -d ${Picdir}) then + mkdir ${Picdir} +endif +if (! -d ${icdir}) then + mkdir ${icdir} +endif + + +set doThis99 = 1 +if ($doThis99 == 1) then + +# atm, lnd initial conditions +set atmcase = eami.HICCUP-ERA5-CATALYST +set lndcase = v21.LR.I20TRELM_CRUNCEP + +# names +set atmfname = ${atmcase}.${year}-${mon}-01.ne30np4.L72.c20230203.nc +set lndfname = ${lndcase}.elm.r.${year}-${mon}-01-00000.nc +set roffname = ${lndcase}.mosart.r.${year}-${mon}-01-00000.nc + +# directories +set atmdir = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/atm/ +#set lnddir = /pscratch/sd/s/sglanvil/archive/s2sLandRun_ICRUELM_final/rest/${year}-${mon}-01-00000/ +set lnddir = /pscratch/sd/n/nanr/v21.SMYLE/v21.LR.I20TRELM_CRUNCEP/archive/rest/${year}-${mon}-01-00000/ + +# rename atm, land IC files +set atmfout = ${case}.eam.i.${year}-${mon}-01-00000.nc +set lndfout = ${case}.elm.r.${year}-${mon}-01-00000.nc +set roffout = ${case}.mosart.r.${year}-${mon}-01-00000.nc + +echo $atmfout + +set doThis = 1 + +if ($doThis == 1) then +cp $atmdir/${atmfname} $icdir/$atmfout +cp $lnddir/${lndfname} $icdir/$lndfout +cp $lnddir/${roffname} $icdir/$roffout +ncatted -a OriginalFile,global,a,c,$atmfname $icdir/$atmfout +ncatted -a OriginalFile,global,a,c,$lndfname $icdir/$lndfout +ncatted -a OriginalFile,global,a,c,$roffname $icdir/$roffout + +endif + +# ocn/ice +# years used for ICs: 0306 (1958) - 0336 (1988) +#1958 = 0306 +#1959 = 0307 +# ... +#1988 = 0336 + +set doThis = 1 +if ($doThis == 1) then + +set ocncase = 20230123.GMPAS-JRA1p4.TL319_EC30to60E2r2.anvil +set first_rest_year = 1958 +set ocean_base_year = 306 + + +# Comment: year translation: if ($year == 2018 ) set ocnyr = 0366 +# years used for ICs: 0306 (1958) - 0366 (2018) +# atmyr 1958 = ocnyr 306 +@ offset = $first_rest_year - $ocean_base_year +@ ocnyr = $year - $offset +set ocndir = /pscratch/sd/l/lvroekel/cycle6_monthly-restarts/restarts_monthly/ + +set icefname = ${ocncase}.mpassi.rst.0${ocnyr}-${mon}-01_00000.nc +set poprfname = ${ocncase}.mpaso.rst.0${ocnyr}-${mon}-01_00000.nc + +set icefout = ${case}.mpassi.rst.${year}-${mon}-01_00000.nc +set poprfout = ${case}.mpaso.rst.${year}-${mon}-01_00000.nc + + +echo $icefname +echo $poprfname + +cp $ocndir/${icefname} $icdir/${icefout} +cp $ocndir/${poprfname} $icdir/${poprfout} + +# fix xtime offline: fix_xtime.sh (using bash) +# ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} +# rm $icdir/${poprfout}.xtime.nc + +ncatted -a OriginalFile,global,a,c,$icefname $icdir/$icefout +ncatted -a OriginalFile,global,a,c,$poprfname $icdir/$poprfout + +# create rpointer files +echo "$case.eam.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.atm +echo "$case.cpl.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.drv +echo "./$case.elm.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.lnd +echo "$case.mosart.r.$year-${mon}-01-00000.nc" > ${icdir}/rpointer.rof +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ice +echo "$year-${mon}-01_00:00:00" > ${icdir}/rpointer.ocn + +endif # doThis2 + +#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +endif # doThis99 + +# ================================== +# generate perturbed cam.i.restarts +# ================================== +#setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 +#./generate_cami_ensemble_offline.py + +# bash + #export CYLC_TASK_CYCLE_POINT=${year}-${mon}-01 + #setenv CYLC_TASK_CYCLE_POINT ${year}-${mon}-01 + #cd ${TOOLS_ROOT}/restarts/ + #PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py + +end +end + +exit + + + + + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/fix_ncol_d.sh b/scripts/v2.LR.LE-rerun/old/restarts/fix_ncol_d.sh new file mode 100755 index 0000000..039707c --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/fix_ncol_d.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=11 +mon=08 + +for year in {1970..2018}; do +for pert in {01..20}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + eamfile=v21.LR.SMYLE_IC.pert.eam.i.$year-${mon}-01-00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/pert.${pert} + echo $case + echo $icdir + ncrename -d ncol_d,ncol $icdir/${eamfile} +done +done + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/fix_xtime.sh b/scripts/v2.LR.LE-rerun/old/restarts/fix_xtime.sh new file mode 100755 index 0000000..5718596 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/fix_xtime.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=08 + +for year in {1970..2018}; do + case=v21.LR.SMYLE_IC.${year}-${mon}.01 + poprfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpaso.rst.$year-${mon}-01_00000.nc + icerfout=v21.LR.SMYLE_IC.${year}-${mon}.01.mpassi.rst.$year-${mon}-01_00000.nc + icdir=/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/${case}/${year}-${mon}-01 + echo $case + echo $icdir + mv $icdir/${poprfout} $icdir/${poprfout}.xtime.nc + mv $icdir/${icerfout} $icdir/${icerfout}.xtime.nc + ncrename -v xtime,xtime.orig $icdir/${poprfout}.xtime.nc $icdir/${poprfout} + ncrename -v xtime,xtime.orig $icdir/${icerfout}.xtime.nc $icdir/${icerfout} +done + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/generate_cami_ensemble_offline.py b/scripts/v2.LR.LE-rerun/old/restarts/generate_cami_ensemble_offline.py new file mode 100755 index 0000000..7b109b1 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/generate_cami_ensemble_offline.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +#PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools + +if e3smroot is None: + raise SystemExit("ERROR: CESM_ROOT must be defined in environment") + +# This is needed for globus_sdk +#_LIBDIR=os.path.join(os.environ.get("HOME"),".local","lib","python3.6","site-packages") +#sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, random, threading, time, shutil +from standard_script_setup import * +from CIME.utils import run_cmd, safe_copy, expect +from argparse import RawTextHelpFormatter +#from globus_utils import * + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.environ.get("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model,int(args.ensemble_start),int(args.ensemble_end) + +def get_rvals(date, ensemble_start,ensemble_end, model): + random.seed(int(date[0:4])+int(date[5:7])+int(date[8:10])) + rvals = random.sample(range(1001),k=ensemble_end//2) + print("Rvals are {}".format(rvals)) + #rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/initial_conditions/","cases","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + rvals_file = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","eamic_"+date+".{}-{}.txt".format(ensemble_start,ensemble_end)) + if not os.path.isfile(rvals_file): + with open(rvals_file,"w") as fd: + fd.write("{}".format(rvals)) + + return rvals + +def create_eam_ic_perturbed(original, ensemble_start,ensemble_end, date, baserundir, model, outroot="v21.LR.SMYLE_IC.pert.eam.i.", factor=0.15): + rvals = get_rvals(date, ensemble_start,ensemble_end, model) + + outfile = os.path.join(baserundir,outroot+date+"-00000.nc") + # first link the original ic file to the 0th ensemble member + if os.path.exists(outfile): + os.unlink(outfile) + expect(os.path.isfile(original),"ERROR file {} not found".format(original)) + #print("Linking {} to {}".format(original, outfile)) + #rundir = os.path.dirname(outfile) + #print("Linking {} to {}".format(original, os.path.join(rundir,os.path.basename(original)))) + #if os.path.isdir(rundir): + #shutil.rmtree(rundir) + #os.makedirs(rundir) + #os.symlink(original, outfile) + #os.symlink(original, os.path.join(rundir,os.path.basename(original))) + + # for each pair of ensemble members create an ic file with same perturbation opposite sign + month = date[5:7] + year = date[0:3] + + local_path = "/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF" + perturb_files = [] + for i in range(ensemble_start,ensemble_end, 2): + print ("HERE rvals[{}] = {}".format((i-1)//2,rvals[(i-1)//2])) + perturb_file = os.path.join("{}".format(month), + "v2.LR.historical_daily-cami_0241.eam.i.M{}.diff.{:03}.nc".format(month,rvals[(i-1)//2])) + dirname = os.path.dirname(os.path.join(local_path,perturb_file)) + if not os.path.isdir(dirname): + print("Creating directory {}".format(dirname)) + os.makedirs(dirname) + perturb_files.append(perturb_file) + + pertroot = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","pert.01") + + for i in range(ensemble_start,ensemble_end, 2): + pfile = os.path.join(local_path, perturb_files.pop(0)) + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-tmp.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-tmp.nc") + print("Creating perturbed init file {}".format(outfile1)) + print("Creating perturbed init file {}".format(outfile2)) + print("Using perturb_file {}".format(pfile)) + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile1, factor)) + t.start() + t = threading.Thread(target=create_perturbed_init_file,args=(original, pfile, outfile2, -1*factor)) + t.start() + while(threading.active_count() > 1): + time.sleep(1) + + if False: + for i in range(ensemble_start, ensemble_end, 2): + outfile1 = os.path.join(pertroot[:-2]+"{:02d}".format(i), outroot+date+"-00000.nc") + outfile2 = os.path.join(pertroot[:-2]+"{:02d}".format(i+1), outroot+date+"-00000.nc") + outdir1 = baserundir[:-3]+"{:03d}".format(i) + outdir2 = baserundir[:-3]+"{:03d}".format(i+1) + origfile = os.path.basename(original) + print("{} {} ".format(outfile1, os.path.join(outdir1,origfile))) + print("{} {} ".format(outfile2, os.path.join(outdir2,origfile))) + for outdir in (outdir1,outdir2): + if not os.path.isdir(outdir): + os.mkdir(outdir) + print("outdir = {} ".format(outdir)) + if i != 1: + if os.path.isfile(os.path.join(outdir1,origfile)): + os.unlink(os.path.join(outdir1,origfile)) + os.symlink(outfile1, os.path.join(outdir1,origfile)) + print("I made it here = {} ".format(outdir)) + if os.path.isfile(os.path.join(outdir2,origfile)): + os.unlink(os.path.join(outdir2,origfile)) + os.symlink(outfile2, os.path.join(outdir2,origfile)) + + +def create_perturbed_init_file(original, perturb_file, outfile, weight): + ncflint = "ncflint" + if not os.path.isdir(os.path.dirname(outfile)): + os.makedirs(os.path.dirname(outfile)) + pertfile = outfile.replace("-tmp.nc","-00000.nc") + if os.path.isfile(pertfile): + print("Found existing file {}".format(pertfile)) + return # file exists nothing more to do + safe_copy(original, outfile) + if "BWHIST" in original: + cmd = ncflint + " -A -v US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + else: + #cmd = ncflint+" -O -C -v lat,lon,slat,slon,lev,ilev,hyai,hybi,hyam,hybm,US,VS,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + cmd = ncflint+" -O -C -v time_bnds,lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w {},1.0 {} {} {}".format(weight, perturb_file, original, outfile) + run_cmd(cmd, verbose=True) + if os.path.isfile(outfile): + os.rename(outfile, outfile.replace("-tmp.nc","-00000.nc")) + else: + print("Rename of {} failed".format(outfile)) + +def _main_func(description): + date, model,ensemble_start,ensemble_end = parse_command_line(sys.argv, description) + + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC."+date[0:7]+".01","{}".format(date)) + user = os.getenv("USER") + baserundir = os.path.join("/pscratch/sd/n/{}/".format(user),"v21.SMYLE","v21.LR.BSMYLE."+date[0:7]+".001","run.{:03d}".format(ensemble_start)) + eaminame = os.path.join(sdrestdir,"v21.LR.SMYLE_IC.{}.01.eam.i.{date}-00000.nc".format(date[:7],date=date)) + outroot = "v21.LR.SMYLE_IC.pert.eam.i." + + create_eam_ic_perturbed(eaminame,ensemble_start,ensemble_end, date,baserundir, model, outroot=outroot) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/v2.LR.LE-rerun/old/restarts/run_generate.sh b/scripts/v2.LR.LE-rerun/old/restarts/run_generate.sh new file mode 100755 index 0000000..567097e --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/run_generate.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +#module load e4s +#spack env activate gcc +#spack load nco +#module load cudatoolkit/11.5 +#module load python + +mon=08 +for year in {1970..1975}; do + export CYLC_TASK_CYCLE_POINT=$year-${mon}-01 + PYTHONPATH=/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/cime/CIME/Tools ./generate_cami_ensemble_offline.py +done + diff --git a/scripts/v2.LR.LE-rerun/old/restarts/setEnv.scr b/scripts/v2.LR.LE-rerun/old/restarts/setEnv.scr new file mode 100755 index 0000000..3e3284a --- /dev/null +++ b/scripts/v2.LR.LE-rerun/old/restarts/setEnv.scr @@ -0,0 +1,4 @@ +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 diff --git a/scripts/v2.LR.LE-rerun/scripts/run.v2.LR.E3SM-xtraOutput.sh b/scripts/v2.LR.LE-rerun/scripts/run.v2.LR.E3SM-xtraOutput.sh new file mode 100755 index 0000000..f9bf6df --- /dev/null +++ b/scripts/v2.LR.LE-rerun/scripts/run.v2.LR.E3SM-xtraOutput.sh @@ -0,0 +1,519 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0121 0151 0191 0211 0251 ) +#syr=( 1976 1975 1976 1976 1975 ) +array=( 0121 ) +syr=( 1976 ) +for iyr in "${array[@]}" +do + +main() { +doyear=${syr[@]} +echo ${iyr} +echo ${doyear} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical_xtraOutput_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="$doyear-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.historical_${iyr}/archive/rest/$doyear-01-01-00000/" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${doyear}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/E3SMv2-Rerun/${CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_BUILD_DIR="/pscratch/sd/n/nanr/E3SMv2-Rerun/exeroot/build" +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="24" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="24" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/E3SMv2-Rerun/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${iyr} -eq "0121" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','I','A','I' + ! monthly (h0) A + ! daily (h1) A + fincl2 = 'CAPE','CLDTOT','FLDSC','FLNR','FLNS','FLNT','FLUT','FSNR','FSNS','FSNT','LHFLX','SWCF','LWCF','SHFLX','PBLH','PHIS','PRECC','PRECL','PRECZ','PS','SST','TAUBLJX','TAUBLJY','TAUGWX','TAUGWY','TAUX','TAUY','TMQ','PCONVT','TROP_P','TROP_T','TROP_Z','PRECT','FLNSC','FLNTC','FLUTC','FSDSC','FSNSC','FSNTC','FSNTOAC','U','V','OMEGA','Q','T','Z3','ZMDT','ZMDQ','DTCOND','DCQ','CLDICE','CLDLIQ','CLOUD','RELHUM' + ! 6hourly (h2) I + fincl3 = 'IVT','UIVT','VIVT','TMQ','U850','V850','Z200','Z500','U200','V200' + ! 6hourly (h3) A + ! 3hourly (h4) I + fincl5 = 'PRECC','PRECL','PRECZ' + ! monthly (h6) I + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +#patch_mpas_streams() { + +#echo + +#} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + #./xmlchange EXEROOT="/pscratch/sd/n/nanr/E3SMv2-Rerun/exeroot/build" + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Lastly - copy Initial conditions + ics="/pscratch/sd/n/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${doyear}-01-01-00000/" + + # Finally, run CIME case.setup + ./case.setup --reset + + # pre-stage ICs + mkdir ${CASE_RUN_DIR} + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + cp ${ics}/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/v2.LR* ${CASE_RUN_DIR}/ + + # fix the modules + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.LR.LE-rerun/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N},STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam new file mode 100644 index 0000000..b3dd577 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam @@ -0,0 +1,17 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS','FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z','CAPE','CLDTOT','FLDSC','FLNR','FLNS','FLNT','FLUT','FSNR','FSNS','FSNT','LHFLX','SWCF','LWCF','SHFLX','PBLH','PHIS','PRECC','PRECL','PRECZ','PS','SST','TAUBLJX','TAUBLJY','TAUGWX','TAUGWY','TAUX','TAUY','TMQ','PCONVT','TROP_P','TROP_T','TROP_Z','FLNSC','FLNTC','FLUTC','FSDSC','FSNSC','FSNTC','FSNTOAC', 'U','V','OMEGA','Q','T','Z3','ZMDT','ZMDQ','DTCOND','DCQ','CLDICE','CLDLIQ','CLOUD','RELHUM' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I', 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I', 'IVT:I','UIVT:I','VIVT:I','TMQ:I','U850:I','V850:I','Z200:I','Z500','U200','V200' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT:I','PRECC:I','TUQ','TVQ','QFLX','SHFLX','U90M','V90M','PRECZ:I' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + diff --git a/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam-withCosp b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam-withCosp new file mode 100644 index 0000000..4327697 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_eam-withCosp @@ -0,0 +1,20 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS','FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z', 'SST' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I', 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + diff --git a/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_elm b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_elm new file mode 100644 index 0000000..f1fa0b2 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_elm @@ -0,0 +1,9 @@ + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_year_consistency = .true. + diff --git a/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_mosart b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_mosart new file mode 100644 index 0000000..ab66396 --- /dev/null +++ b/scripts/v2.LR.LE-rerun/user_nl_files/user_nl_mosart @@ -0,0 +1,5 @@ + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 + diff --git a/scripts/v2.LR.SF-LE/README.levels b/scripts/v2.LR.SF-LE/README.levels new file mode 100644 index 0000000..9b22167 --- /dev/null +++ b/scripts/v2.LR.SF-LE/README.levels @@ -0,0 +1,46 @@ +bash +module load cray-hdf5 +module load cray-netcdf +module load e4s +spack env activate gcc +spack load nco +module load cudatoolkit/11.5 +module load python + + +ncks -v lat_vertices,lon_vertices,time_bnds v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000.nc eam.vars.nc + + double time_bnds(time, nbnd) ; + time_bnds:_FillValue = NaN ; + time_bnds:long_name = "time interval endpoints" ; + double lat_vertices(ncol, nv) ; + lat_vertices:_FillValue = NaN ; + lat_vertices:long_name = "Gridcell latitude vertices" ; + lat_vertices:coordinates = "lon lat" ; + double lon_vertices(ncol, nv) ; + lon_vertices:_FillValue = NaN ; + lon_vertices:long_name = "Gridcell longitude vertices" ; + lon_vertices:coordinates = "lon lat" ; + +Original: +netcdf v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000 { +dimensions: + time = 1 ; + ncol = 48602 ; + lev = 72 ; + ilev = 73 ; + nv = 1 ; + nbnd = 2 ; + +Perturbed +netcdf v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000 { +dimensions: + time = UNLIMITED ; // (1 currently) + ncol_d = 48602 ; + lev = 72 ; + ilev = 73 ; + + :history = "Thu Jul 13 10:44:31 2023: ncflint -O -C -v lev,ilev,hyai,hybi,hyam,hybm,U,V,T,Q,PS -w 0.15,1.0 /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/S2S_perts_DIFF/11/v2.LR.historical_daily-cami_0241.eam.i.M11.diff.762.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1971-11.01/1971-11-01/v21.LR.SMYLE_IC.1971-11.01.eam.i.1971-11-01-00000.nc /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1971-11.01/pert.03/v21.LR.SMYLE_IC.pert.eam.i.1971-11-01-tmp.nc\n", + "Wed May 24 11:27:41 2023: ncdiff /global/cfs/cdirs/mp9/SMYLE/S2S_perts/11//v2.LR.historical_daily-cami_0241.eam.i.1992-11-11-00000.nc /global/cfs/cdirs/mp9/SMYLE/S2S_perts/11//v2.LR.historical_daily-cami_0241.eam.i.2006-11-07-00000.nc /global/cfs/cdirs/mp9/SMYLE/S2S_perts_DIFF/11//v2.LR.historical_daily-cami_0241.eam.i.M11.diff.762.nc\n", + "created on 02/19/23 15:32:29" ; + diff --git a/scripts/v2.LR.SF-LE/env_mach/env_mach_specific.xml b/scripts/v2.LR.SF-LE/env_mach/env_mach_specific.xml new file mode 100644 index 0000000..32320b7 --- /dev/null +++ b/scripts/v2.LR.SF-LE/env_mach/env_mach_specific.xml @@ -0,0 +1,87 @@ + + +
+ These variables control the machine dependent environment including + the paths to compilers and libraries external to cime such as netcdf, + environment variables for use in the running job should also be set here. +
+ + + char + executable name + + + char + redirect for job output + + + + /usr/share/lmod/8.3.1/init/perl + /usr/share/lmod/8.3.1/init/python + /usr/share/lmod/8.3.1/init/sh + /usr/share/lmod/8.3.1/init/csh + /usr/share/lmod/lmod/libexec/lmod perl + /usr/share/lmod/lmod/libexec/lmod python + module + module + + cray-hdf5-parallel + cray-netcdf-hdf5parallel + cray-parallel-netcdf + PrgEnv-gnu + PrgEnv-nvidia + cudatoolkit + craype-accel-nvidia80 + craype-accel-host + perftools-base + perftools + darshan + + + PrgEnv-gnu/8.3.3 + gcc/11.2.0 + + + PrgEnv-nvidia + nvidia/22.5 + + + PrgEnv-aocc + aocc/3.2.0 + + + craype-accel-host + cray-libsci + craype + cray-mpich/8.1.27 + cray-hdf5-parallel/1.12.2.7 + cray-netcdf-hdf5parallel/4.9.0.7 + cray-parallel-netcdf/1.12.3.7 + cmake/3.22.0 + + + + 1 + 1 + 128M + spread + threads + FALSE + /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch + software + MPI_Bcast + + + -1 + + + srun + + --label + -n {{ total_tasks }} -N {{ num_nodes }} + -c $SHELL{echo 256/`./xmlquery --value MAX_MPITASKS_PER_NODE`|bc} + $SHELL{if [ 128 -ge `./xmlquery --value MAX_MPITASKS_PER_NODE` ]; then echo "--cpu_bind=cores"; else echo "--cpu_bind=threads";fi;} + -m plane=$SHELL{echo `./xmlquery --value MAX_MPITASKS_PER_NODE`} + + + diff --git a/scripts/v2.LR.SF-LE/scripts/README.ENDWD_COL b/scripts/v2.LR.SF-LE/scripts/README.ENDWD_COL new file mode 100644 index 0000000..4838a11 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/README.ENDWD_COL @@ -0,0 +1,31 @@ +Jon Wolf: +I'm looking at what they did to "fix" the default restart file and it seems like they just stuck on an array with no values in it. This is what I get from the fixed file: + +ncdump -h v2.LR.historical_0101.elm.r.2015-01-01-00000.nc | grep -i endwb +double ENDWB_COL(column) ; +ENDWB_COL:long_name = "col-level water mass end of the time step" ; +ENDWB_COL:units = "mm" ; +ENDWB_COL:interpinic_flag = 1 ; +ENDWB_COL:interpinic_flag_meanings = "1=nearest neighbor, 2=copy directly, 3=skip" ; +ENDWB_COL:_FillValue = 1.e+36 ; +ENDWB_COL:missing_value = 1.e+36 ; + +but looking at it with ncview says "All values are missing!" + +--------------- +srcuni + +# Make a safe copy +cp v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.ORIG.nc + +# Extract variable to use as template fpr ENDWB_COL +ncks -v URBAN_HEAT v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc URBAN_HEAT.nc +ncrename -v URBAN_HEAT,ENDWB_COL URBAN_HEAT.nc +cp URBAN_HEAT.nc input.nc + +# Set ENDWB_COL to 0.0 +ncflint -O -v ENDWB_COL -w 0,0 input.nc input.nc output.nc +mv output.nc ENDWB_COL.nc + +# Append ENDWB_COL to restart file +ncks -A ENDWB_COL.nc v2.LR.hist-GHG_0141.elm.r.1904-01-01-00000.nc diff --git a/scripts/v2.LR.SF-LE/scripts/README.aer b/scripts/v2.LR.SF-LE/scripts/README.aer new file mode 100644 index 0000000..98a3996 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/README.aer @@ -0,0 +1,62 @@ +https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/3052995085/v2.LR.hist-aer + +(1) GHGs settings: also set CCSM_CO2_PPMV +./xmlchange --id CCSM_CO2_PPMV --val "284.317" + +user_nl_eam: + +nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + + bndtvghg = ' ' + ch4vmr = 808.249e-9 + co2vmr = 284.317000e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' + +! (2) aeorosols and precursors + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/lcrc/group/e3sm/data/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + diff --git a/scripts/v2.LR.SF-LE/scripts/branch.v2.LR.GHG-SF_0161.sh b/scripts/v2.LR.SF-LE/scripts/branch.v2.LR.GHG-SF_0161.sh new file mode 100755 index 0000000..1cfe6c6 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/branch.v2.LR.GHG-SF_0161.sh @@ -0,0 +1,588 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +#array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +#array=( 0141 ) +array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-GHG_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" +START_DATE="1891-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/archive/rest/${REFYEAR}-01-01-00000" +RUN_REFDIR="/pscratch/sd/n/nanr/v2.LR.GHG-SF/v2.LR.hist-GHG_0161-old/archive/rest/${START_DATE}-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFCASE="v2.LR.piControl" +#RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' +RUN_REFCASE="v2.LR.hist-GHG_0161" +RUN_REFDATE="${START_DATE}" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CODE_ROOT="/global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.0/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.GHG-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + + ext_frc_cycle_yr = 1850 + ext_frc_type = 'CYCLICAL' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/cp_restarts-plus-JOB_QUEUE.csh b/scripts/v2.LR.SF-LE/scripts/cp_restarts-plus-JOB_QUEUE.csh new file mode 100755 index 0000000..2129ab6 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/cp_restarts-plus-JOB_QUEUE.csh @@ -0,0 +1,30 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +#foreach mbr ( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +foreach mbr ( 0101 0151 0201 0251 0301 ) +#foreach mbr ( 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/run +#set REFROOT = /global/cscratch1/sd/nanr/archive/v2.LR.historical_${mbr}/archive/rest/1991-01-01-00000/ +set REFROOT = /global/cscratch1/sd/nanr/archive/v2.LR.historical_${mbr}/archive/rest/1990-01-01-00000/ + +cd $CASEROOT/ +./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/ +./xmlchange JOB_QUEUE=debug --subgroup case.st_archive +./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive + +mv case.st_archive tmp.st_archive +cat tmp.st_archive | sed 's/101/1/' > case.st_archive + +cd $RUNROOT/ +cp $REFROOT/rpointer* . +ln -s $REFROOT/v2.*.nc . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SF-LE/scripts/ddd b/scripts/v2.LR.SF-LE/scripts/ddd new file mode 100644 index 0000000..d647fb4 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/ddd @@ -0,0 +1,10 @@ + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.classic.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.classic.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.classic.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.classic.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' diff --git a/scripts/v2.LR.SF-LE/scripts/ddd2 b/scripts/v2.LR.SF-LE/scripts/ddd2 new file mode 100644 index 0000000..005216d --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/ddd2 @@ -0,0 +1,10 @@ + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' diff --git a/scripts/v2.LR.SF-LE/scripts/env_mach/env_mach_specific.xml b/scripts/v2.LR.SF-LE/scripts/env_mach/env_mach_specific.xml new file mode 100644 index 0000000..32320b7 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/env_mach/env_mach_specific.xml @@ -0,0 +1,87 @@ + + +
+ These variables control the machine dependent environment including + the paths to compilers and libraries external to cime such as netcdf, + environment variables for use in the running job should also be set here. +
+ + + char + executable name + + + char + redirect for job output + + + + /usr/share/lmod/8.3.1/init/perl + /usr/share/lmod/8.3.1/init/python + /usr/share/lmod/8.3.1/init/sh + /usr/share/lmod/8.3.1/init/csh + /usr/share/lmod/lmod/libexec/lmod perl + /usr/share/lmod/lmod/libexec/lmod python + module + module + + cray-hdf5-parallel + cray-netcdf-hdf5parallel + cray-parallel-netcdf + PrgEnv-gnu + PrgEnv-nvidia + cudatoolkit + craype-accel-nvidia80 + craype-accel-host + perftools-base + perftools + darshan + + + PrgEnv-gnu/8.3.3 + gcc/11.2.0 + + + PrgEnv-nvidia + nvidia/22.5 + + + PrgEnv-aocc + aocc/3.2.0 + + + craype-accel-host + cray-libsci + craype + cray-mpich/8.1.27 + cray-hdf5-parallel/1.12.2.7 + cray-netcdf-hdf5parallel/4.9.0.7 + cray-parallel-netcdf/1.12.3.7 + cmake/3.22.0 + + + + 1 + 1 + 128M + spread + threads + FALSE + /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch + software + MPI_Bcast + + + -1 + + + srun + + --label + -n {{ total_tasks }} -N {{ num_nodes }} + -c $SHELL{echo 256/`./xmlquery --value MAX_MPITASKS_PER_NODE`|bc} + $SHELL{if [ 128 -ge `./xmlquery --value MAX_MPITASKS_PER_NODE` ]; then echo "--cpu_bind=cores"; else echo "--cpu_bind=threads";fi;} + -m plane=$SHELL{echo `./xmlquery --value MAX_MPITASKS_PER_NODE`} + + + diff --git a/scripts/v2.LR.SF-LE/scripts/findEm b/scripts/v2.LR.SF-LE/scripts/findEm new file mode 100755 index 0000000..813c147 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/findEm @@ -0,0 +1,10 @@ +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_BMB-SF.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_BMB-SF.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-MBM-SF/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_BMB-SF.nc +ls -l /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc +ls -l /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_BMB-SF.nc +ls -l /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc + diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh new file mode 100755 index 0000000..4862b4d --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh @@ -0,0 +1,594 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +#array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +array=( 0121 0131 0141 ) +#array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-BMB_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.BMB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + +!! cycling non-BB aerosols @ 1850 values; transient BB + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/ + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + #./xmlchange JOB_QUEUE=debug + #./xmlchange JOB_WALLCLOCK_TIME=00:30:00 + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-bad b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-bad new file mode 100755 index 0000000..0cb71fc --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-bad @@ -0,0 +1,591 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +array=( 0111 0121 0131 0171 ) +#array=( 0131 ) +#array=( 0141 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-BMB_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.BMB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying +! Nanr +! | hist-BMB | 1850 | varying (BMB)| 1850 | 1850 | 1850 | 1850 + +! (1) GHGs settings + +! (2) aeorosols and precursors + + +!! cycling non-BB aerosols @ 1850 values; transient BB + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + +!! Cycling over surface emmissions; no BB aerosols. + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-old b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-old new file mode 100755 index 0000000..177e206 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-old @@ -0,0 +1,531 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +array=( 0111 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +#COMPSET="WCYCL20TR" # 20th century transient +COMPSET="WCYCL1850" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.BMB-SF_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.BMB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_BMB-SF.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_BMB-SF.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-MBM-SF/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_BMB-SF.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed-BMB-SF/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_BMB-SF.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-safe b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-safe new file mode 100755 index 0000000..89b7306 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.BMB-SF_YYYY.sh-safe @@ -0,0 +1,594 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +#array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +array=( 0121 0131 0141 ) +#array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-BMB_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.BMB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + +!! cycling non-BB aerosols @ 1850 values; transient BB + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/ + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + #./xmlchange JOB_QUEUE=debug + #./xmlchange JOB_WALLCLOCK_TIME=00:30:00 + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh new file mode 100755 index 0000000..f4d9bde --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh @@ -0,0 +1,583 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +#array=( 0141 ) +#array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-GHG_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.GHG-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + + ext_frc_cycle_yr = 1850 + ext_frc_type = 'CYCLICAL' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh-safe b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh-safe new file mode 100755 index 0000000..4faec4b --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.GHG-SF_YYYY.sh-safe @@ -0,0 +1,563 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 ) +refarray=( 1850 1850 1850 1850 1850 1850 1850 1850 1850 1850 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${iyr} +echo ${refyear} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +#COMPSET="WCYCL20TR" # 20th century transient +COMPSET="WCYCL1850" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-GHG_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.piControl/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.piControl_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/E3SMv2-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/E3SMv2-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam +nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + + ext_frc_cycle_yr = 1850 + ext_frc_type = 'CYCLICAL' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/lcrc/group/e3sm/data/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.PIcntl.sh b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.PIcntl.sh new file mode 100755 index 0000000..08f0ca3 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.PIcntl.sh @@ -0,0 +1,580 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +array=( 0111 ) +#array=( 0121 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL1850" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.PIcntl_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.GHG-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + + ext_frc_cycle_yr = 1850 + ext_frc_type = 'CYCLICAL' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /lcrc/group/e3sm/data/ +inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + # user_nl + + # Finally, run CIME case.setup + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v2.LR.SMBB-SF_YYYY.sh b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.SMBB-SF_YYYY.sh new file mode 100755 index 0000000..6b9d755 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v2.LR.SMBB-SF_YYYY.sh @@ -0,0 +1,604 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +#array=( 0171 ) +#array=( 0121 ) +#array=( 0141 ) +#array=( 0121 0131 0141 ) +array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-SMBB_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.SMBB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.SMBB-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + + bndtvghg = ' ' + ch4vmr = 808.249e-9 + co2vmr = 284.317000e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' + + +! (2) aeorosols and precursors + +!! cycling non-BB aerosols @ 1850 values; transient BB + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.SMBB-SF/EXEROOT/build/ + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + #./xmlchange JOB_QUEUE=debug + #./xmlchange JOB_WALLCLOCK_TIME=00:30:00 + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/run.v21.LR.SF-LE-GHG.sh-fromSMYLE b/scripts/v2.LR.SF-LE/scripts/run.v21.LR.SF-LE-GHG.sh-fromSMYLE new file mode 100755 index 0000000..ca60cba --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/run.v21.LR.SF-LE-GHG.sh-fromSMYLE @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1850 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +#CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/test-build.sh b/scripts/v2.LR.SF-LE/scripts/test-build.sh new file mode 100755 index 0000000..94617ba --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/test-build.sh @@ -0,0 +1,595 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +#array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +#array=( 0141 ) +array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.hist-BMB_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.BMB-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/e3sm.exe" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + +!! cycling non-BB aerosols @ 1850 values; transient BB + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016_1850const.20250206.nc', + 'SOAG -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016_1850const.20250207.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/mp9/E3SMv2-SF/BMB-forcing/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016_1850const.20250206.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.BMB-SF/EXEROOT/build/ + #./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + ./xmlchange JOB_QUEUE=debug + ./xmlchange JOB_WALLCLOCK_TIME=00:30:00 + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/test.sh b/scripts/v2.LR.SF-LE/scripts/test.sh new file mode 100755 index 0000000..a748d80 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/test.sh @@ -0,0 +1,583 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +# array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0111 ) +array=( 0171 ) +#array=( 0121 ) +#array=( 0131 ) +#array=( 0141 ) +#array=( 0161 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.testhist-GHG_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/archive/rest/${REFYEAR}-01-01-00000" +#RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/${REFYEAR}-01-01-00000" +RUN_REFCASE="v2.LR.piControl" +RUN_REFDATE="${REFYEAR}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +MY_PATH="/global/cfs/cdirs/mp9" +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr/" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/pscratch/sd/n/nanr/v2.LR.GHG-SF/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + #PELAYOUT="L" + WALLTIME="24:00:00" + STOP_OPTION="nyears" + STOP_N="5" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="10" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +! Historical, vs single forcing configurations + +! | Configuration | GHGs | Aerosols and | Ozone | Solar | Volcanoes | Land use +! | | | precursors | | | | +! ----------------------------------------------------------------------------------------------- +! | historical | varying | varying | varying | varying | varying | varying +! | hist-GHG | varying | 1850 | 1850 | 1850 | 1850 | 1850 +! | hist-aer | 1850 | varying | 1850 | 1850 | 1850 | 1850 +! | hist-all-xGHG-xaer | 1850 | 1850 | varying | varying | varying | varying + +! (1) GHGs settings + +! (2) aeorosols and precursors + + ext_frc_cycle_yr = 1850 + ext_frc_type = 'CYCLICAL' + + srf_emis_cycle_yr = 1850 + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160416.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'CYCLICAL' + + tracer_cnst_cycle_yr = 1849 + tracer_cnst_type = 'CYCLICAL' + +! (3) ozone + + chlorine_loading_fixed_ymd = 18500101 + chlorine_loading_type = 'FIXED' + + linoz_data_cycle_yr = 1850 + linoz_data_type = 'CYCLICAL' + +! (4) solar + + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850control_input4MIPS_c20181106.nc' + solar_data_type = 'FIXED' + solar_data_ymd = 18500101 + solar_htng_spctrl_scl = .true. + + +! (5) volcanoes + + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_type = 'CYCLICAL' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! (6) Land use and cover + + do_transient_pfts = .false. + flanduse_timeseries = '' + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/pscratch/sd/n/nanr/v2.LR.GHG-SF/EXEROOT/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + ./case.setup --reset + + #cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SF-LE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + #exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/1982.out b/scripts/v2.LR.SF-LE/scripts/toss/1982.out new file mode 100644 index 0000000..feb1661 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/1982.out @@ -0,0 +1,4679 @@ +1 +001 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.001 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:56:16 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:17 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:17 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:17 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:56:18 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:56:18 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:56:18 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:56:18 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:56:18 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:56:18 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.001 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:56:20 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:20 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:21 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:21 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:56:21 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:56:21 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:56:21 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:56:21 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:56:21 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:56:21 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.001/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175625' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:56:29 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:29 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:29 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:31 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:56:33 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:56:33 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:56:33 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:56:33 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:56:33 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:56:33 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.001/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225364 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.001/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225364 case.st_archive --resubmit +Submitted job id is 17225365 +Submitted job case.run with id 17225364 +Submitted job case.st_archive with id 17225365 + +----- All done ----- + +2 +002 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.002 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:56:46 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:46 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:47 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:47 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:56:47 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:56:47 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:56:47 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:56:47 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:56:47 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:56:47 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.002 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +02 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:56:49 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:50 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:50 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:50 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:56:50 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:56:50 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:56:50 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:56:50 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:56:50 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:56:50 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.002/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175654' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:56:58 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:56:58 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:56:58 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:56:59 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:57:02 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:57:02 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:57:02 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:57:02 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:57:02 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:57:02 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.002/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225367 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.002/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225367 case.st_archive --resubmit +Submitted job id is 17225369 +Submitted job case.run with id 17225367 +Submitted job case.st_archive with id 17225369 + +----- All done ----- + +3 +003 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.003 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:57:28 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:57:28 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:57:28 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:57:28 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:57:29 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:57:29 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:57:29 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:57:29 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:57:29 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:57:29 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.003 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +03 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:57:31 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:57:31 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:57:31 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:57:31 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:57:32 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:57:32 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:57:32 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:57:32 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:57:32 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:57:32 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.003/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175736' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:57:39 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:57:40 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:57:40 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:57:41 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:57:43 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:57:44 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:57:44 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:57:44 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:57:44 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:57:44 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.003/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225374 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.003/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225374 case.st_archive --resubmit +Submitted job id is 17225375 +Submitted job case.run with id 17225374 +Submitted job case.st_archive with id 17225375 + +----- All done ----- + +4 +004 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.004 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:57:57 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:57:57 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:57:57 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:57:57 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:57:58 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:57:58 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:57:58 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:57:58 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:57:58 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:57:58 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.004 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +04 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:58:00 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:58:00 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:58:00 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:58:01 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:58:01 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:58:01 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:58:01 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:58:01 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:58:01 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:58:01 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.004/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175805' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:58:09 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:58:09 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:58:09 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:58:10 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:58:13 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:58:13 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:58:13 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:58:13 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:58:13 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:58:13 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.004/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225377 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.004/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225377 case.st_archive --resubmit +Submitted job id is 17225379 +Submitted job case.run with id 17225377 +Submitted job case.st_archive with id 17225379 + +----- All done ----- + +5 +005 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.005 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:58:32 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:58:32 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:58:33 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:58:33 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:58:33 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:58:33 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:58:33 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:58:33 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:58:33 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:58:33 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.005 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +05 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:58:35 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:58:35 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:58:36 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:58:36 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:58:36 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:58:36 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:58:36 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:58:36 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:58:36 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:58:36 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.005/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175840' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:58:44 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:58:44 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:58:44 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:58:45 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:58:48 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:58:48 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:58:48 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:58:48 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:58:48 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:58:48 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.005/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225385 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.005/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225385 case.st_archive --resubmit +Submitted job id is 17225386 +Submitted job case.run with id 17225385 +Submitted job case.st_archive with id 17225386 + +----- All done ----- + +6 +006 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.006 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:59:01 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:01 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:02 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:02 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:02 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:02 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:02 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:02 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:02 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:02 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.006 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +06 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:59:04 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:04 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:05 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:05 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:05 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:05 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:05 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:05 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:05 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:05 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.006/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175909' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:59:13 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:13 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:13 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:14 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:17 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:17 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:17 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:17 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:17 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:17 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.006/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225389 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.006/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225389 case.st_archive --resubmit +Submitted job id is 17225390 +Submitted job case.run with id 17225389 +Submitted job case.st_archive with id 17225390 + +----- All done ----- + +7 +007 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.007 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 17:59:28 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:28 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:28 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:29 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:29 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:29 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:29 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:29 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:29 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:29 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.007 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +07 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 17:59:31 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:31 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:32 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:32 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:32 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:32 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:32 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:32 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:32 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:32 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.007/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-175936' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 17:59:40 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 17:59:40 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 17:59:40 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 17:59:41 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 17:59:44 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 17:59:44 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 17:59:44 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 17:59:44 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 17:59:44 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 17:59:44 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.007/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225393 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.007/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225393 case.st_archive --resubmit +Submitted job id is 17225395 +Submitted job case.run with id 17225393 +Submitted job case.st_archive with id 17225395 + +----- All done ----- + +8 +008 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.008 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:00:02 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:02 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:03 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:03 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:03 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:03 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:03 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:03 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:03 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:03 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.008 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +08 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:00:05 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:06 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:06 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:06 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:06 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:06 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:06 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:06 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:06 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:06 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.008/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180011' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:00:15 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:15 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:15 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:17 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:19 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:20 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:20 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:20 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:20 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:20 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.008/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225399 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.008/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225399 case.st_archive --resubmit +Submitted job id is 17225400 +Submitted job case.run with id 17225399 +Submitted job case.st_archive with id 17225400 + +----- All done ----- + +9 +009 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.009 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:00:33 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:34 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:34 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:34 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:34 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:34 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:34 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:34 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:34 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:34 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.009 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +09 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:00:37 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:37 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:37 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:37 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:37 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:37 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:37 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:37 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:37 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:37 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.009/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180042' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:00:45 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:00:45 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:00:45 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:00:47 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:00:49 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:00:49 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:00:49 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:00:49 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:00:49 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:00:49 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.009/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225401 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.009/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225401 case.st_archive --resubmit +Submitted job id is 17225402 +Submitted job case.run with id 17225401 +Submitted job case.st_archive with id 17225402 + +----- All done ----- + +10 +010 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.010 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:01:07 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:08 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:08 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:08 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:08 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:08 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:08 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:08 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:08 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:08 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.010 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +10 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:01:10 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:11 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:11 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:11 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:11 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:11 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:11 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:11 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:11 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:11 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.010/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180116' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:01:19 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:19 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:19 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:21 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:23 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:23 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:23 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:23 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:23 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:23 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.010/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225404 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.010/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225404 case.st_archive --resubmit +Submitted job id is 17225405 +Submitted job case.run with id 17225404 +Submitted job case.st_archive with id 17225405 + +----- All done ----- + +11 +011 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.011 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:01:38 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:39 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:39 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:39 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:39 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:39 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:39 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:39 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:39 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:39 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.011 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +11 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:01:42 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:42 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:42 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:42 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:42 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:43 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:43 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:43 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:43 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:43 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.011/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180147' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:01:50 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:01:50 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:01:50 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:01:52 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:01:54 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:01:55 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:01:55 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:01:55 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:01:55 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:01:55 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.011/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225413 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.011/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225413 case.st_archive --resubmit +Submitted job id is 17225415 +Submitted job case.run with id 17225413 +Submitted job case.st_archive with id 17225415 + +----- All done ----- + +12 +012 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.012 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:02:11 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:02:11 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:02:12 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:02:12 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:02:12 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:02:12 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:02:12 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:02:12 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:02:12 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:02:12 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.012 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +12 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:02:15 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:02:15 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:02:15 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:02:16 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:02:16 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:02:16 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:02:16 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:02:16 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:02:16 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:02:16 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.012/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180221' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:02:25 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:02:25 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:02:26 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:02:27 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:02:29 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:02:29 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:02:29 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:02:29 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:02:29 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:02:29 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.012/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225426 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.012/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225426 case.st_archive --resubmit +Submitted job id is 17225430 +Submitted job case.run with id 17225426 +Submitted job case.st_archive with id 17225430 + +----- All done ----- + +13 +013 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.013 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:02:51 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:02:51 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:02:51 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:02:51 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:02:52 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:02:52 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:02:52 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:02:52 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:02:52 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:02:52 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.013 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +13 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:02:54 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:02:54 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:02:54 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:02:55 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:02:55 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:02:55 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:02:55 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:02:55 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:02:55 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:02:55 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.013/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180259' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:03:02 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:02 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:03 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:04 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:06 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:06 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:06 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:06 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:06 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:06 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.013/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225446 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.013/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225446 case.st_archive --resubmit +Submitted job id is 17225447 +Submitted job case.run with id 17225446 +Submitted job case.st_archive with id 17225447 + +----- All done ----- + +14 +014 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.014 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:03:18 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:18 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:19 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:19 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:19 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:19 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:19 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:19 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:19 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:19 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.014 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +14 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:03:21 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:22 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:22 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:22 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:22 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:22 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:22 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:22 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:22 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:22 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.014/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180326' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:03:30 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:30 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:30 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:32 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:34 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:34 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:34 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:34 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:34 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:34 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.014/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225448 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.014/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225448 case.st_archive --resubmit +Submitted job id is 17225449 +Submitted job case.run with id 17225448 +Submitted job case.st_archive with id 17225449 + +----- All done ----- + +15 +015 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.015 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:03:48 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:49 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:49 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:49 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:49 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:50 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:50 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:50 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:50 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:50 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.015 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +15 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:03:52 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:03:52 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:03:52 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:03:52 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:03:53 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:03:53 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:03:53 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:03:53 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:03:53 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:03:53 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.015/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180357' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:04:00 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:00 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:01 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:02 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:04 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:05 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:05 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:05 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:05 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:05 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.015/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225451 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.015/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225451 case.st_archive --resubmit +Submitted job id is 17225453 +Submitted job case.run with id 17225451 +Submitted job case.st_archive with id 17225453 + +----- All done ----- + +16 +016 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.016 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:04:23 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:24 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:24 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:24 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:24 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:24 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:24 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:24 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:24 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:24 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.016 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +16 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:04:27 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:27 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:27 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:27 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:27 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:28 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:28 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:28 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:28 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:28 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.016/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180432' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:04:35 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:35 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:35 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:37 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:39 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:39 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:39 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:39 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:39 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:39 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.016/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225460 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.016/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225460 case.st_archive --resubmit +Submitted job id is 17225461 +Submitted job case.run with id 17225460 +Submitted job case.st_archive with id 17225461 + +----- All done ----- + +17 +017 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.017 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:04:55 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:55 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:55 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:56 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:56 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:56 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:56 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:56 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:56 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:56 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.017 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +17 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:04:58 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:04:58 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:04:59 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:04:59 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:04:59 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:04:59 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:04:59 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:04:59 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:04:59 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:04:59 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.017/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180503' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:05:07 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:07 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:07 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:09 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:11 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:11 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:11 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:11 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:11 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:11 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.017/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225464 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.017/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225464 case.st_archive --resubmit +Submitted job id is 17225465 +Submitted job case.run with id 17225464 +Submitted job case.st_archive with id 17225465 + +----- All done ----- + +18 +018 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.018 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:05:23 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:24 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:24 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:24 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:24 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:24 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:24 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:24 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:24 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:24 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.018 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +18 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:05:27 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:27 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:27 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:27 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:27 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:28 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:28 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:28 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:28 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:28 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.018/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180532' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:05:36 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:36 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:36 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:38 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:40 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:40 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:40 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:40 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:40 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:40 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.018/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225467 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.018/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225467 case.st_archive --resubmit +Submitted job id is 17225468 +Submitted job case.run with id 17225467 +Submitted job case.st_archive with id 17225468 + +----- All done ----- + +19 +019 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.019 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:05:54 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:54 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:55 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:55 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:55 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:55 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:55 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:55 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:55 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:55 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.019 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +19 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:05:57 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:05:57 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:05:58 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:05:58 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:05:58 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:05:58 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:05:58 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:05:58 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:05:58 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:05:58 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.019/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180603' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:06:06 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:06:06 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:06:07 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:06:08 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:06:10 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:06:10 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:06:10 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:06:10 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:06:10 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:06:10 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.019/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225469 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.019/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225469 case.st_archive --resubmit +Submitted job id is 17225470 +Submitted job case.run with id 17225469 +Submitted job case.st_archive with id 17225470 + +----- All done ----- + +20 +020 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/allactive/config_pesall.xml +Machine is pm-cpu +Pes setting: grid match is a%ne30.+_oi%.*EC.*to +Pes setting: machine match is pm-cpu|alvarez +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 640, 'NTASKS_LND': 640, 'NTASKS_ROF': 640, 'NTASKS_ICE': 640, 'NTASKS_OCN': 256, 'NTASKS_CPL': 80} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 640, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {'PSTRID_CPL': 8} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '128'} +Pes comments: -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '128'} + Compset is: 20TRSOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: 0f362f469f +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.020 + +----- Starting case_setup ----- + +Successfully cleaned env_mach_specific.xml +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +job is case.run USER_REQUESTED_WALLTIME 12:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2023-10-20 18:06:24 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:06:24 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:06:24 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:06:25 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:06:25 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:06:25 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:06:25 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:06:25 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:06:25 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:06:25 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run +Here is the RUNDIR /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/run.020 +atm_in +atm_modelio.nml +CASEROOT +cpl_modelio.nml +drv_flds_in +drv_in +esp_modelio.nml +glc_modelio.nml +iac_modelio.nml +ice_modelio.nml +lnd_in +lnd_modelio.nml +mosart_in +mpaso_in +mpassi_in +ocn_modelio.nml +rof_modelio.nml +seq_maps.rc +streams.ocean +streams.seaice +wav_modelio.nml +20 + +----- case_build ----- + +Using $OLD_EXECUTABLE = /pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +cp: -r not specified; omitting directory '/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build' +WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user. +For your changes to take effect, run: +./case.build --clean-all +./case.build +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) + 2023-10-20 18:06:27 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:06:27 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:06:28 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:06:28 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:06:28 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:06:28 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:06:28 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:06:28 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:06:28 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:06:28 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml + +----- Starting runtime_options ----- + +Warning: $MODEL_START_TYPE = hybrid +$RUN_REFDIR = /global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.1982-11.01/ +$RUN_REFCASE = v21.LR.SMYLE_IC.1982-11.01 +$RUN_REFDATE = 1982-11-01 + + +----- Saving run script for provenance ----- + +'run.v21.LR.BSMYLE.sh' -> '/pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.020/run_script_provenance/run.v21.LR.BSMYLE.sh.20231020-180632' + +----- Starting case_submit ----- + + +env_batch.xml appears to have changed, regenerating batch scripts +manual edits to these file will be lost! + +Creating batch scripts +Writing case.run script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/template.st_archive +Creating file case.st_archive +Setting resource.RLIMIT_STACK to -1 from (8388608, -1) +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. + 2023-10-20 18:06:35 atm +Create namelist for component eam + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/eam//cime_config/buildnml + 2023-10-20 18:06:36 lnd +Create namelist for component elm + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/elm/cime_config/buildnml + 2023-10-20 18:06:36 ice +Create namelist for component mpassi + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-seaice/cime_config/buildnml + 2023-10-20 18:06:37 ocn +Create namelist for component mpaso + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mpas-ocean/cime_config/buildnml + 2023-10-20 18:06:39 rof +Create namelist for component mosart + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/mosart//cime_config/buildnml + 2023-10-20 18:06:39 glc +Create namelist for component sglc + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sglc/cime_config/buildnml + 2023-10-20 18:06:39 wav +Create namelist for component swav + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/swav/cime_config/buildnml + 2023-10-20 18:06:39 iac +Create namelist for component siac + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/siac/cime_config/buildnml + 2023-10-20 18:06:39 esp +Create namelist for component sesp + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/components/stub_comps/sesp/cime_config/buildnml + 2023-10-20 18:06:39 cpl +Create namelist for component drv + Calling /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/driver-mct/cime_config/buildnml +Checking that inputdata is available as part of case submission +Setting resource.RLIMIT_STACK to -1 from (-1, -1) +Loading input file list: 'Buildconf/mosart.input_data_list' +Loading input file list: 'Buildconf/mpaso.input_data_list' +Loading input file list: 'Buildconf/cpl.input_data_list' +Loading input file list: 'Buildconf/elm.input_data_list' +Loading input file list: 'Buildconf/eam.input_data_list' +Loading input file list: 'Buildconf/mpassi.input_data_list' +GET_REFCASE is false, the user is expected to stage the refcase to the run directory. +Check case OK +submit_jobs case.run +Submit job case.run +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.020/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 12:00:00 -q regular --account mp9 .case.run --resubmit +Submitted job id is 17225472 +Submit job case.st_archive +Deprecated "arg" node detected in /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1982-11.001/case_scripts.020/env_batch.xml, check files /global/cfs/cdirs/mp9/e3sm_tags/E3SMv2.1/cime_config/machines/config_batch.xml +Submitting job script sbatch --time 00:20:00 -q regular --account mp9 --dependency=afterok:17225472 case.st_archive --resubmit +Submitted job id is 17225473 +Submitted job case.run with id 17225472 +Submitted job case.st_archive with id 17225473 + +----- All done ----- + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/README.testcases b/scripts/v2.LR.SF-LE/scripts/toss/README.testcases new file mode 100644 index 0000000..3d8b51e --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/README.testcases @@ -0,0 +1,4 @@ +./create_newcase --case $SCRATCH/testBcase.20TR --compset=WCYCL20TR --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu + +./create_newcase --case $SCRATCH/testBcase --compset=WCYCLSSP370 --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu +./create_newcase --case $SCRATCH/testBcase2 --compset=WCYCLSSP370 --res=ne30pg2_EC30to60E2r2 --machine=pm-cpu diff --git a/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py b/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py new file mode 100755 index 0000000..be28b2b --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +#s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +s2sfcstroot = "/global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/" + +# Machine and project +MACHINE="pm-cpu" +PROJECT="mp9" + +if e3smroot is None: + raise SystemExit("ERROR: E3SM_ROOT must be defined in environment") + +_LIBDIR = os.path.join(e3smroot,"cime","scripts","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts","lib") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, glob, shutil +# import CIME.build as build +from standard_script_setup import * +from CIME.case import Case +from CIME.utils import safe_copy +from argparse import RawTextHelpFormatter +from CIME.locked_files import lock_file, unlock_file + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.getenv("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model, int(args.ensemble_start), int(args.ensemble_end) + +def stage_refcase(rundir, refdir, date, basecasename): + if not os.path.isdir(rundir): + os.makedirs(rundir) + nfname = "v21.LR.SMYLE_IC.ne30np4" + #else: + # nfname = "b.e21.f09_g17" + print ("refdir="+refdir) + print ("rundir="+rundir) + for reffile in glob.iglob(refdir+"/*"): + print ("theNEWFILE="+reffile) + if os.path.basename(reffile).startswith("rpointer"): + safe_copy(reffile, rundir) + else: + newfile = os.path.basename(reffile) + newfile = os.path.join(rundir,newfile) + if "cam.i" in newfile: + if not "001" in newfile: + os.symlink(reffile,newfile.replace(".nc","-original.nc")) + else: + if os.path.lexists(newfile): + os.unlink(newfile) + os.symlink(reffile, newfile) + +def per_run_case_updates(case, date, sdrestdir): + caseroot = case.get_value("CASEROOT") + basecasename = os.path.basename(caseroot)[:-4] + member = os.path.basename(caseroot)[-3:] + + unlock_file("env_case.xml",caseroot=caseroot) + casename = basecasename+"."+member + case.set_value("CASE",casename) + case.flush() + lock_file("env_case.xml",caseroot=caseroot) + + case.set_value("CONTINUE_RUN",False) + case.set_value("BUILD_COMPLETE",True) + case.set_value("RUN_REFDATE",date) + case.set_value("RUN_STARTDATE",date) + case.set_value("RUN_REFDIR",sdrestdir) + + case.case_setup() + + stage_refcase(rundir, sdrestdir, date, basecasename) + case.set_value("BATCH_SYSTEM", "none") + case.set_value("RUNDIR", rundir+member) + safe_copy(os.path.join(caseroot,"env_batch.xml"),os.path.join(caseroot,"LockedFiles","env_batch.xml")) + + +#def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + #sdrestdir, machine, project): +def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + sdrestdir, machine): + caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start),"case_scripts") + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + with Case(caseroot, read_only=False) as case: + if not os.path.isdir(caseroot): + #case.create(os.path.basename(caseroot), e3smroot, compset, res, + # run_unsupported=True, answer="r",walltime="12:00:00", + # user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + #case.create(os.path.basename(caseroot), e3smroot, compset, res, run_unsupported=True, answer="r",walltime="12:00:00",machine="pm-cpu",project="mp9") + case.create(os.path.basename(caseroot), e3smroot, compset, res, walltime="12:00:00",machine_name="pm-cpu",driver="mct",user_mods_dirs=[user_mods_dir]) + # make sure that changing the casename will not affect these variables + user = os.getenv("USER") + cimeroot = os.path.join("/pscratch/sd/n/nanr/".format(user),"v21.LR.SMYLE") + exeroot = os.path.join("/pscratch/sd/n/nanr/".format(user),"v21.LR.SMYLE/exerootdir/bld") + case.set_value("CIME_OUTPUT_ROOT",cimeroot) + if exeroot and os.path.exists(os.path.join(exeroot,"e3sm.exe")): + case.set_value("EXEROOT",exeroot) + else: + case.set_value("EXEROOT",case.get_value("EXEROOT", resolved=True)) + + case.set_value("RUNDIR",case.get_value("RUNDIR",resolved=True)+".001") + + case.set_value("RUN_TYPE","hybrid") + # case.set_value("JOB_QUEUE","economy") + case.set_value("GET_REFCASE",False) + case.set_value("RUN_REFDIR",sdrestdir) + case.set_value("RUN_REFCASE", "v21.SMYLE_IC.ne30np4.{}.01".format(date[:7])) + case.set_value("NTHRDS", 1) + + case.set_value("STOP_OPTION","nmonths") + case.set_value("STOP_N", 24) + #case.set_value("REST_OPTION","nmonths") + #case.set_value("REST_N", 24) + + case.set_value("CCSM_BGC","CO2A") + case.set_value("EXTERNAL_WORKFLOW",True) + + nint = 3 + n2nt = 11 + rundir = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start),"run") + #rundir = case.get_value("RUNDIR") + #print("rundir 1 = {}".format(rundir)) + #rundir = rundir[:-n2nt]+"001/run.{:03d}".format(ensemble_start) + #print("rundir 2 = {}".format(rundir)) + #rundir = rundir[:-nint]+"{:03d}".format(ensemble_start) + #print("rundir 3 = {}".format(rundir)) + #case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir) + #if not exeroot: + #build.case_build(caseroot, case=case) + + return caseroot + +def clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite): + + nint=3 + cloneroot = caseroot + for i in range(ensemble_start+1, ensemble_end+1): + member_string = '{{0:0{0:d}d}}'.format(nint).format(i) + if ensemble_end > ensemble_start: + caseroot = caseroot[:-nint] + member_string + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + if not os.path.isdir(caseroot): + with Case(cloneroot, read_only=False) as clone: + clone.create_clone(caseroot, keepexe=True, + user_mods_dir=user_mods_dir) + with Case(caseroot, read_only=True) as case: + # rundir is initially 00 reset to current member + rundir = case.get_value("RUNDIR") + rundir = rundir[:-nint]+member_string + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir) + +def _main_func(description): + #date, basecasename = parse_command_line(sys.argv, description) + date, model, ensemble_start, ensemble_end = parse_command_line(sys.argv, description) + basecasename = "v21.LR.BSMYLE.ne30np4" + + # TODO make these input vars + + basemonth = int(date[5:7]) + baseyear = int(date[0:4]) + baseroot = "/pscratch/sd/n/nanr/v21.LR.SMYLE/" + res = "ne30pg2_EC30to60E2r2" + waccm = False + #if model == "e3smsmyle": + #compset = "WCYCL20TR" + compset = "WCYCL20TR" + + print ("baseyear is {} basemonth is {}".format(baseyear,basemonth)) + + overwrite = True + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.LR.SMYLE_IC.ne30np4."+date[0:7]+".01","{}".format(date)) + machine = "pm-cpu" + project = "mp9" + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + + # END TODO + print("basemonth = {}".format(basemonth)) + #caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + #compset, overwrite, sdrestdir, machine, project) + caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + compset, overwrite, sdrestdir,machine) + clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, overwrite) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py-safe b/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py-safe new file mode 100755 index 0000000..ee5ac7c --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/buildcase-v2.SMYLE.py-safe @@ -0,0 +1,234 @@ +#!/usr/bin/env python +import os, sys +e3smroot = "/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1/E3SM/" +s2sfcstroot = os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), os.path.pardir) +s2sfcstroot = "/global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/" + +if e3smroot is None: + raise SystemExit("ERROR: E3SM_ROOT must be defined in environment") + +_LIBDIR = os.path.join(e3smroot,"cime","scripts","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","scripts","lib") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME","Tools") +sys.path.append(_LIBDIR) +_LIBDIR = os.path.join(e3smroot,"cime","CIME") +sys.path.append(_LIBDIR) + +import datetime, glob, shutil +# import CIME.build as build +from standard_script_setup import * +from CIME.case import Case +from CIME.utils import safe_copy +from argparse import RawTextHelpFormatter +from CIME.locked_files import lock_file, unlock_file + +def parse_command_line(args, description): + parser = argparse.ArgumentParser(description=description, + formatter_class=RawTextHelpFormatter) + CIME.utils.setup_standard_logging_options(parser) + parser.add_argument("--date", + help="Specify a start Date") + + parser.add_argument("--ensemble-start",default=1, + help="Specify the first ensemble member") + parser.add_argument("--ensemble-end",default=20, + help="Specify the last ensemble member") + + parser.add_argument("--model",help="Specify a case (e3smsmyle)", default="e3smsmyle") + + args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser) + cdate = os.getenv("CYLC_TASK_CYCLE_POINT") + + if args.date: + try: + date = datetime.datetime.strptime(args.date, '%Y-%m-%d') + except ValueError: + raise ValueError("Incorrect data format, should be YYYY-MM-DD or YYYY-MM") + elif cdate: + date = datetime.datetime.strptime(cdate, '%Y-%m-%d') + else: + date = datetime.date.today() + date = date.replace(day=date.day-1) + + return date.strftime("%Y-%m-%d"), args.model, int(args.ensemble_start), int(args.ensemble_end) + +def stage_refcase(rundir, refdir, date, basecasename): + if not os.path.isdir(rundir): + os.makedirs(rundir) + nfname = "v21.LR.SMYLE_IC.ne30np4" + #else: + # nfname = "b.e21.f09_g17" + print ("refdir="+refdir) + print ("rundir="+rundir) + for reffile in glob.iglob(refdir+"/*"): + print ("theNEWFILE="+reffile) + if os.path.basename(reffile).startswith("rpointer"): + safe_copy(reffile, rundir) + else: + newfile = os.path.basename(reffile) + #newfile = "{}.cice.r.{}-00000.nc".format(nfname,date) + newfile = os.path.join(rundir,newfile) + if "cam.i" in newfile: + if not "001" in newfile: + os.symlink(reffile,newfile.replace(".nc","-original.nc")) + else: + if os.path.lexists(newfile): + os.unlink(newfile) + os.symlink(reffile, newfile) + +def per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir): + caseroot = case.get_value("CASEROOT") + basecasename = os.path.basename(caseroot)[:-4] + member = os.path.basename(caseroot)[-3:] + + unlock_file("env_case.xml",caseroot=caseroot) + casename = basecasename+"."+member + case.set_value("CASE",casename) + case.flush() + lock_file("env_case.xml",caseroot=caseroot) + + case.set_value("CONTINUE_RUN",False) + case.set_value("BUILD_COMPLETE",True) + case.set_value("RUN_REFDATE",date) + case.set_value("RUN_STARTDATE",date) + case.set_value("RUN_REFDIR",sdrestdir) + #case.set_value("PROJECT","NCGD0047") +# case.set_value("PROJECT","mp9") +# case.set_value("OCN_TRACER_MODULES","iage cfc ecosys") +# dout_s_root = case.get_value("DOUT_S_ROOT") +# dout_s_root = os.path.join(os.path.dirname(dout_s_root),casename) +# if dout_s_root.startswith("/glade/scratch"): +# dout_s_root = dout_s_root.replace("/glade/scratch/","/glade/p/nsc/ncgd0042/") +# case.set_value("DOUT_S_ROOT",dout_s_root) + # restage user_nl files for each run + for usermod in glob.iglob(user_mods_dir+"/user*"): + safe_copy(usermod, caseroot) + + case.case_setup() + + stage_refcase(rundir, sdrestdir, date, basecasename) + case.set_value("BATCH_SYSTEM", "none") + safe_copy(os.path.join(caseroot,"env_batch.xml"),os.path.join(caseroot,"LockedFiles","env_batch.xml")) +# lock_file("env_batch.xml",caseroot=caseroot) + + +def build_base_case(date, baseroot, basecasename, basemonth,res, ensemble_start, compset, overwrite, + sdrestdir, user_mods_dir, pecount=None): + caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".{:03d}".format(ensemble_start)) + #caseroot = os.path.join(baseroot,basecasename+"."+date[:7]+".001") + #caseroot = os.path.join(baseroot,basecasename+".{}".format(date[:7])+".00") + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + + with Case(caseroot, read_only=False) as case: + if not os.path.isdir(caseroot): + #case.create(os.path.basename(caseroot), e3smroot, compset, res, + # run_unsupported=True, answer="r",walltime="12:00:00", + # user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + case.create(os.path.basename(caseroot), e3smroot, compset, res, + run_unsupported=True, answer="r",walltime="12:00:00", + user_mods_dir=user_mods_dir, pecount=pecount, project="NCGD0047", workflowid="timeseries") + # make sure that changing the casename will not affect these variables + user = os.getenv("USER") + cimeroot = os.path.join("/glade/scratch/{}/".format(user),"SMYLE") + exeroot = os.path.join("/glade/scratch/{}/".format(user),"SMYLE/exerootdir/bld") + case.set_value("CIME_OUTPUT_ROOT",cimeroot) + #case.set_value("CIME_OUTPUT_ROOT","/glade/scratch/nanr/SMYLE") + if exeroot and os.path.exists(os.path.join(exeroot,"e3sm.exe")): + case.set_value("EXEROOT",exeroot) + else: + case.set_value("EXEROOT",case.get_value("EXEROOT", resolved=True)) + + case.set_value("RUNDIR",case.get_value("RUNDIR",resolved=True)+".001") + case.set_value("CAM_CONFIG_OPTS",case.get_value("CAM_CONFIG_OPTS")+" -cosp ") + + case.set_value("RUN_TYPE","hybrid") + #case.set_value("JOB_QUEUE","economy",subgroup="case.run") + case.set_value("JOB_QUEUE","economy") + case.set_value("GET_REFCASE",False) + case.set_value("RUN_REFDIR",sdrestdir) + case.set_value("RUN_REFCASE", "b.e21.SMYLE_IC.f09_g17.{}.01".format(date[:7])) + case.set_value("OCN_TRACER_MODULES","") + case.set_value("OCN_TRACER_MODULES","iage") + case.set_value("OCN_CHL_TYPE","diagnostic") + case.set_value("NTHRDS", 1) + + case.set_value("STOP_OPTION","nmonths") + case.set_value("STOP_N", 24) + #case.set_value("REST_OPTION","nmonths") + #case.set_value("REST_N", 24) + + case.set_value("CCSM_BGC","CO2A") + case.set_value("EXTERNAL_WORKFLOW",True) + case.set_value("CLM_NAMELIST_OPTS", "use_init_interp=.true.") + + nint = 3 + n2nt = 11 + rundir = case.get_value("RUNDIR") + print("rundir 1 = {}".format(rundir)) + #rundir = os.path.join("/glade/scratch/nanr/","SMYLE","b.e21.BSMYLE.f09_g17."+date[0:7]+".001", + #rundir = os.path.join(baseroot,basecasename+"."+date[:7]+".001") + rundir = rundir[:-n2nt]+"001/run.{:03d}".format(ensemble_start) + print("rundir 2 = {}".format(rundir)) + rundir = rundir[:-nint]+"{:03d}".format(ensemble_start) + print("rundir 3 = {}".format(rundir)) + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir) + #if not exeroot: + #build.case_build(caseroot, case=case) + + return caseroot + +def clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite): + + nint=3 + cloneroot = caseroot + for i in range(ensemble_start+1, ensemble_end+1): + member_string = '{{0:0{0:d}d}}'.format(nint).format(i) + if ensemble_end > ensemble_start: + caseroot = caseroot[:-nint] + member_string + if overwrite and os.path.isdir(caseroot): + shutil.rmtree(caseroot) + if not os.path.isdir(caseroot): + with Case(cloneroot, read_only=False) as clone: + clone.create_clone(caseroot, keepexe=True, + user_mods_dir=user_mods_dir) + with Case(caseroot, read_only=True) as case: + # rundir is initially 00 reset to current member + rundir = case.get_value("RUNDIR") + rundir = rundir[:-nint]+member_string + case.set_value("RUNDIR",rundir) + per_run_case_updates(case, date, sdrestdir, user_mods_dir, rundir) + +def _main_func(description): + #date, basecasename = parse_command_line(sys.argv, description) + date, model, ensemble_start, ensemble_end = parse_command_line(sys.argv, description) + basecasename = "v21.BSMYLE.ne30np4" + + # TODO make these input vars + + basemonth = int(date[5:7]) + baseyear = int(date[0:4]) + baseroot = "/pscratch/sd/n/nanr/v21.SMYLE/" + res = "ne30pg2_EC30to60E2r2" + waccm = False + if model == "e3smsmyle": + compset = "WCYCL20TR" + + print ("baseyear is {} basemonth is {}".format(baseyear,basemonth)) + + overwrite = True + sdrestdir = os.path.join("/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init","v21.SMYLE_IC.ne30np4."+date[0:7]+".01","{}".format(date)) + + user_mods_dir = os.path.join(s2sfcstroot,"user_mods","e3smsmyle") + + # END TODO + print("basemonth = {}".format(basemonth)) + caseroot = build_base_case(date, baseroot, basecasename, basemonth, res, ensemble_start, + compset, overwrite, sdrestdir, user_mods_dir+'.base', pecount="S") + clone_base_case(date, caseroot, ensemble_start, ensemble_end, sdrestdir, user_mods_dir, overwrite) + +if __name__ == "__main__": + _main_func(__doc__) diff --git a/scripts/v2.LR.SF-LE/scripts/toss/do-st.archive.sh b/scripts/v2.LR.SF-LE/scripts/toss/do-st.archive.sh new file mode 100755 index 0000000..61a185e --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/do-st.archive.sh @@ -0,0 +1,128 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1987 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="m4417" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20231020" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +#MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +#CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} +CASE_ARCHIVE_DIR=/global/cfs/cdirs/mp9/archive/v21.LR.SMYLE/${MAIN_CASE_NAME}/archive.${mbr} + +CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} +CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + +# Make directories created by this script world-readable +umask 022 + +# st_archive +case_archive + +# All done +echo $'\n----- All done -----\n' + +} + + +#----------------------------------------------------- +case_archive() { + + echo $'\n----- Starting case_archive -----\n' + echo ${CASE_SCRIPTS_DIR} + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.st_archive + + popd +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/doh.sh b/scripts/v2.LR.SF-LE/scripts/toss/doh.sh new file mode 100755 index 0000000..10040d4 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/doh.sh @@ -0,0 +1,31 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 001 002 ) +for mbr in "${array[@]}" +do + +main() { + +echo ${mbr} + +# For debugging, uncomment libe below +#set -x + +useyear=1959 + + +# --- Configuration flags ---- + +CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +if (${mbr} == "001") +then + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +fi diff --git a/scripts/v2.LR.SF-LE/scripts/toss/fix.test b/scripts/v2.LR.SF-LE/scripts/toss/fix.test new file mode 100644 index 0000000..7d1d665 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/fix.test @@ -0,0 +1,12 @@ +./xmlchange RUN_REFCASE=v21.LR.SMYLE_IC.ne30np4.1972-11.01 +./xmlquery RUN_REFDATE +./xmlchange RUN_REFDATE=1972-11-01 +./xmlchange RUN_STARTDATE=1972-11-01 +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/v2* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase.20TR/run/ +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/rpointer.* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase.20TR/run/ + +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/v2* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +cp /pscratch/sd/n/nanr/v21.LR.SMYLE/v21.LR.BSMYLE.1972-11.001/run/rpointer.* /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +cd /pscratch/sd/n/nanr/e3sm_scratch/pm-cpu/testBcase/run/ +rm v21.LR.BSMYLE.1972-11.001.mpas* + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE-TESTME.sh-NotUsing b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE-TESTME.sh-NotUsing new file mode 100755 index 0000000..fc5cc99 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE-TESTME.sh-NotUsing @@ -0,0 +1,506 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1977 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +#CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=false +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +#user_nl() { + + +#cat << EOF >> user_nl_elm + +#! Override +#check_finidat_fsurdat_consistency = .false. +#check_dynpft_consistency = .false. + +#EOF + + +#} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + #${CODE_ROOT}/cime/scripts/create_newcase \ + #--case ${CASE_NAME} \ + #--case-group ${CASE_GROUP} \ + #--output-root ${CASE_ROOT} \ + #--script-root ${CASE_SCRIPTS_DIR} \ + #--handle-preexisting-dirs u \ + #--compset ${COMPSET} \ + #--res ${RESOLUTION} \ + #--machine ${MACHINE} \ + #--project ${PROJECT} \ + #--walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + echo "${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/" + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + # rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + # ln -s ${ics}/pert.${shortmbr}/${eamic} $MAIN_CASE_ROOT/run.${mbr}/ + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + # ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + #if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ## ./xmlchange DEBUG=${DEBUG_COMPILE^^} + #fi + + # Run CIME case.build + # ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + # ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + # ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + # ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + # ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + # ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + # ./xmlchange BUDGETS=TRUE + + # Set resubmissions + #if (( RESUBMIT > 0 )); then + # ./xmlchange RESUBMIT=${RESUBMIT} + #fi + + # Run type + # Start from default of user-specified initial conditions + #if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + # ./xmlchange RUN_TYPE="startup" + # ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + #elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + # ./xmlchange CONTINUE_RUN="TRUE" + + #elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + # ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + # ./xmlchange GET_REFCASE=${GET_REFCASE} + # ./xmlchange RUN_REFDIR=${RUN_REFDIR} + # ./xmlchange RUN_REFCASE=${RUN_REFCASE} + # ./xmlchange RUN_REFDATE=${RUN_REFDATE} + #echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + #echo '$RUN_REFDIR = '${RUN_REFDIR} + #echo '$RUN_REFCASE = '${RUN_REFCASE} + #echo '$RUN_REFDATE = '${START_DATE} + + #else + #echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + #exit 380 + #fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + # ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + # cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh new file mode 100755 index 0000000..793bd69 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh @@ -0,0 +1,551 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +#useyear=1980 +#useyear=1981 +#useyear=1982 +useyear=1980 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + #MAIN_CASE_NAME="v21.LR.BSMYLE-MYTEST.${useyear}-${usemonth}.${mbr}" + MAIN_CASE_NAME="v2.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v2.LR.SMYLE/${MAIN_CASE_NAME}/" + +# Sub-directories +#CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +CASE_BUILD_DIR=/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="28" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nmonths" + REST_N="28" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" +OLD_EXECUTABLE="/pscratch/sd/n/nanr/v2.LR.SMYLE/exeroot/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=false +else + do_case_build=false +fi +do_case_submit=true +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,365,146,292,292, 73, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + perteamic="v21.LR.SMYLE_IC.pert.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + ln -s ${ics}/${useyear}-${usemonth}-01/v21.* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${imbr} -ne "1" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s ${ics}/pert.${shortmbr}/${perteamic} $MAIN_CASE_ROOT/run.${mbr}/${eamic} + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking new file mode 100755 index 0000000..181786e --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking @@ -0,0 +1,540 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { +# years=( 1958 1959 ) +useyear=1960 + +#array=( 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 ) +array=( 001 ) +for mbr in "${array[@]}" +do + +# For debugging, uncomment libe below +#set -x + + +# Year array YYYY: + +echo ${mbr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${mbr}" +#if [[ ${mbr} -lt 10 ]] +#then +# CASE_NAME="v21.LR.BSMYLE.00${mbr}" +#else +# CASE_NAME="v21.LR.BSMYLE.0${mbr}" +#fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-11-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv21-SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="24" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="${useyear}1101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="12" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv21-SMYLE/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/trop_mozart_aero/emis/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016-c230601.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + #if + # Build with COSP, except for a data atmosphere (datm) + # if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + # echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + # echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # copy restarts + if ${mbr}==1 then + USE_REFDIR="${USE_RUN_REFDIR}/${useyear}-11/" + elif + if ${mbr}<10 then + PERT_REFDIR="${USE_REFDIR}/pert.0${mbr}/" + else + USE_RUN_REFDIR="${USE_REFDIR}/pert.${mbr}/" + fi + fi + cp $USE_REFDIR/rpointer* $RUNDIR/ + ln -s $USE_REFDIR/v21.* $RUNDIR/ + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking2 b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking2 new file mode 100755 index 0000000..8a1d68e --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-notWorking2 @@ -0,0 +1,541 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { +# years=( 1958 1959 ) +useyear=1960 + +#array=( 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 ) +#array=( 001 ) +#for mbr in "${array[@]}" +for (( mbr=1; i<10; i++)) +do + +# For debugging, uncomment libe below +#set -x + + +# Year array YYYY: + +echo ${mbr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${mbr}" +#if [[ ${mbr} -lt 10 ]] +#then +# CASE_NAME="v21.LR.BSMYLE.00${mbr}" +#else +# CASE_NAME="v21.LR.BSMYLE.0${mbr}" +#fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-11-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv21-SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="24" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="${useyear}1101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="12" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv21-SMYLE/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/inputdata/atm/cam/chem/trop_mozart_aero/emis/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016-c230601.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + #if + # Build with COSP, except for a data atmosphere (datm) + # if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + # echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + # echo $'\nConfiguring E3SM to use the COSP simulator\n' + #./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # copy restarts + if ${mbr}==1 then + USE_REFDIR="${USE_RUN_REFDIR}/${useyear}-11/" + elif + if ${mbr}<10 then + PERT_REFDIR="${USE_REFDIR}/pert.0${mbr}/" + else + USE_RUN_REFDIR="${USE_REFDIR}/pert.${mbr}/" + fi + fi + cp $USE_REFDIR/rpointer* $RUNDIR/ + ln -s $USE_REFDIR/v21.* $RUNDIR/ + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe1 b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe1 new file mode 100755 index 0000000..5601196 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe1 @@ -0,0 +1,458 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 001 ) +for mbr in "${array[@]}" +do + +main() { + +echo ${mbr} + +# For debugging, uncomment libe below +#set -x + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.1960-11.${mbr}" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1958-11-01" + +useyear=1960 +GET_REFCASE=true +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + ./xmlchange DATM_CLMNCEP_YR_ALIGN=1958 + ./xmlchange DATM_CLMNCEP_YR_START=1958 + ./xmlchange DATM_CLMNCEP_YR_END=2010 + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe2 b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe2 new file mode 100755 index 0000000..969c377 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.BSMYLE.sh-safe2 @@ -0,0 +1,456 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 001 002 003 004 005 006 007 008 009 010 ) +for mbr in "${array[@]}" +do + +main() { + +echo ${mbr} + +# For debugging, uncomment libe below +#set -x + +useyear=1959 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1959-11-01" + +GET_REFCASE=true +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-11.01" +RUN_REFDATE="${useyear}-11-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/run.${mbr} + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.FROM-AUFIRE.sh b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.FROM-AUFIRE.sh new file mode 100755 index 0000000..27ce9a2 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.FROM-AUFIRE.sh @@ -0,0 +1,563 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +# MACHINE=cori-knl +MACHINE=pm-cpu +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SMYLE-TEST_jul13.${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1972-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=FALSE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +#RUN_REFDIR="/global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_${iyr}/archive/rest/archive/rest/2019-01-01-00000" +RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/" +RUN_REFCASE="v2.LR.SSP370_${iyr}" +RUN_REFDATE="2019-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="${SCRATCH}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nmonths" + STOP_N="18" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20190701" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nmonths" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/pscratch/sd/n/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/v2.AUFIRE/forcingFiles/cmip6_ssp370_mam4_smoothed_bc_a4_elev_2018-202007_c230612.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # cp ${RUN_REFDIR}/* ${CASE_RUN_DIR} + mkdir -p ${CASE_RUN_DIR} + # cp ${RUN_REFDIR}/rpointer.atm ${CASE_RUN_DIR} + # ln -s ${RUN_REFDIR}/v2.* ${CASE_RUN_DIR} + + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.atm ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.drv ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ice ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.lnd ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.ocn ${CASE_RUN_DIR} + cp /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/rpointer.rof ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.cpl.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h1.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h2.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h3.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h4.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h5.2018-12-12-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.h6.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.i.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.eam.rs.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h0.2018-12.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.h1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.elm.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.r.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh0.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mosart.rh1.2019-01-01-00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpaso.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + ln -s /pscratch/sd/n/nanr/archive/v2.LR.SSP370_0111/rest/2019-01-01-00000/v2.LR.SSP370_0111.mpassi.rst.2019-01-01_00000.nc ${CASE_RUN_DIR} + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase new file mode 100755 index 0000000..3da9c9a --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRCRUELMBGC.sh-missingBGC-usecase @@ -0,0 +1,468 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="I20TRCRUELMBGC" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.${COMPSET}_test" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +#MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +#START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +#GET_REFCASE=TRUE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +#RUN_REFCASE="v2.LR.historical_${iyr}" +#RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + PELAYOUT="L" + WALLTIME="30:00:00" + STOP_OPTION="nyears" + STOP_N="30" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRELM.sh b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRELM.sh new file mode 100755 index 0000000..9c858dd --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.I20TRELM.sh @@ -0,0 +1,447 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="I20TRELM" +COMPSET="20TR_DATM%CRU_ELM%SP_SICE_SOCN_MOSART_SGLC_SWAV_SIAC_SESP" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.I20TRELM_CRUNCEP" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="1958-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=true +RUN_REFDIR="/pscratch/sd/n/nanr/archive/s2sLandSpinupSE_perl_ICRUELM/rest/0401-01-01-00000" +RUN_REFCASE="s2sLandSpinupSE_perl_ICRUELM" +RUN_REFDATE="0401-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + + +cat << EOF >> user_nl_elm + +! Override +check_finidat_fsurdat_consistency = .false. +check_dynpft_consistency = .false. + +EOF + + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + ./xmlchange DATM_CLMNCEP_YR_ALIGN=1958 + ./xmlchange DATM_CLMNCEP_YR_START=1958 + ./xmlchange DATM_CLMNCEP_YR_END=2010 + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.ICRUELM.sh b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.ICRUELM.sh new file mode 100755 index 0000000..b836deb --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/run.v2.LR.ICRUELM.sh @@ -0,0 +1,468 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +COMPSET="ICRUELM" +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.${COMPSET}_test" +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +#MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +#START_DATE="1976-01-01" + +# Additional options for 'branch' and 'hybrid' +#GET_REFCASE=TRUE +#RUN_REFDIR="/global/cscratch1/sd/nanr/archive/AMIP/v2.LR.historical_${iyr}/archive/rest/1976-01-01-00000" +#RUN_REFCASE="v2.LR.historical_${iyr}" +#RUN_REFDATE="1976-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.SMYLE/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + PELAYOUT="L" + WALLTIME="30:00:00" + STOP_OPTION="nyears" + STOP_N="30" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override +check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/test-main.sh b/scripts/v2.LR.SF-LE/scripts/toss/test-main.sh new file mode 100755 index 0000000..48d4633 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/test-main.sh @@ -0,0 +1,46 @@ +#!/bin/bash -fe +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 2 3 4 5 6 7 8 9 10 ) +for mbr in "${array[@]}" +do + +# For debugging, uncomment libe below +#set -x + + +echo ${mbr} + +smbr=${mbr:1:3} +echo $smbr + +if [[ ${mbr} -lt "10" ]] +then + usembr="00${mbr}" + echo ${usembr} + MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +else + usembr="0${mbr}" + echo ${usembr} +fi + + +# --- Configuration flags ---- + +#CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +#if (${mbr} == "001") +#then + #MAIN_CASE_NAME="v21.LR.BSMYLE.${useyear}-11.${mbr}" +#fi + +if [[ ${mbr} -eq "001" ]] +then + CASE_NAME="v21.LR.BSMYLE.00${mbr}" + CASE_NAME="v21.LR.BSMYLE.${mbr}" + echo "$CASE_NAME 1" +else + CASE_NAME="v21.LR.BSMYLE.${mbr}" + echo "$CASE_NAME 2" +fi +done + + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/test.sh b/scripts/v2.LR.SF-LE/scripts/toss/test.sh new file mode 100755 index 0000000..b867a64 --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/test.sh @@ -0,0 +1,25 @@ +#!/bin/bash -fe +array=( 001 ) +for mbr in "${array[@]}" +do + +# For debugging, uncomment libe below +#set -x + + +echo ${mbr} + +# --- Configuration flags ---- + +if [[ ${mbr} -lt 10 ]] +then + CASE_NAME="v21.LR.BSMYLE.00${mbr}" + CASE_NAME="v21.LR.BSMYLE.${mbr}" + echo "$CASE_NAME 1" +else + CASE_NAME="v21.LR.BSMYLE.${mbr}" + echo "$CASE_NAME 2" +fi +done + + diff --git a/scripts/v2.LR.SF-LE/scripts/toss/testSmyle.sh b/scripts/v2.LR.SF-LE/scripts/toss/testSmyle.sh new file mode 100755 index 0000000..ff152fb --- /dev/null +++ b/scripts/v2.LR.SF-LE/scripts/toss/testSmyle.sh @@ -0,0 +1,517 @@ +#!/bin/bash +###!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 001 002 003 004 005 006 007 008 009 010 ) +array=( 1 ) +#array=( 001 002 ) +for imbr in "${array[@]}" +do + +main() { + +echo ${imbr} + +if [[ ${imbr} -lt "10" ]] +then + mbr="00${imbr}" + echo ${mbr} + #MAIN_CASE_NAME="v21.LR.BSMYLE.TEST2.${useyear}-${usemonth}.${mbr}" +else + mbr="0${imbr}" + echo ${mbr} +fi + + +# For debugging, uncomment libe below +#set -x + +useyear=1972 +usemonth=11 + + +# --- Configuration flags ---- + +# Machine and project +MACHINE=pm-cpu +PROJECT="mp9" + +# Simulation +#COMPSET="WCYCLSSP370" # SSP370 transient +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v21.LR.BSMYLE.${useyear}-${usemonth}.${mbr}" +if [[ ${imbr} -eq "1" ]] +then + MAIN_CASE_NAME="v21.LR.BSMYLE.TEST2.${useyear}-${usemonth}.${mbr}" +fi +CASE_GROUP="v21.LR" + +# Code and compilation +CHECKOUT="20230602" +BRANCH="maint-2.1" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${useyear}-${usemonth}-01" + +GET_REFCASE=false +RUN_REFDIR="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-${usemonth}.01/" +RUN_REFCASE="v21.LR.SMYLE_IC.ne30np4.${useyear}-${usemonth}.01" +RUN_REFDATE="${useyear}-${usemonth}-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +RUN_REFDIR="/global/cfs/cdirs/mp9/archive/E3SMv2-SMBB/v2.LR.historical-smbb_0301/archive/rest/1991-01-01-00000/" +RUN_REFCASE="v21.LR.historical-smbb_0301" +RUN_REFDATE="1991-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + + +# Additional options for 'branch' and 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/E3SM/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +MAIN_CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_NAME}" +CASE_ROOT="/pscratch/sd/n/${USER}/v21.LR.SMYLE/${MAIN_CASE_ROOT}/" + +# Sub-directories +CASE_BUILD_DIR=${MAIN_CASE_ROOT}/build +#CASE_BUILD_DIR=/pscratch/sd/n/nanr/v21.LR.SMYLE/exeroot/build +CASE_ARCHIVE_DIR=${MAIN_CASE_ROOT}/archive.${mbr} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + #CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts.${mbr} + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run.${mbr} + PELAYOUT=${layout} + WALLTIME="2:00:00" + STOP_OPTION=${units} + STOP_N=${length} + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts.${mbr} + #CASE_RUN_DIR=${MAIN_CASE_ROOT}/run.${mbr} + CASE_SCRIPTS_DIR=${MAIN_CASE_ROOT}/case_scripts + CASE_RUN_DIR=${MAIN_CASE_ROOT}/run + #PELAYOUT="L" + WALLTIME="12:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + REST_OPTION="nyears" + REST_N="5" # How often to write a restart file + RESUBMIT="4" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="5" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +#OLD_EXECUTABLE="${MAIN_CASE_ROOT}/build" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +if [[ ${imbr} -eq "1" ]] +then + do_case_build=true +else + do_case_build=false +fi +do_case_submit=false +do_get_restarts=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +#user_nl() { + + +#cat << EOF >> user_nl_elm + +#! Override +#check_finidat_fsurdat_consistency = .false. +#check_dynpft_consistency = .false. + +#EOF + + +#} + +# ===================================== +# Customize MPAS stream files if needed +# ===================================== + +patch_mpas_streams() { + +echo + +} + + + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + #if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + #echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + #else + #echo $'\nConfiguring E3SM to use the COSP simulator\n' + ## ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + #fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + # Lastly - copy Initial conditions + echo "Here is the RUNDIR ${CASE_RUN_DIR}" + eamic="v21.LR.SMYLE_IC.ne30np4.${useyear}-${usemonth}.01.eam.i.${useyear}-${usemonth}-01-00000.nc" + ics="/global/cfs/cdirs/mp9/E3SMv2.1-SMYLE/inputdata/e3sm_init/v21.LR.SMYLE_IC.ne30np4.${useyear}-${usemonth}.01/" + + ls ${CASE_RUN_DIR} + + # pre-stage ICs + #cp ${ics}/${useyear}-${usemonth}-01/rpointer.* ${CASE_RUN_DIR}/ + #ln -s ${ics}/${useyear}-${usemonth}-01/v2* ${CASE_RUN_DIR}/ + + cp $RUN_REFDIR/rpointer.* ${CASE_RUN_DIR}/ + ln -s $RUN_REFDIR/v2* ${CASE_RUN_DIR}/ + + + # perturb the atmosphere IC + if [[ ${mbr} -ne "001" ]] + then + shortmbr=${mbr:1:3} + echo $shortmbr + rm $MAIN_CASE_ROOT/run.${mbr}/${eamic} + ln -s -fp ${ics}/${useyear}-${usemonth}-01/pert.${shortmbr}/${eamic} $MAIN_CASE_ROOT/run.${mbr}/ + fi + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + #do_case_build = true + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SF-LE/user_nl_files/user_nl_eam b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_eam new file mode 100644 index 0000000..8e6e264 --- /dev/null +++ b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_eam @@ -0,0 +1,26 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', + 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', +'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', + 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700' +, 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I', +'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ', +'CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + diff --git a/scripts/v2.LR.SF-LE/user_nl_files/user_nl_elm b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_elm new file mode 100644 index 0000000..f1fa0b2 --- /dev/null +++ b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_elm @@ -0,0 +1,9 @@ + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_year_consistency = .true. + diff --git a/scripts/v2.LR.SF-LE/user_nl_files/user_nl_mosart b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_mosart new file mode 100644 index 0000000..ab66396 --- /dev/null +++ b/scripts/v2.LR.SF-LE/user_nl_files/user_nl_mosart @@ -0,0 +1,5 @@ + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 + diff --git a/scripts/v2.LR.SMBB/README.bundle b/scripts/v2.LR.SMBB/README.bundle new file mode 100644 index 0000000..e761fc4 --- /dev/null +++ b/scripts/v2.LR.SMBB/README.bundle @@ -0,0 +1,12 @@ +Rough instructions +# set up runs +./run.v2.LR.historical-smbb_YYYY.sh + +# copy restarts, set EXEROOT +./change_JOB-QUEUE.csh + +# submit job +sbatch ./v2.LR.SMBB_bundle.sh + +# Reset CONTINUE_RUN if necessary (e.g., if I've already submitted the bundle but it fails) +./change_CONT_RUN.scr diff --git a/scripts/v2.LR.SMBB/change_CONT_RUN.scr b/scripts/v2.LR.SMBB/change_CONT_RUN.scr new file mode 100755 index 0000000..7646350 --- /dev/null +++ b/scripts/v2.LR.SMBB/change_CONT_RUN.scr @@ -0,0 +1,19 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +#foreach mbr ( 0111 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +foreach mbr ( 0101 0151 0201 0251 0241 0261 0271 0281 0291 0301 ) + +#set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/case_scripts +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange CONTINUE_RUN=TRUE +#./xmlchange DOUT_S=FALSE + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/change_PROJ.csh b/scripts/v2.LR.SMBB/change_PROJ.csh new file mode 100755 index 0000000..ff01f8d --- /dev/null +++ b/scripts/v2.LR.SMBB/change_PROJ.csh @@ -0,0 +1,16 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange PROJECT=m4195 +#./xmlchange PROJECT=mp9 + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/change_REFCASE-SSP370.csh b/scripts/v2.LR.SMBB/change_REFCASE-SSP370.csh new file mode 100755 index 0000000..ff26d1b --- /dev/null +++ b/scripts/v2.LR.SMBB/change_REFCASE-SSP370.csh @@ -0,0 +1,19 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${mbr}/run +set REFROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/archive/rest/2015-01-01-00000/ + +cd $CASEROOT/ +./xmlchange RUN_REFCASE=v2.LR.historical-smbb_${mbr} +./xmlchange RUN_REFDIR=$REFROOT + + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE-SSP370.csh b/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE-SSP370.csh new file mode 100755 index 0000000..fecd730 --- /dev/null +++ b/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE-SSP370.csh @@ -0,0 +1,31 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${mbr}/run +set REFROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/archive/rest/2015-01-01-00000/ + +set doThis = 0 +if (doThis == 1) then +cd $CASEROOT/ +./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/ +./xmlchange JOB_QUEUE=debug --subgroup case.st_archive +./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive + +mv case.st_archive tmp.st_archive +cat tmp.st_archive | sed 's/101/1/' > case.st_archive + +endif + +cd $RUNROOT/ +echo $REFROOT/rpointer.atm +cp $REFROOT/rpointer* . +ln -s $REFROOT/v2.*.nc . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE.csh b/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE.csh new file mode 100755 index 0000000..2129ab6 --- /dev/null +++ b/scripts/v2.LR.SMBB/cp_restarts-plus-JOB_QUEUE.csh @@ -0,0 +1,30 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +#foreach mbr ( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +foreach mbr ( 0101 0151 0201 0251 0301 ) +#foreach mbr ( 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/run +#set REFROOT = /global/cscratch1/sd/nanr/archive/v2.LR.historical_${mbr}/archive/rest/1991-01-01-00000/ +set REFROOT = /global/cscratch1/sd/nanr/archive/v2.LR.historical_${mbr}/archive/rest/1990-01-01-00000/ + +cd $CASEROOT/ +./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/ +./xmlchange JOB_QUEUE=debug --subgroup case.st_archive +./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive + +mv case.st_archive tmp.st_archive +cat tmp.st_archive | sed 's/101/1/' > case.st_archive + +cd $RUNROOT/ +cp $REFROOT/rpointer* . +ln -s $REFROOT/v2.*.nc . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/cp_user_nl_eam.csh b/scripts/v2.LR.SMBB/cp_user_nl_eam.csh new file mode 100755 index 0000000..0fafa2e --- /dev/null +++ b/scripts/v2.LR.SMBB/cp_user_nl_eam.csh @@ -0,0 +1,14 @@ +#!/bin/csh +### set env variables + +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/case_scripts + +cp /global/u2/n/nanr/CESM_tools/e3sm/v2/user_nl_files/HISTsmbb/user_nl_eam $CASEROOT/ + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/doh.sh b/scripts/v2.LR.SMBB/doh.sh new file mode 100755 index 0000000..2acc743 --- /dev/null +++ b/scripts/v2.LR.SMBB/doh.sh @@ -0,0 +1,33 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +echo ===== Start of v2.LR.SMBB_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +# for n in \ +# 0111 0121 0131 0141 0161 0171 0181 0191 \ +# 0211 0221 0231 +for n in \ +0241 0261 0271 0281 0291 0101 0151 0201 \ +0251 0301 9999 +do + + echo === Starting member ${n} === + + echo $n + + if [ "$n" == "9999" ] + then + echo "garbage" + + fi + echo ============================ + +done + diff --git a/scripts/v2.LR.SMBB/extractRESTARTS.scr b/scripts/v2.LR.SMBB/extractRESTARTS.scr new file mode 100644 index 0000000..c50afd4 --- /dev/null +++ b/scripts/v2.LR.SMBB/extractRESTARTS.scr @@ -0,0 +1,4 @@ +bash +source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh +zstash extract --hpss=/home/g/golaz/E3SMv2/v2.LR.historical_0131 "archive/rest/2015*" +zstash extract --hpss=/home/f/forsyth/E3SMv2/v2.LR.historical_0201 "archive/rest/1990*" diff --git a/scripts/v2.LR.SMBB/notNeeded/cp_restarts.scr b/scripts/v2.LR.SMBB/notNeeded/cp_restarts.scr new file mode 100755 index 0000000..a197aaf --- /dev/null +++ b/scripts/v2.LR.SMBB/notNeeded/cp_restarts.scr @@ -0,0 +1,15 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${mbr}/run + +cd $RUNROOT/ +cp $SCRATCH/archive/v2.LR.historical_${mbr}/archive/rest/1991-01-01-00000/rpointer* . +ln -s $SCRATCH/archive/v2.LR.historical_${mbr}/archive/rest/1991-01-01-00000/v2.LR* . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SMBB/run.v2.LR.SSP370-smbb_YYYY.sh b/scripts/v2.LR.SMBB/run.v2.LR.SSP370-smbb_YYYY.sh new file mode 100755 index 0000000..e96794d --- /dev/null +++ b/scripts/v2.LR.SMBB/run.v2.LR.SSP370-smbb_YYYY.sh @@ -0,0 +1,525 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) +array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="m4195" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370-smbb_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFCASE="v2.LR.historical-smbb_${iyr}" +RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-SMBB/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SMBB/run.v2.LR.historical-smbb_YYYY.sh b/scripts/v2.LR.SMBB/run.v2.LR.historical-smbb_YYYY.sh new file mode 100755 index 0000000..099b8b6 --- /dev/null +++ b/scripts/v2.LR.SMBB/run.v2.LR.historical-smbb_YYYY.sh @@ -0,0 +1,534 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear = ${refarray[$ctr]} +echo ${refyear} +ctr=$((ctr+1)) + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="m4195" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical-smbb_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2-SMBB/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="20150101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SMBB/t.sh b/scripts/v2.LR.SMBB/t.sh new file mode 100755 index 0000000..29c9258 --- /dev/null +++ b/scripts/v2.LR.SMBB/t.sh @@ -0,0 +1,26 @@ +#!/bin/bash -l + + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + #if [ "$n" == "0111" || "$n" == "0121" || "$n" == "0191" || "$n" == "0221" ]; then + if [ "$n" == "0111" ] | [ "$n" == "0121" ]; then + echo "first" + else + echo "second" + fi + echo "smbb" + +done + diff --git a/scripts/v2.LR.SMBB/test.sh b/scripts/v2.LR.SMBB/test.sh new file mode 100755 index 0000000..45d38a8 --- /dev/null +++ b/scripts/v2.LR.SMBB/test.sh @@ -0,0 +1,33 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +set ctr=0 +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +#main() { + +# Year array YYYY: + +echo "Doing it!" + +echo ${iyr} +echo ${refarray[$ctr]} +ctr=$((ctr+1)) +#} +done diff --git a/scripts/v2.LR.SMBB/testmeagain.sh b/scripts/v2.LR.SMBB/testmeagain.sh new file mode 100755 index 0000000..21f25c0 --- /dev/null +++ b/scripts/v2.LR.SMBB/testmeagain.sh @@ -0,0 +1,69 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 ) + array=( 0241 0261 0271 0281 0291 0101 0151 0201 0251 0301 ) +refarray=( 1991 1991 1991 1991 1991 1990 1990 1990 1990 1990 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set ctr=0 + +echo "garbage" + +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + + +# Year array YYYY: + +echo ${iyr} +echo ${refarray[$ctr]} +set refyear=${refarray[$ctr]} +echo "here is the ${refarray[$ctr]}" + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="m4195" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCL20TR" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.historical-smbb_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="${refarray[$ctr]}-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +REFYEAR=${iyr} +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/${refarray[$ctr]}-01-01-00000" +echo $RUN_REFDIR +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="${refarray[$ctr]}-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' +echo $RUN_REFDATE +ctr=$((ctr+1)) + +done diff --git a/scripts/v2.LR.SMBB/toss/v2.LR.SMBB_bundle.sh b/scripts/v2.LR.SMBB/toss/v2.LR.SMBB_bundle.sh new file mode 100755 index 0000000..43c795e --- /dev/null +++ b/scripts/v2.LR.SMBB/toss/v2.LR.SMBB_bundle.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.LE +#SBATCH --account=mp9 +#SBATCH --nodes=1618 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.LE.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.LE_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0241 0261 0271 0281 0291 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP + . ./v2.LR.LE_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +. ./v2.LR.LE_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.LE_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-1.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-1.sh new file mode 100755 index 0000000..d012cc9 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-1.sh @@ -0,0 +1,111 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1215 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.2-SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0241 0261 0271 0281 0291 0101 0151 0201 \ +0251 0301 9999 0480 +do + + echo === Starting member ${n} === + if [ "$n" == "0480" ]; then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="5" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "0480.002" + elif [ "$n" == "9999" ]; then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv2/v2.1.LR.1pctCO2_EXTEND.001 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "1PCT-EXTND" + else + echo "smbb" + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + fi + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-LastSubmit.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-LastSubmit.sh new file mode 100755 index 0000000..a57db9a --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus-LastSubmit.sh @@ -0,0 +1,99 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + if [ "$n" == "0131" ]; then + ./xmlchange STOP_N="8" + elif [ "$n" == "0161" ] | [ "$n" == "0171" ] | [ "$n" == "0181" ] | [ "$n" == "0211" ] | [ "$n" == "0231" ]; then + ./xmlchange STOP_N="7" + else + ./xmlchange STOP_N="10" + fi + echo "smbb" + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.TEST.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.TEST.sh new file mode 100755 index 0000000..c618fe7 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.TEST.sh @@ -0,0 +1,49 @@ +#!/bin/bash -l + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0480 0320 +do + + echo === Starting member ${n} === + + echo === Starting member ${n} === + if [ "$n" == "0480" ]; then + #cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + #./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + #./xmlchange STOP_N="5" + #./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + #PID=$! + #echo $PID + echo "0480.002" + + elif [ "$n" == "0320" ]; then + #cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 + #./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + #./xmlchange STOP_N="5" + #./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + #PID=$! + #echo $PID + echo "0320.003" + + else + echo "smbb" + #cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + #./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + #./xmlchange STOP_N="10" + #./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + #PID=$! + #echo $PID + fi + + echo ============================ + +done + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.sh new file mode 100755 index 0000000..0569acd --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle-Plus.sh @@ -0,0 +1,113 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1315 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0480 0320 +do + + echo === Starting member ${n} === + if [ "$n" == "0480" ]; then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="5" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "0480.002" + + elif [ "$n" == "0320" ]; then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="5" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "0320.003" + + else + echo "smbb" + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + fi + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle.sh new file mode 100644 index 0000000..7f7dbd9 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle.sh @@ -0,0 +1,97 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SMBB_SSP370_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + #if (${n} != 0221 ) then + # ./xmlchange STOP_N="4" + #else + # ./xmlchange STOP_N="5" + #fi + + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle2.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle2.sh new file mode 100755 index 0000000..9327da1 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle2.sh @@ -0,0 +1,98 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.2-SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0241 0261 0271 0281 0291 0101 0151 0201 \ +0251 0301 0141 +do + + echo === Starting member ${n} === + echo "smbb" + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + if [ "$n" == "0141" ]; then + ./xmlchange STOP_N="6" + else + ./xmlchange STOP_N="10" + fi + + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle3.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle3.sh new file mode 100755 index 0000000..7a92f0e --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle3.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1416 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.2-SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0131 0141 0191 0231 0241 0261 0271 \ +0281 0291 0101 0151 0201 0251 0301 +do + + echo === Starting member ${n} === + echo "smbb" + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle4.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle4.sh new file mode 100755 index 0000000..c0baa27 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_SSP370_bundle4.sh @@ -0,0 +1,101 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SSP370-SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.2-SSP.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SSP370_Plus.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0241 0261 0271 \ +0281 0291 0101 0151 0201 0251 0301 0999 +do + + if [ "$n" == "0999" ]; then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv2/v2.1.LR.1pctCO2_EXTEND.001 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "1PCT-EXTND" + else + echo === Starting member ${n} === + echo "smbb" + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + fi + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_extra.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_extra.sh new file mode 100644 index 0000000..530b9b5 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_extra.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +# v2.LR.historical-smbb_0101.eam.r.2014-01-01-00000.nc +# v2.LR.historical-smbb_0151.eam.r.2014-01-01-00000.nc +# v2.LR.historical-smbb_0201.eam.r.2014-01-01-00000.nc +# v2.LR.historical-smbb_0251.eam.r.2014-01-01-00000.nc +# v2.LR.historical-smbb_0301.eam.r.2014-01-01-00000.nc +# v2.LR.SSP370-smbb_0241 +# v2.LR.SSP370-smbb_0251 +# v2.LR.SSP370-smbb_0251 + +echo ======================================================== +#n="0131" +#echo Attempting to restart ${n} +#date +#cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SSP370-smbb_${n}/case_scripts +#./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " +#./xmlchange STOP_N="2" +########./xmlchange STOP_N="3" +#./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & +#PID=$! +#echo $PID +#echo ======================================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-LastSubmit.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-LastSubmit.sh new file mode 100644 index 0000000..ddecac1 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-LastSubmit.sh @@ -0,0 +1,97 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SMBB.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SMBB_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + if (${n} != 0221 ) then + ./xmlchange STOP_N="4" + else + ./xmlchange STOP_N="5" + fi + + #./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus-LastSubmit.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus-LastSubmit.sh new file mode 100644 index 0000000..e8c9ca3 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus-LastSubmit.sh @@ -0,0 +1,108 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.hist +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.hist.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SMBB_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0101 0151 0201 0241 0251 0261 0271 0281 0291 \ +0301 9999 +do + + echo === Starting member ${n} === + if [ "$n" == "9999" ] + then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv2/v2.1.LR.1pctCO2_EXTEND.001 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "1PCT-EXTND" + else + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + if (${n} == 0101 || ${n} == 0151 || ${n} == 0201 || ${n} == 0251 || ${n} == 0301 ) then + ./xmlchange STOP_N="5" + else + ./xmlchange STOP_N="4" + fi + + #./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + + fi + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus.sh new file mode 100644 index 0000000..6f42706 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle-Plus.sh @@ -0,0 +1,108 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.hist +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.hist.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SMBB_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +# for n in \ +# 0111 0121 0131 0141 0161 0171 0181 0191 \ +# 0211 0221 0231 +for n in \ +0241 0261 0271 0281 0291 0101 0151 0201 \ +0251 0301 0231 9999 +do + + echo === Starting member ${n} === + if [ "$n" == "9999" ] + then + cd /global/cfs/cdirs/ccsm1/people/nanr/cases/e3smv2/v2.1.LR.1pctCO2_EXTEND.001 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + echo "1PCT-EXTND" + else + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo "smbb" + + fi + + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle.sh b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle.sh new file mode 100644 index 0000000..fd8a9d2 --- /dev/null +++ b/scripts/v2.LR.SMBB/v2.LR.SMBB_historical_bundle.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.SMBB +#SBATCH --account=m4195 +#SBATCH --nodes=1113 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.SMBB.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.SMBB_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2-SMBB/v2.LR.historical-smbb_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB + . ./v2.LR.SMBB_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SMBB +. ./v2.LR.SMBB_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.SMBB_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SSP370-LE/README.bundle b/scripts/v2.LR.SSP370-LE/README.bundle new file mode 100644 index 0000000..f607833 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/README.bundle @@ -0,0 +1,12 @@ +Rough instructions +# set up runs +./run.v2.LR.SSP370_YYYY.sh + +# copy restarts, set EXEROOT +./change_JOB-QUEUE.csh + +# submit job +sbatch ./v2.LR.LE_bundle.sh + +# Reset CONTINUE_RUN if necessary (e.g., if I've already submitted the bundle but it fails) +./change_CONT_RUN.scr diff --git a/scripts/v2.LR.SSP370-LE/README.git b/scripts/v2.LR.SSP370-LE/README.git new file mode 100644 index 0000000..4d0b1e6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/README.git @@ -0,0 +1 @@ +https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/v2/ diff --git a/scripts/v2.LR.SSP370-LE/README.interactive b/scripts/v2.LR.SSP370-LE/README.interactive new file mode 100644 index 0000000..b70294b --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/README.interactive @@ -0,0 +1,32 @@ +salloc --nodes 1 --qos interactive --time 00:30:00 --constraint knl + + +sprio -j + + +sbatch v2.LR.LE_bundle.sh + + +NEED format for --last-date +cd //case_scripts +./case.st_archive --last-date 0051-01-01 --force-move --no-incomplete-logs +cd //archive +ls + + + +# check what jobs are running +squeue --job=${BUNDLE_ID} --steps +squeue --job=58091257 --steps + +# Restart failed jobs +## watch output of current bundle log +cd /global/cscratch1/sd/nanr/E3SMv2 +tail -F v2.LR.LE.o58091257 + +# Open a second window +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +edit v2.LR.LE_extra.sh + edit the file to restart the failed node. +>>>>> DANGER: this file is read ever 5 minutes so as soon as I see the tail of the log file record that it's restarting, then +immediately comment out these likes in the v2.LR.LE_extra.sh file and save so that it isn't read again and resubmits. diff --git a/scripts/v2.LR.SSP370-LE/README.postprocess-Chris b/scripts/v2.LR.SSP370-LE/README.postprocess-Chris new file mode 100644 index 0000000..eddc88b --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/README.postprocess-Chris @@ -0,0 +1,52 @@ +https://e3sm-project.github.io/zstash/_build/html/master/best_practices.html#best-practices-for-e3sm + + +bash +source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh +zppy -c post.v2.LR.SSP370_0111.cfg + + + +Check all status files for 'OK' message. + +Interactive: + +remember cori node +cori03 +bash +screen +n/nanr% salloc --nodes 1 --qos interactive --time 04:00:00 --constraint haswell + +salloc: Pending job allocation 61191352 +salloc: job 61191352 queued and waiting for resources +salloc: job 61191352 has been allocated resources +salloc: Granted job allocation 61191352 +salloc: Waiting for resource configuration +salloc: Nodes nid00219 are ready for job + + +run interactively +cd /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0111/post/scripts +screen +post/scripts% bash +nanr@nid00219:/global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0111/post/scripts> bash zppy_0111.bash 2>&1 | tee zppy_0111.o${SLURM_JOBID} + +detach from screen +exit +exit + + ssh dtn01.nersc.gov + screen -r + bash + cd /global/cscratch1/sd/golaz/E3SM/simulations/20180129.DECKv1b_piControl.ne30_oEC.edison + zstash create --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \ + --maxsize 128 . 2>&1 | tee zstash_create_20190226.log + +$ ssh dtn01.nersc.gov +$ screen -r +bash +source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh +$ cd /global/cscratch1/sd/nanr/checkZstash/365 +$ zstash check --hpss=2018/E3SM_simulations/20180129.DECKv1b_piControl.ne30_oEC.edison \ + --exclude="archive/rest/???[!05]-*/" \ + --maxsize 128 . 2>&1 | tee zstash_create_20190226.log diff --git a/scripts/v2.LR.SSP370-LE/README.rerun-111 b/scripts/v2.LR.SSP370-LE/README.rerun-111 new file mode 100644 index 0000000..5509987 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/README.rerun-111 @@ -0,0 +1,4 @@ +cori11:archive/ocn/hist>ls -Fl v2.LR.SSP370_0111.mpaso.hist.am.timeSeriesStatsMonthly.202* | egrep -v "4066581028" +-rwxr-xr-- 1 nanr mp9 3080454144 Jun 30 2022 v2.LR.SSP370_0111.mpaso.hist.am.timeSeriesStatsMonthly.2021-07-01.nc* +-rwxr-xr-- 1 nanr mp9 3016228864 Jun 30 2022 v2.LR.SSP370_0111.mpaso.hist.am.timeSeriesStatsMonthly.2022-11-01.nc* +-rwxr-xr-- 1 nanr mp9 3328966656 Jun 30 2022 v2.LR.SSP370_0111.mpaso.hist.am.timeSeriesStatsMonthly.2022-12-01.nc* diff --git a/scripts/v2.LR.SSP370-LE/badNode.141 b/scripts/v2.LR.SSP370-LE/badNode.141 new file mode 100644 index 0000000..772c486 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/badNode.141 @@ -0,0 +1,2 @@ +5120: PIO: FATAL ERROR: Aborting... FATAL ERROR: No such file or directory (/global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/externals/scorpio/src/clib/pioc_support.c: 3128) +nid05065 3128 diff --git a/scripts/v2.LR.SSP370-LE/change_CONT_RUN.scr b/scripts/v2.LR.SSP370-LE/change_CONT_RUN.scr new file mode 100755 index 0000000..27b357e --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/change_CONT_RUN.scr @@ -0,0 +1,15 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange CONTINUE_RUN=FALSE +./xmlchange DOUT_S=FALSE + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SSP370-LE/change_DOUT.scr b/scripts/v2.LR.SSP370-LE/change_DOUT.scr new file mode 100755 index 0000000..a29972c --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/change_DOUT.scr @@ -0,0 +1,14 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/case_scripts + +cd $CASEROOT/ +./xmlchange DOUT_S=FALSE + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SSP370-LE/change_JOB-QUEUE.csh b/scripts/v2.LR.SSP370-LE/change_JOB-QUEUE.csh new file mode 100755 index 0000000..d227dff --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/change_JOB-QUEUE.csh @@ -0,0 +1,25 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/case_scripts +set RUNROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/run +set REFROOT = /global/cscratch1/sd/nanr/archive/v2.LR.historical_${mbr}/archive/rest/2015-01-01-00000/ + +cd $CASEROOT/ +./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/ +./xmlchange JOB_QUEUE=debug --subgroup case.st_archive +./xmlchange JOB_WALLCLOCK_TIME=00:30:00 --subgroup case.st_archive + +mv case.st_archive tmp.st_archive +cat tmp.st_archive | sed 's/101/1/' > case.st_archive + +cd $RUNROOT/ +cp $REFROOT/rpointer* . +ln -s $REFROOT/v2.*.nc . + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SSP370-LE/countFiles.scr b/scripts/v2.LR.SSP370-LE/countFiles.scr new file mode 100755 index 0000000..901b551 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/countFiles.scr @@ -0,0 +1,24 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +foreach dec ( 202 203 204 205 206 207 ) +#foreach mbr ( 0231 ) +#foreach dec ( 202 203 204 205 206 207 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/archive/atm/hist/ +set DESTROOT = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_${mbr}/archive/atm/hist/ + +cd $CASEROOT/ +echo scratch $mbr and $dec +ls -l *.$dec* | wc +cd $DESTROOT/ +echo cfs $mbr and $dec +ls -l *.$dec* | wc +echo '==================' + +end # dec loop +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SSP370-LE/doh b/scripts/v2.LR.SSP370-LE/doh new file mode 100755 index 0000000..079bcde --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/doh @@ -0,0 +1,6 @@ +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0271/case_scripts +./case.st_archive --last-date 2065-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0281/case_scripts +./case.st_archive --last-date 2065-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0291/case_scripts +./case.st_archive --last-date 2034-01-01 --force-move --no-incomplete-logs; diff --git a/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh b/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh new file mode 100755 index 0000000..b896dca --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh @@ -0,0 +1,486 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment line below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="WCYCLSSP370" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_0101.t05" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20211103" +readonly BRANCH="master" +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" # for SSP370 this is set automatically via the compset config + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/jjbenedi/ssp370/rest_for_hybrid/v2.LR.historical_0101/init/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_0101" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' +#readonly RUN_REFDIR="/global/cscratch1/sd/forsyth/E3SMv2/v2.LR.piControl/init" +#readonly RUN_REFCASE="20210625.v2rc3c-GWD.piControl.ne30pg2_EC30to60E2r2.chrysalis" +#readonly RUN_REFDATE="1001-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly CODE_ROOT="${HOME}/cori/model/E3SMv2_master_ssp370_${CHECKOUT}" +#readonly CASE_ROOT="${CODE_ROOT}/cases/${CASE_NAME}" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/bld +#readonly CASE_ARCHIVE_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/archive +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_2x5_ndays' +#readonly run='M_1x10_ndays' +#readonly run='L_1x10_ndays' +readonly run='production' + +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="02:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="0" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +#cat << EOF >> user_nl_eam +# nhtfrq = 0,-24,-6,-6,-3,-24,0 +# mfilt = 1,30,120,120,240,30,1 +# avgflag_pertape = 'A','A','I','A','A','A','I' +# fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' +# fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' +# fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' +# fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' +# fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' +# fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' +# fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' +# fincl7 = 'O3', 'PS', 'TROP_P' +# +#! Additional retuning +# clubb_tk1 = 268.15D0 +# gw_convect_hcf = 10.0 +#EOF + +cat << EOF >> user_nl_elm + do_transient_pfts = .true. + check_finidat_fsurdat_consistency = .false. + check_finidat_year_consistency = .true. + check_finidat_pct_consistency = .true. + check_dynpft_consistency = .false. + hist_dov2xy = .true. + hist_fincl1 = 'PCT_NAT_PFT' + hist_mfilt = 1 + hist_nhtfrq = 0 + hist_avgflag_pertape = 'A' +EOF + +#cat << EOF >> user_nl_elm +# hist_dov2xy = .true.,.true. +# hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' +# hist_mfilt = 1,365 +# hist_nhtfrq = 0,-24 +# hist_avgflag_pertape = 'A','A' +#EOF + +#cat << EOF >> user_nl_mosart +# rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' +# rtmhist_mfilt = 1,365 +# rtmhist_ndens = 2 +# rtmhist_nhtfrq = 0,-24 +#EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh_original b/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh_original new file mode 100755 index 0000000..b896dca --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/fromJimBenedict/run_e3sm.v2.LR.SSP370_0101.t05.sh_original @@ -0,0 +1,486 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment line below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="WCYCLSSP370" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_0101.t05" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20211103" +readonly BRANCH="master" +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" # for SSP370 this is set automatically via the compset config + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/jjbenedi/ssp370/rest_for_hybrid/v2.LR.historical_0101/init/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_0101" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' +#readonly RUN_REFDIR="/global/cscratch1/sd/forsyth/E3SMv2/v2.LR.piControl/init" +#readonly RUN_REFCASE="20210625.v2rc3c-GWD.piControl.ne30pg2_EC30to60E2r2.chrysalis" +#readonly RUN_REFDATE="1001-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly CODE_ROOT="${HOME}/cori/model/E3SMv2_master_ssp370_${CHECKOUT}" +#readonly CASE_ROOT="${CODE_ROOT}/cases/${CASE_NAME}" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/bld +#readonly CASE_ARCHIVE_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/archive +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_2x5_ndays' +#readonly run='M_1x10_ndays' +#readonly run='L_1x10_ndays' +readonly run='production' + +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="02:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="0" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=true + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +#cat << EOF >> user_nl_eam +# nhtfrq = 0,-24,-6,-6,-3,-24,0 +# mfilt = 1,30,120,120,240,30,1 +# avgflag_pertape = 'A','A','I','A','A','A','I' +# fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' +# fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' +# fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' +# fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' +# fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' +# fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' +# fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' +# fincl7 = 'O3', 'PS', 'TROP_P' +# +#! Additional retuning +# clubb_tk1 = 268.15D0 +# gw_convect_hcf = 10.0 +#EOF + +cat << EOF >> user_nl_elm + do_transient_pfts = .true. + check_finidat_fsurdat_consistency = .false. + check_finidat_year_consistency = .true. + check_finidat_pct_consistency = .true. + check_dynpft_consistency = .false. + hist_dov2xy = .true. + hist_fincl1 = 'PCT_NAT_PFT' + hist_mfilt = 1 + hist_nhtfrq = 0 + hist_avgflag_pertape = 'A' +EOF + +#cat << EOF >> user_nl_elm +# hist_dov2xy = .true.,.true. +# hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' +# hist_mfilt = 1,365 +# hist_nhtfrq = 0,-24 +# hist_avgflag_pertape = 'A','A' +#EOF + +#cat << EOF >> user_nl_mosart +# rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' +# rtmhist_mfilt = 1,365 +# rtmhist_ndens = 2 +# rtmhist_nhtfrq = 0,-24 +#EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/newRuns b/scripts/v2.LR.SSP370-LE/newRuns new file mode 100644 index 0000000..3a6f03e --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/newRuns @@ -0,0 +1,98 @@ +0131 + +----- Skipping fetch_code ----- + + +----- Starting create_newcase ----- + +Compset longname is SSP370SOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV +Compset specification file is /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/cime_config/allactive/config_compsets.xml +Automatically adding SIAC to compset +Automatically adding SESP to compset +Compset forcing is +ATM component is cam5 physics:EAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- CMIP6-DECK: +LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: +ICE component is MPAS seaice: +OCN component is mpas ocean default: +ROF component is MOSART: +GLC component is Stub glacier (land ice) component +WAV component is Stub wave component +IAC component is Stub iac component +ESP component is Stub external system processing (ESP) component +Pes specification file is /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/cime_config/allactive/config_pesall.xml +Compset specific settings: name is RUN_STARTDATE and value is 2015-01-01 +Machine is cori-knl +Pes setting: grid match is a%ne30np4.pg.+_oi%EC30to60E2r2 +Pes setting: machine match is cori-knl +Pes setting: compset_match is .*EAM.+ELM.+MPASSI.+MPASO.+MOSART.+ +Pes setting: pesize match is L +Pes setting: grid is a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 +Pes setting: compset is SSP370SOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP +Pes setting: tasks is {'NTASKS_ATM': 5440, 'NTASKS_LND': 192, 'NTASKS_ROF': 128, 'NTASKS_ICE': 5120, 'NTASKS_OCN': 1024, 'NTASKS_CPL': 5440} +Pes setting: threads is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1} +Pes setting: rootpe is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 5120, 'ROOTPE_ROF': 5312, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 5440, 'ROOTPE_CPL': 0} +Pes setting: pstrid is {} +Pes other settings: {'MAX_MPITASKS_PER_NODE': '64', 'MAX_TASKS_PER_NODE': '272'} +Pes comments: -compset A_WCYCL* -res ne30pg*EC30to60* v2LR 101 nodes n101a64x1b ~6.8 sypd +setting additional fields from config_pes: {'MAX_MPITASKS_PER_NODE': '64', 'MAX_TASKS_PER_NODE': '272'} + Compset is: SSP370SOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP + Grid is: a%ne30np4.pg2_l%ne30np4.pg2_oi%EC30to60E2r2_r%r05_g%null_w%null_z%null_m%EC30to60E2r2 + Components in compset are: ['eam', 'elm', 'mpassi', 'mpaso', 'mosart', 'sglc', 'swav', 'siac', 'sesp'] +No charge_account info available, using value from PROJECT +e3sm model version found: b80567211d +Batch_system_type is nersc_slurm +job is case.run USER_REQUESTED_WALLTIME 48:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S + Creating Case directory /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0131/case_scripts + +----- Starting case_setup ----- + + +Configuring E3SM to use the COSP simulator + +Successfully cleaned env_mach_specific.xml +job is case.run USER_REQUESTED_WALLTIME 48:00:00 USER_REQUESTED_QUEUE None WALLTIME_FORMAT %H:%M:%S +Creating batch scripts +Writing case.run script from input template /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/cime_config/machines/template.case.run +Creating file .case.run +Writing case.st_archive script from input template /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/cime_config/machines/template.st_archive +Creating file case.st_archive +Creating file .case.run.sh +Creating user_nl_xxx files for components and cpl +If an old case build already exists, might want to run 'case.build --clean' before building +Generating component namelists as part of setup + 2022-06-14 07:21:19 atm +Create namelist for component eam + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/eam//cime_config/buildnml + 2022-06-14 07:21:20 lnd +Create namelist for component elm + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/elm//cime_config/buildnml + 2022-06-14 07:21:21 ice +Create namelist for component mpassi + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/mpas-seaice/cime_config/buildnml + 2022-06-14 07:21:21 ocn +Create namelist for component mpaso + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/mpas-ocean/cime_config/buildnml + 2022-06-14 07:21:22 rof +Create namelist for component mosart + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/mosart//cime_config/buildnml + 2022-06-14 07:21:22 glc +Create namelist for component sglc + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/stub_comps/sglc/cime_config/buildnml + 2022-06-14 07:21:22 wav +Create namelist for component swav + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/stub_comps/swav/cime_config/buildnml + 2022-06-14 07:21:22 iac +Create namelist for component siac + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/stub_comps/siac/cime_config/buildnml + 2022-06-14 07:21:22 esp +Create namelist for component sesp + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/components/stub_comps/sesp/cime_config/buildnml + 2022-06-14 07:21:22 cpl +Create namelist for component drv + Calling /global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM/driver-mct/cime_config/buildnml +You can now run './preview_run' to get more info on how your case will be run + +----- case_build ----- + +ERROR: $OLD_EXECUTABLE = /global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe does not exist or is not an executable file. diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0111.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0111.cfg new file mode 100755 index 0000000..6038797 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0111.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0111 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0111 +case = v2.LR.SSP370_0111 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0111" + +[bundle] + + [[ zppy_0111 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0111' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0111" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0111" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0121.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0121.cfg new file mode 100755 index 0000000..94fc0b9 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0121.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0121 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0121 +case = v2.LR.SSP370_0121 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0121" + +[bundle] + + [[ zppy_0121 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0121' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0121" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0121" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0131.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0131.cfg new file mode 100755 index 0000000..d4c8ae5 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0131.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0131 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0131 +case = v2.LR.SSP370_0131 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0131" + +[bundle] + + [[ zppy_0131 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0131' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0131" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0131" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0141.cfg-pending b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0141.cfg-pending new file mode 100755 index 0000000..3eaf880 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0141.cfg-pending @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0141 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0141 +case = v2.LR.SSP370_0141 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0141" + +[bundle] + + [[ zppy_0141 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0141' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0141" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0141" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg new file mode 100755 index 0000000..14eddc6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0161 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0161 +case = v2.LR.SSP370_0161 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0161" + +[bundle] + + [[ zppy_0161 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0161' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0161" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0161" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg-pending b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg-pending new file mode 100755 index 0000000..14eddc6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0161.cfg-pending @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0161 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0161 +case = v2.LR.SSP370_0161 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0161" + +[bundle] + + [[ zppy_0161 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0161' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0161" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0161" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0171.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0171.cfg new file mode 100755 index 0000000..feb661c --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0171.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0171 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0171 +case = v2.LR.SSP370_0171 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0171" + +[bundle] + + [[ zppy_0171 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0171' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0171" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0171" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0181.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0181.cfg new file mode 100755 index 0000000..698b219 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0181.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0181 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0181 +case = v2.LR.SSP370_0181 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0181" + +[bundle] + + [[ zppy_0181 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0181' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0181" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0181" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0211.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0211.cfg new file mode 100755 index 0000000..d280be9 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0211.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0211 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0211 +case = v2.LR.SSP370_0211 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0211" + +[bundle] + + [[ zppy_0211 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0211' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0211" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0211" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0301.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0301.cfg new file mode 100755 index 0000000..ac0c07a --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.SSP370_0301.cfg @@ -0,0 +1,116 @@ +[default] +input = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0301 +output = /global/cfs/cdirs/mp9/archive/E3SMv2/v2.LR.SSP370_0301 +case = v2.LR.SSP370_0301 +www = /global/cfs/cdirs/mp9/www/nanr/E3SMv2/ +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" +bundle = "zppy_0301" + +[bundle] + + [[ zppy_0301 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +walltime = "6:00:00" +years = "2071:2100:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "2015-2100","2071-2100" + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "2071:2100:30", +ts_num_years = 30 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.SSP370_0301' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = 'diurnal_8xdaily' + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 2015 +ts_years = "2015-2100", +enso_years = "2015-2100", +climo_years = "2071-2100", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "2015-2100", +ts_num_years = 86 +figstr= "v2_SSP370_0301" +moc_file = "mocTimeSeries_2015-2100.nc" +experiment_name = "v2.LR.SSP370_0301" +ts_years = "2015-2100", +climo_years = "2071-2100", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.LR.historical_0111.cfg b/scripts/v2.LR.SSP370-LE/post.v2.LR.historical_0111.cfg new file mode 100644 index 0000000..acd6687 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.LR.historical_0111.cfg @@ -0,0 +1,106 @@ +[default] +input = /global/cfs/cdirs/e3sm/golaz/E3SMv2/v2.LR.historical_0111 +output = /global/cfs/cdirs/e3sm/golaz/E3SMv2/v2.LR.historical_0111 +case = v2.LR.historical_0111 +www = /global/cfs/cdirs/e3sm/www/golaz/E3SMv2 +partition = haswell +environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh" + +[climo] +active = True +walltime = "6:00:00" +years = "1865:2014:30", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + frequency = "monthly" + + [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + frequency = "monthly_diurnal_8xdaily" + +[ts] +active = True +years = "1850:2014:5", + + [[ atm_monthly_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + + [[ atm_daily_180x360_aave ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "PRECT" + + [[ atm_monthly_glb ]] + input_subdir = "archive/atm/hist" + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + vars = "FSH,RH2M" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +years = "1865:2014:30", +ts_num_years = 5 +ref_start_yr = 1985 +ref_final_yr = 2014 + + [[ atm_monthly_180x360_aave ]] + short_name = 'v2.LR.historical_0111' + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' + dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' + climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" + output_format_subplot = "pdf", + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +walltime = "6:00:00" +parallelTaskCount = 6 +anomalyRefYear = 1850 +ts_years = "1850-1894", "1850-1924", "1850-1954", "1850-1984", "1850-2014", +enso_years = "1850-1894", "1850-1924", "1850-1954", "1850-1984", "1850-2014", +climo_years = "1865-1894", "1895-1924", "1925-1954", "1955-1984", "1985-2014", +mesh = "EC30to60E2r2" + +[global_time_series] +active = True +years = "1850-2014", +ts_num_years = 5 +figstr= "v2_historical_0111" +moc_file = "mocTimeSeries_1850-2014.nc" +experiment_name = "v2.LR.historical_0111" +ts_years = "1850-2014", +climo_years = "1985-2014", + diff --git a/scripts/v2.LR.SSP370-LE/post.v2.NARRM.historical_0151.cfg b/scripts/v2.LR.SSP370-LE/post.v2.NARRM.historical_0151.cfg new file mode 100644 index 0000000..7326db6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/post.v2.NARRM.historical_0151.cfg @@ -0,0 +1,125 @@ +[default] +input = /global/cfs/projectdirs/e3sm/tang30/E3SMv2/v2.NARRM.historical_0151 +input_subdir = archive/atm/hist +output = /global/cfs/projectdirs/e3sm/golaz/E3SMv2/v2.NARRM.historical_0151 +case = v2.NARRM.historical_0151 +www = /global/cfs/cdirs/e3sm/www/golaz/E3SMv2 +environment_commands = "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.7.0rc3_cori-haswell.sh" +partition = haswell +bundle = "zppy_0151" + +[bundle] + + [[ zppy_0151 ]] + nodes = 1 + walltime = "24:00:00" + + +[climo] +active = True +years = "1985-2014", + + [[ atm_monthly_180x360_aave ]] + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc + frequency = "monthly" + + [[ atm_monthly_conus_0.23x0.31_aave ]] + grid = 'conus_0.23x0.31_aave' + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_conus_0.23x0.31_nco.20210301.nc + frequency = "monthly" + + [[ atm_diurnal_8xdaily_180x360_aave ]] + input_files = "eam.h4" + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc + vars = "PRECT" + frequency = "diurnal_8xdaily" + +[ts] +active = True +years = "1850:2014:5", + + [[ atm_monthly_180x360_aave ]] + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc + + [[ atm_daily_180x360_aave ]] + input_files = "eam.h1" + frequency = "daily" + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc + vars = "PRECT" + + [[ atm_monthly_conus_0.23x0.31_aave ]] + input_files = "eam.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_conus_0.23x0.31_nco.20210301.nc + + [[ atm_monthly_glb ]] + input_files = "eam.h0" + frequency = "monthly" + mapping_file = "glb" + + [[ land_monthly ]] + input_subdir = "archive/lnd/hist" + input_files = "elm.h0" + frequency = "monthly" + mapping_file = /global/homes/z/zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc + vars = "FSH,RH2M" + extra_vars = "landfrac" + + [[ rof_monthly ]] + input_subdir = "archive/rof/hist" + input_files = "mosart.h0" + frequency = "monthly" + mapping_file = "" + vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" + extra_vars = 'areatotal2' + +[e3sm_diags] +active = True +reference_data_path = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology' +obs_ts = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/time-series' +dc_obs_climo = '/global/cfs/cdirs/e3sm/e3sm_diags/obs_for_e3sm_diags/climatology/' +output_format_subplot = "pdf", +years = "1985-2014", +ts_num_years = 5 +ref_start_yr = 1985 +ref_final_yr = 2014 +short_name = 'v2.NARRM.historical_0151' + + [[ atm_monthly_180x360_aave ]] + grid = '180x360_aave' + sets = 'lat_lon','zonal_mean_xy','zonal_mean_2d','polar','cosp_histogram','meridional_mean_2d','enso_diags','qbo','diurnal_cycle','annual_cycle_zonal_mean','streamflow','zonal_mean_2d_stratosphere' + climo_diurnal_subsection = "atm_diurnal_8xdaily_180x360_aave" + climo_diurnal_frequency = "diurnal_8xdaily" + + [[ atm_monthly_conus_0.23x0.31_aave ]] + grid = 'conus_0.23x0.31_aave' + sets = "lat_lon", + cfg = "e3sm_diags/lat_lon_conus.cfg" + +[e3sm_diags_vs_model] +active = False + +[amwg] +active = False + +[mpas_analysis] +active = True +parallelTaskCount = 6 +anomalyRefYear = 1850 +ts_years = "1850-2014", +enso_years = "1850-2014", +climo_years = "1985-2014", +mesh = "WC14to60E2r3" + +[global_time_series] +active = True +years = "1850-2014", +ts_num_years = 5 +figstr= "v2.NARRM.historical_0151" +moc_file = "mocTimeSeries_1850-2014.nc" +experiment_name = "v2.NARRM.historical_0151" +ts_years = "1850-2014", +climo_years = "1985-2014", + diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh new file mode 100755 index 0000000..9537ac2 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0111" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh-notUsed b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh-notUsed new file mode 100755 index 0000000..2c6a1fe --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0111.sh-notUsed @@ -0,0 +1,487 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_0111" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_0111/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_0111" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh new file mode 100755 index 0000000..46e19a7 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh @@ -0,0 +1,494 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0121" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +#readonly OLD_EXECUTABLE="" +readonly OLD_EXECUTABLE=" /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + ./xmlchange BUILD_COMPLETE=TRUE + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh-notUsed b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh-notUsed new file mode 100755 index 0000000..6fd6cb6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0121.sh-notUsed @@ -0,0 +1,488 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0121" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0131.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0131.sh new file mode 100755 index 0000000..7a93048 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0131.sh @@ -0,0 +1,493 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0131" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + ./xmlchange BUILD_COMPLETE=TRUE + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0141.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0141.sh new file mode 100755 index 0000000..471694f --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0141.sh @@ -0,0 +1,493 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0141" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + ./xmlchange BUILD_COMPLETE=TRUE + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0161.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0161.sh new file mode 100755 index 0000000..91626cd --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0161.sh @@ -0,0 +1,493 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0161" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + #./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + ./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + ./xmlchange BUILD_COMPLETE=TRUE + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + #exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0171.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0171.sh new file mode 100755 index 0000000..4f5b75c --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0171.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0171" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0181.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0181.sh new file mode 100755 index 0000000..28d60d2 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0181.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0181" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0191.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0191.sh new file mode 100755 index 0000000..70515f4 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0191.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0191" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0211.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0211.sh new file mode 100755 index 0000000..ea62d84 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0211.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0211" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0221.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0221.sh new file mode 100755 index 0000000..9b4bb26 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0221.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0221" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + #readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="5" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0231.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0231.sh new file mode 100755 index 0000000..8b6b418 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0231.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0231" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0241.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0241.sh new file mode 100755 index 0000000..bbb040d --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0241.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0241" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0261.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0261.sh new file mode 100755 index 0000000..65b7d27 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0261.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0261" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0271.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0271.sh new file mode 100755 index 0000000..d28747f --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0271.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0271" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0281.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0281.sh new file mode 100755 index 0000000..5e3abc6 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0281.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0281" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0291.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0291.sh new file mode 100755 index 0000000..8b6b418 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_0291.sh @@ -0,0 +1,489 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" +readonly YYYY="0231" + +# Simulation +readonly COMPSET="WCYCLSSP370" # 20th century transient +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_${YYYY}" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${YYYY}/archive/rest/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_${YYYY}" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/build +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="0:30:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" # How often to stop the model, should be a multiple of REST_N + readonly STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" # How often to write a restart file + readonly RESUBMIT="0" # Submissions after initial one + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="1" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_5mbrs.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_5mbrs.sh new file mode 100755 index 0000000..ba4fa89 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_5mbrs.sh @@ -0,0 +1,503 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0101 0201 0301 0401 0501 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override + check_finidat_fsurdat_consistency = .false. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-0111rerun.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-0111rerun.sh new file mode 100755 index 0000000..8435da5 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-0111rerun.sh @@ -0,0 +1,512 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 ) +#array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-5mbrs.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-5mbrs.sh new file mode 100755 index 0000000..759a54c --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY-5mbrs.sh @@ -0,0 +1,510 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0101 0151 0201 0251 0301 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY.sh b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY.sh new file mode 100755 index 0000000..ae28f4b --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run.v2.LR.SSP370_YYYY.sh @@ -0,0 +1,511 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +array=( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +#array=( 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) +for iyr in "${array[@]}" +do + + +# For debugging, uncomment libe below +#set -x + +main() { + +# Year array YYYY: + +echo ${iyr} + +# --- Configuration flags ---- + +# Machine and project +MACHINE=cori-knl +PROJECT="mp9" +#readonly YYYY="0141" +#readonly YYYY=${iyr} + +# Simulation +COMPSET="WCYCLSSP370" # 20th century transient +RESOLUTION="ne30pg2_EC30to60E2r2" +CASE_NAME="v2.LR.SSP370_${iyr}" +CASE_GROUP="v2.LR" + +# Code and compilation +CHECKOUT="20220412" +BRANCH="maint-2.0" # master as of 2021-12-21 +CHERRY=( ) +DEBUG_COMPILE=false + +# Run options +MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +START_DATE="2015-01-01" + +# Additional options for 'branch' and 'hybrid' +GET_REFCASE=TRUE +RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.historical_${iyr}/archive/rest/2015-01-01-00000" +RUN_REFCASE="v2.LR.historical_${iyr}" +RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_1x10_ndays' +#readonly run='S_1x1_nmonths' +#readonly run='M_1x1_nmonths' +#readonly run='L_1x1_nmonths' +#readonly run='XL_1x1_nmonths' +run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + PELAYOUT=${layout} + WALLTIME="0:30:00" + STOP_OPTION=${units} + STOP_N=${length} + STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION=${STOP_OPTION} + REST_N=${STOP_N} + RESUBMIT=${resubmit} + DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + CASE_RUN_DIR=${CASE_ROOT}/run + # nanr changes + #readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + #readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + # end nanr + PELAYOUT="L" + WALLTIME="48:00:00" + STOP_OPTION="nyears" + STOP_N="10" # How often to stop the model, should be a multiple of REST_N + STOP_DATE="21010101" # -999 or specify stop date as yyyyddmm without leading zeros + REST_OPTION="nyears" + REST_N="1" # How often to write a restart file + RESUBMIT="0" # Submissions after initial one + DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +HIST_OPTION="nyears" +HIST_N="1" + +# Leave empty (unless you understand what it does) +#OLD_EXECUTABLE="" +OLD_EXECUTABLE="/global/cscratch1/sd/nanr/E3SMv2/EXEROOT/build/e3sm.exe" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm +! Pointing to new simyr2015 file per Jim Benedict + fsurdat = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/surfdata_map/surfdata_ne30np4.pg2_SSP3_RCP70_simyr2015_c220420.nc' + + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + # nanr changes + #./xmlchange EXEROOT=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/build/ + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + fi + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + echo $'\n----- Preview namelists -----\n' + ./preview_namelists + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # End date + ./xmlchange STOP_DATE=${STOP_DATE} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + +done diff --git a/scripts/v2.LR.SSP370-LE/run_e3sm.v2.LR.SSP370_0101.t05.nanr.sh b/scripts/v2.LR.SSP370-LE/run_e3sm.v2.LR.SSP370_0101.t05.nanr.sh new file mode 100755 index 0000000..39386a5 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run_e3sm.v2.LR.SSP370_0101.t05.nanr.sh @@ -0,0 +1,492 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment line below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="e3sm" + +# Simulation +readonly COMPSET="WCYCLSSP370" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.SSP370_0101.t01" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20211103" +readonly BRANCH="master" +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="2015-01-01" # for SSP370 this is set automatically via the compset config + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/jjbenedi/ssp370/rest_for_hybrid/v2.LR.historical_0101/init/2015-01-01-00000" +readonly RUN_REFCASE="v2.LR.historical_0101" +readonly RUN_REFDATE="2015-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' +#readonly RUN_REFDIR="/global/cscratch1/sd/forsyth/E3SMv2/v2.LR.piControl/init" +#readonly RUN_REFCASE="20210625.v2rc3c-GWD.piControl.ne30pg2_EC30to60E2r2.chrysalis" +#readonly RUN_REFDATE="1001-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly CODE_ROOT="${HOME}/cori/model/E3SMv2_master_ssp370_${CHECKOUT}" +#readonly CODE_ROOT="/global/homes/j/jjbenedi/cori/model/E3SMv2_master_ssp370_${CHECKOUT}" +readonly CODE_ROOT="/global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/sandbox-SSP/E3SM" +#readonly CODE_ROOT="/global/project/projectdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2/E3SM" +#readonly CASE_ROOT="${CODE_ROOT}/cases/${CASE_NAME}" +#readonly CODE_ROOT="${HOME}/E3SMv2/code/${CHECKOUT}" +#readonly CASE_ROOT="/global/cscratch1/sd/${USER}/E3SMv2/${CASE_NAME}" +readonly CASE_ROOT="/global/project/projectdirs/ccsm1/people/nanr/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/bld +#readonly CASE_ARCHIVE_DIR=/global/cscratch1/sd/${USER}/e3sm_scratch/cori-knl/${CASE_NAME}/archive +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=/global/cscratch1/sd/nanr/${CASE_NAME}/bld +readonly CASE_ARCHIVE_DIR=/global/cscratch1/sd/nanr/archive/${CASE_NAME} + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +#readonly run='S_2x5_ndays' +#readonly run='M_1x10_ndays' +#readonly run='L_1x10_ndays' +readonly run='production' + +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="02:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="48:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="10" + readonly STOP_DATE="-999" # -999 or specify stop date as yyyyddmm without leading zeros + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="0" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=false +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +#cat << EOF >> user_nl_eam +# nhtfrq = 0,-24,-6,-6,-3,-24,0 +# mfilt = 1,30,120,120,240,30,1 +# avgflag_pertape = 'A','A','I','A','A','A','I' +# fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' +# fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' +# fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' +# fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' +# fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' +# fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' +# fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' +# fincl7 = 'O3', 'PS', 'TROP_P' +# +#! Additional retuning +# clubb_tk1 = 268.15D0 +# gw_convect_hcf = 10.0 +#EOF + +cat << EOF >> user_nl_elm + do_transient_pfts = .true. + check_finidat_fsurdat_consistency = .false. + check_finidat_year_consistency = .true. + check_finidat_pct_consistency = .true. + check_dynpft_consistency = .false. + hist_dov2xy = .true. + hist_fincl1 = 'PCT_NAT_PFT' + hist_mfilt = 1 + hist_nhtfrq = 0 + hist_avgflag_pertape = 'A' +EOF + +#cat << EOF >> user_nl_elm +# hist_dov2xy = .true.,.true. +# hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' +# hist_mfilt = 1,365 +# hist_nhtfrq = 0,-24 +# hist_avgflag_pertape = 'A','A' +#EOF + +#cat << EOF >> user_nl_mosart +# rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' +# rtmhist_mfilt = 1,365 +# rtmhist_ndens = 2 +# rtmhist_nhtfrq = 0,-24 +#EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main diff --git a/scripts/v2.LR.SSP370-LE/run_st_archive.scr b/scripts/v2.LR.SSP370-LE/run_st_archive.scr new file mode 100755 index 0000000..7fc2d14 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/run_st_archive.scr @@ -0,0 +1,19 @@ +#!/bin/csh +### set env variables + +foreach mbr ( 0111 0121 ) +#foreach mbr ( 0111 0121 0131 0141 0161 0171 0181 0191 0211 0221 0231 0241 0261 0271 0281 0291 ) + +set CASEROOT = /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${mbr}/case_scripts +if ($mbr == 0291) then + ./case.st_archive --last-date 2030-01-01 --force-move --no-incomplete-logs + wait +else + ./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs + wait +endif + +end # mbr loop + +exit + diff --git a/scripts/v2.LR.SSP370-LE/st.archive b/scripts/v2.LR.SSP370-LE/st.archive new file mode 100755 index 0000000..1bfb902 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/st.archive @@ -0,0 +1,32 @@ +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0121/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0131/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0141/case_scripts +./case.st_archive --last-date 2085-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0161/case_scripts +./case.st_archive --last-date 2070-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0171/case_scripts +./case.st_archive --last-date 2081-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0181/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0191/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +#cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0211/case_scripts +#./case.st_archive --last-date 2070-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0221/case_scripts +./case.st_archive --last-date 2080-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0231/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0241/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0261/case_scripts +./case.st_archive --last-date 2080-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0271/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0281/case_scripts +./case.st_archive --last-date 2091-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0291/case_scripts +./case.st_archive --last-date 2040-01-01 --force-move --no-incomplete-logs; diff --git a/scripts/v2.LR.SSP370-LE/st.archive2 b/scripts/v2.LR.SSP370-LE/st.archive2 new file mode 100755 index 0000000..142e03b --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/st.archive2 @@ -0,0 +1,32 @@ +#cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0111/case_scripts +#./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +#cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0121/case_scripts +#./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0131/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0141/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0161/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0171/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0181/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0191/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0211/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0221/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0231/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0241/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0261/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0271/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0281/case_scripts +./case.st_archive --last-date 2050-01-01 --force-move --no-incomplete-logs; +cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_0291/case_scripts +./case.st_archive --last-date 2030-01-01 --force-move --no-incomplete-logs; diff --git a/scripts/v2.LR.SSP370-LE/test.sh b/scripts/v2.LR.SSP370-LE/test.sh new file mode 100755 index 0000000..e80c945 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/test.sh @@ -0,0 +1,25 @@ +#!/bin/bash -l + +# Loop over members +#for n in \ +#0111 0121 0131 0141 0161 0171 0181 0191 \ +#0211 0221 0231 0241 0261 0271 0281 0291 +#do +for n in \ +0111 0121 0131 0181 0191 0231 0241 0271 0281 +do + + echo === Starting member ${n} === + + if [[ ${n} == 0111 || ${n} == 0121 || ${n} == 0131 || ${n} == 0181 || ${n} == 0191 || \ + ${n} == 0231 || ${n} == 0241 || ${n} == 0271 || ${n} == 0281 ]] + then + echo "first stop" ${n} + else + echo "second stop" + fi + + echo ============================ + +done + diff --git a/scripts/v2.LR.SSP370-LE/test2.sh b/scripts/v2.LR.SSP370-LE/test2.sh new file mode 100755 index 0000000..2376333 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/test2.sh @@ -0,0 +1,12 @@ +#!/bin/bash +## declare an array variable +declare -a array=("one" "two" "three") + +# get length of an array +arraylength=${#array[@]} + +# use for loop to read all values and indexes +for (( i=0; i<${arraylength}; i++ )); +do + echo "index: $i, value: ${array[$i]}" +done diff --git a/scripts/v2.LR.SSP370-LE/test3.sh b/scripts/v2.LR.SSP370-LE/test3.sh new file mode 100755 index 0000000..4005fdd --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/test3.sh @@ -0,0 +1,22 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +## declare an array variable +declare -a array=("one" "two" "three") + +# get length of an array +arraylength=${#array[@]} + +# use for loop to read all values and indexes +for (( i=0; i<${arraylength}; i++ )); +do + echo "index: $i, value: ${array[$i]}" +done + + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle-change-STOP_N.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle-change-STOP_N.sh new file mode 100755 index 0000000..252f63d --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle-change-STOP_N.sh @@ -0,0 +1,97 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.LE +#SBATCH --account=mp9 +#SBATCH --nodes=1618 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.LE.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.LE_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0241 0261 0271 0281 0291 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + if [[ ${n} == 0111 || ${n} == 0121 || ${n} == 0131 || ${n} == 0181 || ${n} == 0191 || \ + ${n} == 0231 || ${n} == 0241 || ${n} == 0271 || ${n} == 0281 ]] + then + ./xmlchange STOP_N="4" + else + ./xmlchange STOP_N="10" + fi + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP + . ./v2.LR.LE_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +. ./v2.LR.LE_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.LE_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle.sh new file mode 100755 index 0000000..43c795e --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_bundle.sh @@ -0,0 +1,91 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.LE +#SBATCH --account=mp9 +#SBATCH --nodes=1618 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.LE.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.LE_bundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0241 0261 0271 0281 0291 +do + + echo === Starting member ${n} === + + cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + ./xmlchange STOP_N="10" + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP + . ./v2.LR.LE_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +. ./v2.LR.LE_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.LE_bundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_extra.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_extra.sh new file mode 100644 index 0000000..0329d34 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_extra.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +#echo ======================================================== +#n="0141" +#echo Attempting to restart ${n} +#date +#cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts +#./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " +#./xmlchange STOP_N="5" +#####./xmlchange STOP_N="3" +#./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & +#PID=$! +#echo $PID +#echo ======================================================== + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.TEST.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.TEST.sh new file mode 100755 index 0000000..9fde641 --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.TEST.sh @@ -0,0 +1,39 @@ +#!/bin/bash -l + +echo ===== Start of v2.LR.LE_mixedbundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0101 0141 0151 0161 0201 0211 0251 0291 0301 0320 0480 +do + + echo === Starting member ${n} === + + if [[ ${n} == 0320 || ${n} == 0480 ]] + then + if [[ ${n} == 0480 ]] + then + echo "0480" + fi + + if [[ ${n} == 0320 ]] + then + echo "0320" + fi + echo "do some stuff" + else + echo "do other stuff" + if [[ ${n} == 0211 ]] + then + echo "0211" + else + echo "everyone else" + fi + fi + +done + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.sh new file mode 100755 index 0000000..fa4637c --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_mixedBundle.sh @@ -0,0 +1,119 @@ +#!/bin/bash -l + +#------------------------------------------------------------------------------ +# Batch system directives +#------------------------------------------------------------------------------ +#SBATCH --job-name=v2.LR.LE +#SBATCH --account=mp9 +#SBATCH --nodes=1114 +###SBATCH --nodes=1618 +#SBATCH --output=/global/cscratch1/sd/nanr/E3SMv2/v2.LR.LE.o%j +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --constraint=knl +#SBATCH --qos=regular +#SBATCH --no-kill +#SBATCH --requeue + +echo ===== Start of v2.LR.LE_mixedbundle.sh ===== +date +echo ======================================= + +export SLURM_NNODES=101 + +# Loop over members +for n in \ +0101 0141 0151 0161 0201 0211 0251 0291 0301 0320 0480 +do + + echo === Starting member ${n} === + + if [[ ${n} == 0320 || ${n} == 0480 ]] + then + if [[ ${n} == 0480 ]] + then + cd /global/project/projectdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0480.002/ + fi + + if [[ ${n} == 0320 ]] + then + cd /global/project/projectdirs/ccsm1/people/nanr/cases/e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0320.003/ + fi + + ./xmlchange STOP_N=4 + ./xmlchange JOB_WALLCLOCK_TIME="48:00:00 --subgroup case.run" + ./xmlchange JOB_WALLCLOCK_TIME="00:30:00 --subgroup case.st_archive" + ./xmlchange JOB_QUEUE="debug --subgroup case.st_archive" + ./xmlchange RESUBMIT=0 + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + + else + + cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts + ./xmlchange run_exe="--kill-on-bad-exit=1 --job-name=${n} \${EXEROOT}/e3sm.exe " + if [[ ${n} == 0211 ]] + then + ./xmlchange STOP_N="3" + else + ./xmlchange STOP_N="10" + fi + fi + + ./case.submit --no-batch 2>&1 > ../log.o${SLURM_JOB_ID} & + PID=$! + echo $PID + + echo ============================ + +done + +# Wait loop with external hook +while true +do + + sleep 60 + + # Execute extra instructions + cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP + . ./v2.LR.LE_extra.sh + + # List running background processes. + # (Needed for the stop clause below to work) + k=$((k+1)) + if (( k % 5 == 0 )) + then + echo ============================ + date + jobs -l + echo ---------------------------- + squeue --job=${SLURM_JOBID} --steps + echo ============================ + fi + + # Stop when all processes are done + n=`jobs -l | wc -l` + if (( n == 0 )) + then + echo ============================ + date + echo No running jobs left + echo ============================ + break + fi + +done + +# Wait for all members to complete +wait + +# Post steps +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +. ./v2.LR.LE_post.sh + +# That's all folks! +sleep 10 + +echo ===== End of v2.LR.LE_mixedbundle.sh ===== +date +echo ===================================== + diff --git a/scripts/v2.LR.SSP370-LE/v2.LR.LE_post.sh b/scripts/v2.LR.SSP370-LE/v2.LR.LE_post.sh new file mode 100644 index 0000000..211bdde --- /dev/null +++ b/scripts/v2.LR.SSP370-LE/v2.LR.LE_post.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +for n in \ +0111 0121 0131 0141 0161 0171 0181 0191 \ +0211 0221 0231 0241 0261 0271 0281 0291 +do + cd /global/cscratch1/sd/nanr/E3SMv2/v2.LR.SSP370_${n}/case_scripts + ./xmlchange CONTINUE_RUN="TRUE" +done + +echo === Submitting continuation job === +date +cd /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/SSP +sbatch v2.LR.LE_bundle.sh +echo =================================== diff --git a/scripts/v21.LR.1pctCO2/README.v21 b/scripts/v21.LR.1pctCO2/README.v21 new file mode 100644 index 0000000..f882364 --- /dev/null +++ b/scripts/v21.LR.1pctCO2/README.v21 @@ -0,0 +1,6 @@ +v21 1PCT run +/home/g/golaz/E3SMv2_1/v2_1.LR.1pctCO2_0101 + +bash +source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh +zstash extract --hpss=/home/g/golaz/E3SMv2_1/v2_1.LR.1pctCO2_0101 "archive/rest/0151-01-01-00000/*" diff --git a/scripts/v21.LR.1pctCO2/run.v2.LR.1pctCO2-EXTEND.sh b/scripts/v21.LR.1pctCO2/run.v2.LR.1pctCO2-EXTEND.sh new file mode 100755 index 0000000..303f647 --- /dev/null +++ b/scripts/v21.LR.1pctCO2/run.v2.LR.1pctCO2-EXTEND.sh @@ -0,0 +1,478 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850-1pctCO2" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.1pctCO2_EXTEND.001" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +readonly CHECKOUT="20220318" +readonly BRANCH="maint-2.0" # master as of Dec 21, 2021 +readonly CHERRY=( ) +readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0151-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/lvroekel/hosing_experiments/0151-01-01-00000/" +readonly RUN_REFCASE="v2.LR.1pctCO2_0101" +readonly RUN_REFDATE="0151-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/E3SM/" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +#readonly CASE_ROOT="${SCRATCH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh new file mode 100755 index 0000000..87ea91f --- /dev/null +++ b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh @@ -0,0 +1,479 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850-1pctCO2" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.1pctCO2_EXTEND.001" +readonly CASE_GROUP="v21.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0151-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=FALSE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2_1.LR.1pctCO2_0101/rest/0151-01-01-00000" +readonly RUN_REFCASE="v2_1.LR.1pctCO2_0101" +readonly RUN_REFDATE="0151-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +CASE_ROOT="/pscratch/sd/n/${USER}/${CASE_NAME}/" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + #cp ${RUN_REFDIR}/rpointer.* ${CASE_RUN_DIR}/ + #ln ${RUN_REFDIR}/v2_1.* ${CASE_RUN_DIR}/ + + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-KindOfOriginal b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-KindOfOriginal new file mode 100755 index 0000000..617352d --- /dev/null +++ b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-KindOfOriginal @@ -0,0 +1,478 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850-1pctCO2" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.1pctCO2_EXTEND.001" +readonly CASE_GROUP="v21.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0151-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2_1.LR.1pctCO2_0101/" +readonly RUN_REFCASE="v2_1.LR.1pctCO2_0101" +readonly RUN_REFDATE="0151-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +#readonly CASE_ROOT="${SCRATCH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-preNov b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-preNov new file mode 100755 index 0000000..e2780b8 --- /dev/null +++ b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-EXTEND.sh-preNov @@ -0,0 +1,479 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850-1pctCO2" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.1pctCO2_EXTEND.001" +readonly CASE_GROUP="v21.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0151-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2_1.LR.1pctCO2_0101/rest/0151-01-01-00000" +readonly RUN_REFCASE="v2_1.LR.1pctCO2_0101" +readonly RUN_REFDATE="0151-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +MY_PATH="/global/cfs/cdirs/mp9/" +CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/" +CASE_ROOT="/pscratch/sd/n/${USER}/${CASE_NAME}/" + +# Sub-directories +CASE_BUILD_DIR=${CASE_ROOT}/build +CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + cp /global/u2/n/nanr/CESM_tools/e3sm/v2/scripts/v2.SMYLE/env_mach/env_mach_specific.xml ${CASE_SCRIPTS_DIR}/ + #cp ${RUN_REFDIR}/rpointer.* ${CASE_RUN_DIR}/ + #ln ${RUN_REFDIR}/v2_1.* ${CASE_RUN_DIR}/ + + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-RampDown.sh b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-RampDown.sh new file mode 100755 index 0000000..870e66e --- /dev/null +++ b/scripts/v21.LR.1pctCO2/run.v21.LR.1pctCO2-RampDown.sh @@ -0,0 +1,478 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850-1pctCO2" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.1pctCO2_RampDown.001" +readonly CASE_GROUP="v21.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0151-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/lvroekel/hosing_experiments/0151-01-01-00000/" +readonly RUN_REFCASE="v2.LR.1pctCO2_0101" +readonly RUN_REFDATE="0151-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2/${CASE_NAME}" +#readonly CASE_ROOT="${SCRATCH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/README.Fix-xtime b/scripts/v21.LR.AMOC-hosing/README.Fix-xtime new file mode 120000 index 0000000..bd47f46 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/README.Fix-xtime @@ -0,0 +1 @@ +/global/homes/n/nanr/README.Fix-xtime \ No newline at end of file diff --git a/scripts/v21.LR.AMOC-hosing/README.amocHosingFromLuke b/scripts/v21.LR.AMOC-hosing/README.amocHosingFromLuke new file mode 100644 index 0000000..da5e3c6 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/README.amocHosingFromLuke @@ -0,0 +1,24 @@ +Hi Aixue and Nan, + + I do remember chatting about this previously Aixue and from our AMOC group this could be very beneficial to our project as well. I've forgotten the details of this experiment, is it a fixed in time forcing or monthly varying? Is it applied as a freshwater condition or virtual salt flux? Since I don't remember I'll try outline what we would do for each case + + +Freshwater flux, fixed in time +If easy for distribution to be specified in code (e.g. lat/lon bounds) this is trivial, we could just add on to some freshwater flux in ocn_comp_mct.F +If we need a file, we could potentially read in a freshwaterflux and add it to what is applied. Again not a difficult change +Freshwater flux, monthly varying +Trickier, but we have the pieces in place. It would take more time but we could get there. + + +Virtual salt fluxes would likely be easier, for fixed in time we could do tracer restoring, for monthly varying it would be similar to salinity restoring. + + +Kind of a long way around, but all definitely doable. + + + +I'd really like to help but am not sure I can commit the time. I wonder if perhaps Mat Maltrud and I could tag team this to get it in and passed on to you. I think if we shared the load we could get this done. I know I've spoken with Mat on this before so I'm guessing we can both be counted in, but I'll confirm with him. + + + +Regards, diff --git a/scripts/v21.LR.AMOC-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F b/scripts/v21.LR.AMOC-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F new file mode 100644 index 0000000..50c864f --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F @@ -0,0 +1,372 @@ +! Copyright (c) 2013, Los Alamos National Security, LLC (LANS) +! and the University Corporation for Atmospheric Research (UCAR). +! +! Unless noted otherwise source code is licensed under the BSD license. +! Additional copyright and license information can be found in the LICENSE file +! distributed with this code, or at http://mpas-dev.github.io/license.html +! +!||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| +! +! ocn_thick_surface_flux +! +!> \brief MPAS ocean surface fluxes for thickness +!> \author Doug Jacobsen +!> \date 12/17/12 +!> \details +!> This module contains the routine for computing +!> tendencies for thickness from surface fluxes +! +!----------------------------------------------------------------------- + +module ocn_thick_surface_flux + + use mpas_timer + use mpas_derived_types + use mpas_pool_routines + use mpas_timekeeping + use mpas_forcing + use mpas_stream_manager + use ocn_constants + use ocn_config + use ocn_framework_forcing + use ocn_mesh + + use ocn_forcing + + implicit none + private + save + + !-------------------------------------------------------------------- + ! + ! Public parameters + ! + !-------------------------------------------------------------------- + + !-------------------------------------------------------------------- + ! + ! Public member functions + ! + !-------------------------------------------------------------------- + + public :: ocn_thick_surface_flux_tend, & + ocn_thick_surface_flux_init, & + ocn_get_hosingData, & + ocn_hosing_write_restart + + !-------------------------------------------------------------------- + ! + ! Private module variables + ! + !-------------------------------------------------------------------- + + logical :: surfaceThicknessFluxOn + +!*********************************************************************** + +contains + +!*********************************************************************** +! +! routine ocn_thick_surface_flux_tend +! +!> \brief Computes tendency term from horizontal advection of thickness +!> \author Doug Jacobsen +!> \date 15 September 2011 +!> \details +!> This routine computes the horizontal advection tendency for +!> thicknes based on current state and user choices of forcings. +! +!----------------------------------------------------------------------- + + subroutine ocn_thick_surface_flux_tend(transmissionCoefficients, transmissionCoefficientsRunoff, & + surfaceThicknessFlux, surfaceThicknessFluxRunoff, tend, err)!{{{ + !----------------------------------------------------------------- + ! + ! input variables + ! + !----------------------------------------------------------------- + + real (kind=RKIND), dimension(:,:), intent(in) :: & + transmissionCoefficients, &!< Input: Coefficients for the transmission of surface fluxes + transmissionCoefficientsRunoff !< Input: Coefficients for the transmission of surface fluxes due to river runoff + + real (kind=RKIND), dimension(:), intent(in) :: & + surfaceThicknessFlux, &!< Input: surface flux of thickness + surfaceThicknessFluxRunoff !< Input: surface flux of thickness due to river runoff + + + !----------------------------------------------------------------- + ! + ! input/output variables + ! + !----------------------------------------------------------------- + + real (kind=RKIND), dimension(:,:), intent(inout) :: & + tend !< Input/Output: thickness tendency + + !----------------------------------------------------------------- + ! + ! output variables + ! + !----------------------------------------------------------------- + + integer, intent(out) :: err !< Output: error flag + + !----------------------------------------------------------------- + ! + ! local variables + ! + !----------------------------------------------------------------- + + integer :: iCell, k + + real (kind=RKIND) :: remainingFlux, remainingFluxRunoff + + err = 0 + + if (.not. surfaceThicknessFluxOn) return + + call mpas_timer_start("thick surface flux") + +#ifdef MPAS_OPENACC + !$acc enter data copyin(transmissionCoefficients, transmissionCoefficientsRunoff) + + !$acc parallel loop & + !$acc present(tend, surfaceThicknessFlux, surfaceThicknessFluxRunoff, & + !$acc transmissionCoefficients, transmissionCoefficientsRunoff, minLevelCell, maxLevelCell) & + !$acc private(k, remainingFlux, remainingFluxRunoff) +#else + !$omp parallel + !$omp do schedule(runtime) private(remainingFlux, remainingFluxRunoff, k) +#endif + do iCell = 1, nCellsOwned + remainingFlux = 1.0_RKIND + remainingFluxRunoff = 1.0_RKIND + do k = minLevelCell(iCell), maxLevelCell(iCell) + remainingFlux = remainingFlux - transmissionCoefficients(k, iCell) + remainingFluxRunoff = remainingFluxRunoff - transmissionCoefficientsRunoff(k, iCell) + + tend(k, iCell) = tend(k, iCell) + surfaceThicknessFlux(iCell) * transmissionCoefficients(k, iCell) & + + surfaceThicknessFluxRunoff(iCell) * transmissionCoefficientsRunoff(k, iCell) + end do + + if(maxLevelCell(iCell) > 0 .and. remainingFlux > 0.0_RKIND) then + tend(maxLevelCell(iCell), iCell) = tend(maxLevelCell(iCell), iCell) + remainingFlux * surfaceThicknessFlux(iCell) + end if + + if(maxLevelCell(iCell) > 0 .and. remainingFluxRunoff > 0.0_RKIND) then + tend(maxLevelCell(iCell), iCell) = tend(maxLevelCell(iCell), iCell) & + + remainingFluxRunoff * surfaceThicknessFluxRunoff(iCell) + end if + end do +#ifndef MPAS_OPENACC + !$omp end do + !$omp end parallel +#endif + +#ifdef MPAS_OPENACC + !$acc exit data delete(transmissionCoefficients, transmissionCoefficientsRunoff) +#endif + + call mpas_timer_stop("thick surface flux") + + !-------------------------------------------------------------------- + + end subroutine ocn_thick_surface_flux_tend!}}} + +!*********************************************************************** +! +! routine ocn_thick_surface_flux_init +! +!> \brief Initializes ocean horizontal thickness surface fluxes +!> \author Doug Jacobsen +!> \date 12/17/12 +!> \details +!> This routine initializes quantities related to thickness +!> surface fluxes in the ocean. +! +!----------------------------------------------------------------------- + + subroutine ocn_thick_surface_flux_init(domain,err)!{{{ + + !-------------------------------------------------------------------- + + !----------------------------------------------------------------- + ! + ! call individual init routines for each parameterization + ! + !----------------------------------------------------------------- + + type (domain_type), intent(inout) :: domain !< Input/Output: domain information + + integer, intent(out) :: err !< Output: error flag + +!----------------------------------------------------------------- +! variables for freshwater hosing +!----------------------------------------------------------------- + + type (mpas_pool_type), pointer :: forcingPool + + real (kind=RKIND), dimension(:), pointer :: & + hosingFlux, & ! input flux in forcing pool + freshWaterHosing ! input flux in stream + + character(len=strKIND) :: & + forcingIntervalAnnual, & + forcingReferenceTimeAnnual + + type (MPAS_Time_Type) :: currTime + integer :: ierr + character(len=StrKIND) :: timeStamp + + err = 0 + + surfaceThicknessFluxOn = .true. + + if (config_disable_thick_sflux) then + surfaceThicknessFluxOn = .false. + end if + +!----------------------------------------------------------------- +! freshwater hosing +!----------------------------------------------------------------- + + if (.not. config_use_hosing_flux) return + +!----------------------------------------------------------------------- +! initialize annual forcing to be read from file +!----------------------------------------------------------------------- + + forcingIntervalAnnual = "0001-00-00_00:00:00" + forcingReferenceTimeAnnual = "1850-07-01_00:00:00" + + currTime = mpas_get_clock_time(domain % clock, MPAS_NOW, ierr) + call mpas_get_time(curr_time=currTime, dateTimeString=timeStamp, ierr=ierr) + + call MPAS_forcing_init_group( forcingGroupHead, & + "hosingAnnualFreshwaterForcing", & + domain, & + trim(timeStamp), & + '1850-07-01_00:00:00', & + '0600-00-00_00:00:00', & + config_do_restart) + + call mpas_pool_get_subpool(domain % blocklist % structs, 'forcing', forcingPool) + call mpas_pool_get_array(forcingPool, 'freshWaterHosing', freshWaterHosing) + + call MPAS_forcing_init_field( domain % streamManager, & + forcingGroupHead, & + 'hosingAnnualFreshwaterForcing', & + 'freshWaterHosing', & + 'hosing_annual_flux', & + 'forcing', & + 'freshWaterHosing', & + 'linear', & + forcingReferenceTimeAnnual, & + forcingIntervalAnnual) + + call MPAS_forcing_init_field_data( forcingGroupHead, & + 'hosingAnnualFreshwaterForcing', & + domain % streamManager, & + config_do_restart, & + .false.) + + !-------------------------------------------------------------------- + + end subroutine ocn_thick_surface_flux_init!}}} + +!*********************************************************************** +!*********************************************************************** +! +! routine ocn_get_hosingData +! +!> \brief retrieve data needed for freshwater hosing surface fluxes +!> \author Mathew Maltrud +!> \date 11/30/2022 +!> \details +!> This routine calls mpas_forcing routines to acquire needed hsoing data and interpolates +!> between time levels. directly copied from ocn_get_CFCData. +! +!----------------------------------------------------------------------- + + subroutine ocn_get_hosingData( streamManager, & + domain, & + simulationClock, & + firstTimeStep) !{{{ + + type (MPAS_streamManager_type), intent(inout) :: streamManager + + type (domain_type) :: domain + type (MPAS_timeInterval_type) :: timeStepHosing + type (MPAS_clock_type) :: simulationClock + + logical, intent(in) :: firstTimeStep + real(kind=RKIND) :: dt + + type (mpas_pool_type), pointer :: forcingPool + type (mpas_pool_type), pointer :: meshPool + + real (kind=RKIND), dimension(:), pointer :: & + hosingFlux, & + freshWaterHosing + + integer, pointer :: nCells + integer :: iCell + + call mpas_set_timeInterval(timeStepHosing,timeString=config_dt) + call mpas_get_timeInterval(timeStepHosing,dt=dt) + + if (firstTimeStep .and. config_do_restart) then + call MPAS_forcing_get_forcing(forcingGroupHead, & + 'hosingAnnualFreshwaterForcing', streamManager, 0.0_RKIND) + else + call MPAS_forcing_get_forcing(forcingGroupHead, & + 'hosingAnnualFreshwaterForcing', streamManager, dt) + endif + +!maltrud debug +!write(*,*)' DEBUG02 : in get_hosingData' + + call mpas_pool_get_subpool(domain % blocklist % structs, 'mesh', meshPool) + call mpas_pool_get_subpool(domain % blocklist % structs, 'forcing', forcingPool) + call mpas_pool_get_array(forcingPool, 'hosingFlux', hosingFlux) + call mpas_pool_get_array(forcingPool, 'freshWaterHosing', freshWaterHosing) + + call mpas_pool_get_dimension(meshPool, 'nCells', nCells) + + do iCell = 1, nCells + hosingFlux(iCell) = freshWaterHosing(iCell) +!maltrud debug +!if(hosingFlux(iCell) > 0.0_RKIND) write(*,*)' DEBUG04 : ', iCell, hosingFlux(iCell) + + enddo + + end subroutine ocn_get_hosingData!}}} + +!*********************************************************************** +! +! routine ocn_hosing_write_restart +! +!> \brief writes restart timestamp for hosing data to be read in on future restart +!> \author Mathew Maltrud +!> \date 11/30/2022 + +! +!----------------------------------------------------------------------- + + subroutine ocn_hosing_write_restart(domain)!{{{ + + type(domain_type) :: domain + + call MPAS_forcing_write_restart_times(forcingGroupHead) + + end subroutine ocn_hosing_write_restart!}}} + +!----------------------------------------------------------------------- + +end module ocn_thick_surface_flux + +!||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| + +! vim: foldmethod=marker diff --git a/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh b/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh new file mode 100755 index 0000000..25f3e3d --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh @@ -0,0 +1,479 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.hosingRampUp.beta1.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0108-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cfs/cdirs/mp9/archive/e3smv2/v21.LR.hosingRampUp.beta1.001/rest/" +readonly RUN_REFCASE="v21.LR.hosingRampUp.beta1.001" +readonly RUN_REFDATE="0108-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +#readonly CASE_ROOT="${SCRATCH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh-cori-knl b/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh-cori-knl new file mode 100755 index 0000000..048c44d --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/badScripts/run.v21.LR.hosing.sh-cori-knl @@ -0,0 +1,479 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=cori-knl +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.hosingRampUp.beta1.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0001-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/global/cscratch1/sd/nanr/archive/v2.LR.piControl/archive/rest/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/project/projectdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +#readonly CASE_ROOT="${SCRATCH}/cases/e3smv2/${CASE_NAME}" + +# Sub-directories +#readonly CASE_BUILD_DIR=${CASE_ROOT}/build +#readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=true +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +!! setting constant co2vmr +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP6_1pctCO2_c20180216.nc' + ch4vmr = 808.249e-9 + co2vmr = 1264.76e-6 + f11vmr = 32.1102e-12 + f12vmr = 0.0 + flbc_list = ' ' + n2ovmr = 273.0211e-9 + scenario_ghg = 'FIXED' +/ + + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/constant-0.2SvForcing/streams.ocean b/scripts/v21.LR.AMOC-hosing/constant-0.2SvForcing/streams.ocean new file mode 100644 index 0000000..ec36440 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/constant-0.2SvForcing/streams.ocean @@ -0,0 +1,1013 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/scripts/v21.LR.AMOC-hosing/constant_neg10xSvForcing/streams.ocean b/scripts/v21.LR.AMOC-hosing/constant_neg10xSvForcing/streams.ocean new file mode 100644 index 0000000..9279413 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/constant_neg10xSvForcing/streams.ocean @@ -0,0 +1,1013 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/scripts/v21.LR.AMOC-hosing/env_mach_pes/env_mach_pes.xml b/scripts/v21.LR.AMOC-hosing/env_mach_pes/env_mach_pes.xml new file mode 100644 index 0000000..3f76401 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/env_mach_pes/env_mach_pes.xml @@ -0,0 +1,196 @@ + + +
+ These variables CANNOT be modified once case_setup has been + invoked without first invoking case_setup -reset + + NTASKS: the total number of MPI tasks, a negative value indicates nodes rather than tasks. + NTHRDS: the number of OpenMP threads per MPI task. + ROOTPE: the global mpi task of the component root task, if negative, indicates nodes rather than tasks. + PSTRID: the stride of MPI tasks across the global set of pes (for now set to 1) + NINST : the number of component instances (will be spread evenly across NTASKS) + + for example, for NTASKS = 8, NTHRDS = 2, ROOTPE = 32, NINST = 2 + the MPI tasks would be placed starting on global pe 32 and each pe would be threaded 2-ways + These tasks will be divided amongst both instances (4 tasks each). + + Note: PEs that support threading never have an MPI task associated + with them for performance reasons. As a result, NTASKS and ROOTPE + are relatively independent of NTHRDS and they determine + the layout of mpi processors between components. NTHRDS is used + to determine how those mpi tasks should be placed across the machine. + + The following values should not be set by the user since they'll be + overwritten by scripts: TOTALPES, NTASKS_PER_INST +
+ -compset A_WCYCL* -res ne30pg2_oECv3 with MPASO on 7 nodes, 128x1 c8 + + + integer + pes or cores used relative to MAX_MPITASKS_PER_NODE for accounting (0 means TOTALPES is valid) + + + integer + total number of physical cores used (setup automatically - DO NOT EDIT) + + + integer + maximum number of tasks/ threads allowed per node + + + integer + pes or cores per node for mpitasks + + + integer + pes or cores per node for accounting purposes + + + + + logical + TRUE,FALSE + Allocate some spare nodes to handle node failures. The system will pick a reasonable number + + + integer + Force this exact number of spare nodes to be allocated + + + integer + + 640 + 80 + 256 + 1 + 1 + 640 + 640 + 640 + 1 + 1 + + number of tasks for each component + + + integer + + 640 + 256 + 1 + 1 + 640 + 640 + 640 + 1 + 1 + + Number of tasks per instance for each component. DO NOT EDIT: Set automatically by case.setup based on NTASKS, NINST and MULTI_DRIVER + + + integer + + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + + number of threads for each task in each component + + + integer + + 0 + 0 + 640 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + + ROOTPE (mpi task in MPI_COMM_WORLD) for each component + + + logical + TRUE,FALSE + MULTI_DRIVER mode provides a separate driver/coupler component for each + ensemble member. All components must have an equal number of members. If + MULTI_DRIVER mode is False prognostic components must have the same number + of members but data or stub components may also have 1 member. + + + integer + + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + + Number of instances for each component. If MULTI_DRIVER is True + the NINST_MAX value will be used. + + + + char + sequential,concurrent + + concurrent + concurrent + concurrent + concurrent + concurrent + concurrent + concurrent + concurrent + concurrent + + Layout of component instances for each component + + + integer + + 1 + 8 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + + The mpi global processors stride associated with the mpi tasks for the a component + + + integer + + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + + Stride of MPI tasks owned exclusively by a component. If 0, exclusivity is disabled. + + + diff --git a/scripts/v21.LR.AMOC-hosing/hosingGuide.txt b/scripts/v21.LR.AMOC-hosing/hosingGuide.txt new file mode 120000 index 0000000..d2ef882 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/hosingGuide.txt @@ -0,0 +1 @@ +/global/cfs/cdirs/ccsm1/people/nanr/e3sm_tags/E3SMv2.1-hosing/hosing/hosingGuide.txt \ No newline at end of file diff --git a/scripts/v21.LR.AMOC-hosing/run.v2.LR.hosingConstant-neg0.5Sv.sh b/scripts/v21.LR.AMOC-hosing/run.v2.LR.hosingConstant-neg0.5Sv.sh new file mode 100755 index 0000000..8ece11c --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/run.v2.LR.hosingConstant-neg0.5Sv.sh @@ -0,0 +1,485 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +#readonly COMPSET="WCYCL1850" +readonly COMPSET="WCYCL20TR" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v2.LR.20TR_neg0.5Sv.001" +readonly CASE_GROUP="v2.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1850-01-01" + +echo $MODEL_START_TYPE + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/0501-01-01-00000/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +readonly MY_PATH="/global/cfs/cdirs/mp9/" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2-hosing/" +readonly CHECKOUT="20220412" +readonly BRANCH="maint-2.0" # master as of 2021-12-21 +readonly CHERRY=( ) + +readonly CASE_ROOT="${SCRATCH}/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/bld +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,365,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +cat << EOF >> user_nl_mpaso + config_use_hosing_flux = .true. +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Add Hosing SourceMods + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F ./SourceMods/src.mpaso/ + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/v2-constant-0.5SvForcing/streams.ocean ./SourceMods/src.mpaso/ + #cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/env_mach_pes/env_mach_pes.xml . + #cp /pscratch/sd/n/nanr/v21.LR.20TR_neg10xSv.001/case_scripts/env_mach_pes.xml . + cp /pscratch/sd/n/nanr/v2.LR.GHG-SF/v2.LR.hist-GHG_0121/case_scripts/env_mach_pes.xml . + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant-0.2Sv.sh b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant-0.2Sv.sh new file mode 100755 index 0000000..c0e7d19 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant-0.2Sv.sh @@ -0,0 +1,478 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +#readonly COMPSET="WCYCL1850" +readonly COMPSET="WCYCL20TR" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.20TR_neg0.2Sv.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0001-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/0501-01-01-00000/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +readonly CASE_ROOT="${SCRATCH}/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/bld +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,365,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +cat << EOF >> user_nl_mpaso + config_use_hosing_flux = .true. +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Add Hosing SourceMods + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F ./SourceMods/src.mpaso/ + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/constant-0.2SvForcing/streams.ocean ./SourceMods/src.mpaso/ + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant_neg10xSv.sh b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant_neg10xSv.sh new file mode 100755 index 0000000..6ef46ed --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingConstant_neg10xSv.sh @@ -0,0 +1,479 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +#readonly COMPSET="WCYCL1850" +readonly COMPSET="WCYCL20TR" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.20TR_neg10xSv.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="1850-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/0501-01-01-00000/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +readonly CASE_ROOT="${SCRATCH}/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/bld +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,365,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' + +! Override - updated after EAM/ELM fixes + check_finidat_fsurdat_consistency = .false. + check_finidat_pct_consistency = .true. + check_finidat_year_consistency = .true. + +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +cat << EOF >> user_nl_mpaso + config_use_hosing_flux = .true. +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Add Hosing SourceMods + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F ./SourceMods/src.mpaso/ + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/constant_neg10xSvForcing/streams.ocean ./SourceMods/src.mpaso/ + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped-EXTEND.sh b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped-EXTEND.sh new file mode 100755 index 0000000..5f9d1b8 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped-EXTEND.sh @@ -0,0 +1,473 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.hosingRampUp-EXTEND.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0148-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/v21.LR.hosingRampUp.001/archive/rest/0148-01-01-00000/" +readonly RUN_REFCASE="v21.LR.hosingRampUp.001" +readonly RUN_REFDATE="0148-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +readonly MY_PATH="/global/cfs/cdirs/mp9/" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +readonly CASE_ROOT="${SCRATCH}/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/bld +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,365,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +cat << EOF >> user_nl_mpaso + config_use_hosing_flux = .true. +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Add Hosing SourceMods + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F ./SourceMods/src.mpaso/ + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/transientForcing/streams.ocean ./SourceMods/src.mpaso/ + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped.sh b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped.sh new file mode 100755 index 0000000..519942c --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/run.v21.LR.hosingRamped.sh @@ -0,0 +1,473 @@ +#!/bin/bash -fe + +# E3SM Water Cycle v2 run_e3sm script template. +# +# Inspired by v1 run_e3sm script as well as SCREAM group simplified run script. +# +# Bash coding style inspired by: +# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming + +# TO DO: +# - custom pelayout + +main() { + +# For debugging, uncomment libe below +#set -x + +# --- Configuration flags ---- + +# Machine and project +readonly MACHINE=pm-cpu +readonly PROJECT="mp9" + +# Simulation +readonly COMPSET="WCYCL1850" +readonly RESOLUTION="ne30pg2_EC30to60E2r2" +readonly CASE_NAME="v21.LR.hosingRampUp.001" +readonly CASE_GROUP="v2.1.LR" + +# Code and compilation +#readonly CHECKOUT="20221216" +#readonly BRANCH="b02bfec1784525d8f05c9fe7d60cc044ca435c40" # v2.beta 2.1 as of 2022-12-16 +#readonly CHERRY=( ) +#readonly DEBUG_COMPILE=false + +# Run options +readonly MODEL_START_TYPE="hybrid" # 'initial', 'continue', 'branch', 'hybrid' +readonly START_DATE="0001-01-01" + +# Additional options for 'branch' and 'hybrid' +readonly GET_REFCASE=TRUE +readonly RUN_REFDIR="/pscratch/sd/n/nanr/archive/v2.LR.piControl/rest/0501-01-01-00000/" +readonly RUN_REFCASE="v2.LR.piControl" +readonly RUN_REFDATE="0501-01-01" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid' + +# Set paths +#readonly MY_PATH="/global/cfs/cdirs/ccsm1/people/nanr" +readonly MY_PATH="/global/cfs/cdirs/mp9/" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2/code/${CHECKOUT}" +#readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1/code/v2.1.0-beta.3-hosing" +readonly CODE_ROOT="${MY_PATH}/e3sm_tags/E3SMv2.1-hosing/v2.1.0-beta.1" +#readonly CASE_ROOT="${MY_PATH}/cases/e3smv2.1/${CASE_NAME}" +readonly CASE_ROOT="${SCRATCH}/${CASE_NAME}" + +# Sub-directories +readonly CASE_BUILD_DIR=${CASE_ROOT}/bld +readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive +#readonly CASE_BUILD_DIR=$SCRATCH/$CASE_NAME/bld +#readonly CASE_ARCHIVE_DIR=$SCRATCH/archive/$CASE_NAME/ + +# Define type of run +# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays', +# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays' +# or 'production' for full simulation +readonly run='production' +if [ "${run}" != "production" ]; then + + # Short test simulations + tmp=($(echo $run | tr "_" " ")) + layout=${tmp[0]} + units=${tmp[2]} + resubmit=$(( ${tmp[1]%%x*} -1 )) + length=${tmp[1]##*x} + + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/tests/${run}/case_scripts + readonly CASE_RUN_DIR=${CASE_ROOT}/tests/${run}/run + readonly PELAYOUT=${layout} + readonly WALLTIME="2:00:00" + readonly STOP_OPTION=${units} + readonly STOP_N=${length} + readonly REST_OPTION=${STOP_OPTION} + readonly REST_N=${STOP_N} + readonly RESUBMIT=${resubmit} + readonly DO_SHORT_TERM_ARCHIVING=false + +else + + # Production simulation + readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts/ + readonly CASE_RUN_DIR=${SCRATCH}/${CASE_NAME}/run + readonly PELAYOUT="L" + readonly WALLTIME="24:00:00" + readonly STOP_OPTION="nyears" + readonly STOP_N="1" + readonly REST_OPTION="nyears" + readonly REST_N="1" + readonly RESUBMIT="2" + readonly DO_SHORT_TERM_ARCHIVING=false +fi + +# Coupler history +readonly HIST_OPTION="nyears" +readonly HIST_N="5" + +# Leave empty (unless you understand what it does) +readonly OLD_EXECUTABLE="" + +# --- Toggle flags for what to do ---- +do_fetch_code=false +do_create_newcase=true +do_case_setup=true +do_case_build=true +do_case_submit=false + +# --- Now, do the work --- + +# Make directories created by this script world-readable +umask 022 + +# Fetch code from Github +fetch_code + +# Create case +create_newcase + +# Setup +case_setup + +# Build +case_build + +# Configure runtime options +runtime_options + +# Copy script into case_script directory for provenance +copy_script + +# Submit +case_submit + +# All done +echo $'\n----- All done -----\n' + +} + +# ======================= +# Custom user_nl settings +# ======================= + +user_nl() { + +cat << EOF >> user_nl_eam + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,365,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + +cat << EOF >> user_nl_mpaso + config_use_hosing_flux = .true. +EOF + +} + +patch_mpas_streams() { + +echo + +} + +###################################################### +### Most users won't need to change anything below ### +###################################################### + +#----------------------------------------------------- +fetch_code() { + + if [ "${do_fetch_code,,}" != "true" ]; then + echo $'\n----- Skipping fetch_code -----\n' + return + fi + + echo $'\n----- Starting fetch_code -----\n' + local path=${CODE_ROOT} + local repo=e3sm + + echo "Cloning $repo repository branch $BRANCH under $path" + if [ -d "${path}" ]; then + echo "ERROR: Directory already exists. Not overwriting" + exit 20 + fi + mkdir -p ${path} + pushd ${path} + + # This will put repository, with all code + git clone git@github.com:E3SM-Project/${repo}.git . + + # Setup git hooks + rm -rf .git/hooks + git clone git@github.com:E3SM-Project/E3SM-Hooks.git .git/hooks + git config commit.template .git/hooks/commit.template + + # Check out desired branch + git checkout ${BRANCH} + + # Custom addition + if [ "${CHERRY}" != "" ]; then + echo ----- WARNING: adding git cherry-pick ----- + for commit in "${CHERRY[@]}" + do + echo ${commit} + git cherry-pick ${commit} + done + echo ------------------------------------------- + fi + + # Bring in all submodule components + git submodule update --init --recursive + + popd +} + +#----------------------------------------------------- +create_newcase() { + + if [ "${do_create_newcase,,}" != "true" ]; then + echo $'\n----- Skipping create_newcase -----\n' + return + fi + + echo $'\n----- Starting create_newcase -----\n' + + ${CODE_ROOT}/cime/scripts/create_newcase \ + --case ${CASE_NAME} \ + --case-group ${CASE_GROUP} \ + --output-root ${CASE_ROOT} \ + --script-root ${CASE_SCRIPTS_DIR} \ + --handle-preexisting-dirs u \ + --compset ${COMPSET} \ + --res ${RESOLUTION} \ + --machine ${MACHINE} \ + --project ${PROJECT} \ + --walltime ${WALLTIME} \ + --pecount ${PELAYOUT} + + if [ $? != 0 ]; then + echo $'\nNote: if create_newcase failed because sub-directory already exists:' + echo $' * delete old case_script sub-directory' + echo $' * or set do_newcase=false\n' + exit 35 + fi + +} + +#----------------------------------------------------- +case_setup() { + + if [ "${do_case_setup,,}" != "true" ]; then + echo $'\n----- Skipping case_setup -----\n' + return + fi + + echo $'\n----- Starting case_setup -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Setup some CIME directories + ./xmlchange CIME_OUTPUT_ROOT=${CASE_RUN_DIR} + ./xmlchange EXEROOT=${CASE_BUILD_DIR} + ./xmlchange RUNDIR=${CASE_RUN_DIR} + + # Short term archiving + ./xmlchange DOUT_S=${DO_SHORT_TERM_ARCHIVING^^} + ./xmlchange DOUT_S_ROOT=${CASE_ARCHIVE_DIR} + + # Build with COSP, except for a data atmosphere (datm) + if [ `./xmlquery --value COMP_ATM` == "datm" ]; then + echo $'\nThe specified configuration uses a data atmosphere, so cannot activate COSP simulator\n' + else + echo $'\nConfiguring E3SM to use the COSP simulator\n' + # ./xmlchange --id CAM_CONFIG_OPTS --append --val='-cosp' + fi + + # Extracts input_data_dir in case it is needed for user edits to the namelist later + local input_data_dir=`./xmlquery DIN_LOC_ROOT --value` + + # Add Hosing SourceMods + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/SourceMods/src.mpaso/mpas_ocn_thick_surface_flux.F ./SourceMods/src.mpaso/ + cp /global/homes/n/nanr/CESM_tools/e3sm/v2/scripts/amoc-hosing/transientForcing/streams.ocean ./SourceMods/src.mpaso/ + + # Custom user_nl + user_nl + + # Finally, run CIME case.setup + ./case.setup --reset + + popd +} + +#----------------------------------------------------- +case_build() { + + pushd ${CASE_SCRIPTS_DIR} + + # do_case_build = false + if [ "${do_case_build,,}" != "true" ]; then + + echo $'\n----- case_build -----\n' + + if [ "${OLD_EXECUTABLE}" == "" ]; then + # Ues previously built executable, make sure it exists + if [ -x ${CASE_BUILD_DIR}/e3sm.exe ]; then + echo 'Skipping build because $do_case_build = '${do_case_build} + else + echo 'ERROR: $do_case_build = '${do_case_build}' but no executable exists for this case.' + exit 297 + fi + else + # If absolute pathname exists and is executable, reuse pre-exiting executable + if [ -x ${OLD_EXECUTABLE} ]; then + echo 'Using $OLD_EXECUTABLE = '${OLD_EXECUTABLE} + cp -fp ${OLD_EXECUTABLE} ${CASE_BUILD_DIR}/ + else + echo 'ERROR: $OLD_EXECUTABLE = '$OLD_EXECUTABLE' does not exist or is not an executable file.' + exit 297 + fi + fi + echo 'WARNING: Setting BUILD_COMPLETE = TRUE. This is a little risky, but trusting the user.' + ./xmlchange BUILD_COMPLETE=TRUE + + # do_case_build = true + else + + echo $'\n----- Starting case_build -----\n' + + # Turn on debug compilation option if requested + if [ "${DEBUG_COMPILE^^}" == "TRUE" ]; then + ./xmlchange DEBUG=${DEBUG_COMPILE^^} + fi + + # Run CIME case.build + ./case.build + + # Some user_nl settings won't be updated to *_in files under the run directory + # Call preview_namelists to make sure *_in and user_nl files are consistent. + ./preview_namelists + + fi + + popd +} + +#----------------------------------------------------- +runtime_options() { + + echo $'\n----- Starting runtime_options -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Set simulation start date + ./xmlchange RUN_STARTDATE=${START_DATE} + + # Segment length + ./xmlchange STOP_OPTION=${STOP_OPTION,,},STOP_N=${STOP_N} + + # Restart frequency + ./xmlchange REST_OPTION=${REST_OPTION,,},REST_N=${REST_N} + + # Coupler history + ./xmlchange HIST_OPTION=${HIST_OPTION,,},HIST_N=${HIST_N} + + # Coupler budgets (always on) + ./xmlchange BUDGETS=TRUE + + # Set resubmissions + if (( RESUBMIT > 0 )); then + ./xmlchange RESUBMIT=${RESUBMIT} + fi + + # Run type + # Start from default of user-specified initial conditions + if [ "${MODEL_START_TYPE,,}" == "initial" ]; then + ./xmlchange RUN_TYPE="startup" + ./xmlchange CONTINUE_RUN="FALSE" + + # Continue existing run + elif [ "${MODEL_START_TYPE,,}" == "continue" ]; then + ./xmlchange CONTINUE_RUN="TRUE" + + elif [ "${MODEL_START_TYPE,,}" == "branch" ] || [ "${MODEL_START_TYPE,,}" == "hybrid" ]; then + ./xmlchange RUN_TYPE=${MODEL_START_TYPE,,} + ./xmlchange GET_REFCASE=${GET_REFCASE} + ./xmlchange RUN_REFDIR=${RUN_REFDIR} + ./xmlchange RUN_REFCASE=${RUN_REFCASE} + ./xmlchange RUN_REFDATE=${RUN_REFDATE} + echo 'Warning: $MODEL_START_TYPE = '${MODEL_START_TYPE} + echo '$RUN_REFDIR = '${RUN_REFDIR} + echo '$RUN_REFCASE = '${RUN_REFCASE} + echo '$RUN_REFDATE = '${START_DATE} + + else + echo 'ERROR: $MODEL_START_TYPE = '${MODEL_START_TYPE}' is unrecognized. Exiting.' + exit 380 + fi + + # Patch mpas streams files + patch_mpas_streams + + popd +} + +#----------------------------------------------------- +case_submit() { + + if [ "${do_case_submit,,}" != "true" ]; then + echo $'\n----- Skipping case_submit -----\n' + return + fi + + echo $'\n----- Starting case_submit -----\n' + pushd ${CASE_SCRIPTS_DIR} + + # Run CIME case.submit + ./case.submit + + popd +} + +#----------------------------------------------------- +copy_script() { + + echo $'\n----- Saving run script for provenance -----\n' + + local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance + mkdir -p ${script_provenance_dir} + local this_script_name=`basename $0` + local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S` + cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name} + +} + +#----------------------------------------------------- +# Silent versions of popd and pushd +pushd() { + command pushd "$@" > /dev/null +} +popd() { + command popd "$@" > /dev/null +} + +# Now, actually run the script +#----------------------------------------------------- +main + diff --git a/scripts/v21.LR.AMOC-hosing/setup_4xCO2-constant-E3SMv2.csh b/scripts/v21.LR.AMOC-hosing/setup_4xCO2-constant-E3SMv2.csh new file mode 100755 index 0000000..57f351a --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/setup_4xCO2-constant-E3SMv2.csh @@ -0,0 +1,42 @@ +#!/bin/csh -f + +set MACHINE = cori-knl + +set EXPERIMENT = 4CO2 +set RESOLN = ne30_oECv3_ICG +set COMPSET = A_WCYCL20TRS_CMIP6 +set PROJECT = mp9 +set TAG = E3SMv2 + + +set mbr=2 + +if ($mbr < 10) then + setenv CASE v2.LR.${EXPERIMENT}.00${mbr} +else + setenv CASE v2.LR.${EXPERIMENT}.0${mbr} +endif + +set PATH = /global/project/projectdirs/ccsm1/people/$USER/ +set RUNDIR = $SCRATCH/$USER/$CASE/run +set BLDDIR = $SCRATCH/$USER/$CASE/bld +setenv CASEROOT $PATH/cases/e3smv2/$CASE +set TOOLSROOT = /global/homes/n/nanr/CESM_tools/e3sm/e3sm2-hosing/ + +setenv E3SMROOT ${PATH}/e3sm_tags/${TAG}/ + +#$E3SMROOT/cime/scripts/create_newcase --case MCSP_CMT_5_coupled --compset A_WCYCL20TRS_CMIP6 --res ne30_oECv3_ICG --pecount L --handle-preexisting-dirs u --mach cori-knl --output-root /global/cscratch1/sd/sglanvil/ --script-root /global/homes/s/sglanvil/cases/MCSP_CMT_5_coupled -project mp9 + +$E3SMROOT/cime/scripts/create_newcase --case $CASE --compset $COMPSET --res $RESOLN --pecount L --handle-preexisting-dirs u --mach $MACHINE --output-root $SCRATCH --script-root $CASEROOT -project $PROJECT + +cd $CASEROOT +./xmlchange RUN_REFCASE=v2.LR.1pctCO2_0101 +./xmlchange RUN_REFDATE=0151-01-01 +./xmlchange RUN_STARTDATE=0151-01-01 + +cp $TOOLSROOT/SourceMods/src.cam/cam_diagnostics.F90 ./SourceMods/src.cam/ + +cp /global/cscratch1/sd/lvroekel/hosing_experiments/0151-01-01-00000/* $RUNDIR/ + + + diff --git a/scripts/v21.LR.AMOC-hosing/transientForcing/streams.ocean b/scripts/v21.LR.AMOC-hosing/transientForcing/streams.ocean new file mode 100644 index 0000000..b2c688c --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/transientForcing/streams.ocean @@ -0,0 +1,1013 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/scripts/v21.LR.AMOC-hosing/v2-constant-0.5SvForcing/streams.ocean b/scripts/v21.LR.AMOC-hosing/v2-constant-0.5SvForcing/streams.ocean new file mode 100644 index 0000000..c2104d1 --- /dev/null +++ b/scripts/v21.LR.AMOC-hosing/v2-constant-0.5SvForcing/streams.ocean @@ -0,0 +1,1001 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/user_nl_files/.README.hf-output.swp b/user_nl_files/.README.hf-output.swp new file mode 100644 index 0000000..325b480 Binary files /dev/null and b/user_nl_files/.README.hf-output.swp differ diff --git a/user_nl_files/1pct/README.1pct-namelists b/user_nl_files/1pct/README.1pct-namelists new file mode 100644 index 0000000..1525a14 --- /dev/null +++ b/user_nl_files/1pct/README.1pct-namelists @@ -0,0 +1,34 @@ +cat << EOF >> user_nl_eam + h0, h1,h2,h3,h4,h5,h6 + f1, f2,f3,f4,f5,f6,f7 + nhtfrq = 0,-24,-6,-6,-3,-24,0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ +', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005' +, 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + fincl7 = 'O3', 'PS', 'TROP_P' + +EOF + +cat << EOF >> user_nl_elm + hist_dov2xy = .true.,.true. + hist_fincl2 = 'H2OSNO', 'FSNO', 'QRUNOFF', 'QSNOMELT', 'FSNO_EFF', 'SNORDSL', 'SNOW', 'FSDS', 'FSR', 'FLDS', 'FIRE', 'FIRA' + hist_mfilt = 1,365 + hist_nhtfrq = 0,-24 + hist_avgflag_pertape = 'A','A' +EOF + +cat << EOF >> user_nl_mosart + rtmhist_fincl2 = 'RIVER_DISCHARGE_OVER_LAND_LIQ' + rtmhist_mfilt = 1,365 + rtmhist_ndens = 2 + rtmhist_nhtfrq = 0,-24 +EOF + diff --git a/user_nl_files/HISTsmbb-notUsed/atm_in b/user_nl_files/HISTsmbb-notUsed/atm_in new file mode 100644 index 0000000..1a5630e --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/atm_in @@ -0,0 +1,412 @@ +&aerosol_nl + aer_drydep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + aer_wetdep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + seasalt_emis_scale = 0.6 + sol_factb_interstitial = 0.1D0 + sol_facti_cloud_borne = 1.0D0 + sol_factic_interstitial = 0.4D0 + sscav_tuning = .true. +/ +&cam_history_nl + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp', 'extinct_lw_bnd7', 'extinct_lw_inp', 'CLD_CAL', 'TREFMNAV', 'TREFMXAV', 'IEFLX', 'ZMDT', 'ZMDQ', + 'TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + fincl2 = 'FLUT', 'PRECT', 'U200', 'V200', 'U850', 'V850', 'Z500', 'OMEGA500', 'UBOT', 'VBOT', 'TREFHT', 'TREFHTMN:M', + 'TREFHTMX:X', 'QREFHT', 'TS', 'PS', 'TMQ', 'TUQ', 'TVQ', 'TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', + 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', + 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', + 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT', 'PRECC', 'PRECTMX:X', 'PSL', 'RHREFHT', 'U10', 'Z200', 'QRS', + 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', + 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700', + 'T500', 'T010', 'TBOT', 'Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010', 'TROPF_P', + 'TROPF_T', 'TROPF_Z' + fincl3 = 'PSL', 'T200', 'T500', 'U850', 'V850', 'UBOT', 'VBOT', 'TREFHT', 'Z700', 'TBOT:M', 'FLDS', 'FSDS', 'PRECT', + 'PS', 'QREFHT', 'TS', 'TMQ', 'U10', 'Z200:I', 'Z500:I', 'TTQ:I', 'TUQ:I', 'TVQ:I', 'Q:I', 'T:I', 'U:I', + 'V:I', 'Z3:I' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP', 'MEANCLDALB_ISCCP', 'MEANTAU_ISCCP', 'MEANPTOP_ISCCP', 'MEANTB_ISCCP', 'CLDTOT_CAL', + 'CLDTOT_CAL_LIQ', 'CLDTOT_CAL_ICE', 'CLDTOT_CAL_UN', 'CLDHGH_CAL', 'CLDHGH_CAL_LIQ', 'CLDHGH_CAL_ICE', + 'CLDHGH_CAL_UN', 'CLDMED_CAL', 'CLDMED_CAL_LIQ', 'CLDMED_CAL_ICE', 'CLDMED_CAL_UN', 'CLDLOW_CAL', + 'CLDLOW_CAL_LIQ', 'CLDLOW_CAL_ICE', 'CLDLOW_CAL_UN' + fincl7 = 'O3','PS','TROP_P' + mfilt = 1,30,120,120,240,30,1 + nhtfrq = 0,-24,-6,-6,-3,-24,0 +/ +&cam_inparm + bnd_topo = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/topo/USGS-gtopo30_ne30np4pg2_16xdel2.c20200108.nc' + dtime = 1800 + ncdata = 'v2.LR.historical_0111.eam.i.2015-01-01-00000.nc' + phys_chnk_fdim_max = 30 + phys_chnk_fdim_mult = 2 + phys_loadbalance = 2 + print_energy_errors = .false. + raytau0 = 5.0D0 +/ +&chem_inparm + chlorine_loading_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/Linoz_Chlorine_Loading_CMIP6_Hist_SSP370_0003-2503_c20210202.nc' + chlorine_loading_type = 'SERIAL' + clim_soilw_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/clim_soilw.nc' + depvel_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/depvel_monthly.nc' + depvel_lnd_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/regrid_vegetation.nc' + drydep_srf_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mam/atmsrf_ne30pg2_200129.nc' + exo_coldens_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/phot/exo_coldens.nc' + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_elev_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + fstrat_list = ' ' + linoz_data_file = 'linoz_1850-2500_CMIP6_Hist_SSP370_10deg_58km_c20210202.nc' + linoz_data_path = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub' + linoz_data_type = 'INTERP_MISSING_MONTHS' + rsf_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc' + season_wes_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc' + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_surf_2015-2100_c210216.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_surf_2015-2100_c210216.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_surf_2015-2100_c210216.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_surf_2015-2100_c210216.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_surf_2015-2100_c210216.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_surf_2015-2100_c210216.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_surf_2015-2100_c210216.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_surf_2015-2100_c210216.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_SSP370_1.9x2.5_L70_2015-2100_c20211006.nc' + tracer_cnst_filelist = '' + tracer_cnst_specifier = 'cnst_O3:O3','OH','NO3','HO2' + tracer_cnst_type = 'INTERP_MISSING_MONTHS' + xactive_prates = .false. + xs_long_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/phot/temp_prs_GT200nm_JPL10_c130206.nc' +/ +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_2015-2500_c20210509.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ +&cldfrc2m_nl + cldfrc2m_rhmaxi = 1.05D0 + cldfrc2m_rhmini = 0.80D0 +/ +&cldfrc_nl + cldfrc_dp1 = 0.018D0 + cldfrc_dp2 = 500.0D0 + cldfrc_freeze_dry = .true. + cldfrc_ice = .true. + cldfrc_icecrit = 0.93D0 + cldfrc_iceopt = 5 + cldfrc_premib = 700.0D2 + cldfrc_premit = 25000.0D0 + cldfrc_rhminh = 0.800D0 + cldfrc_rhminl = 0.950D0 + cldfrc_rhminl_adj_land = 0.100D0 + cldfrc_sh1 = 0.04D0 + cldfrc_sh2 = 500.0D0 +/ +&cldsed_nl + cldsed_ice_stokes_fac = 1.0D0 +/ +&cldwat_nl + cldwat_conke = 5.0e-6 + cldwat_icritc = 18.0e-6 + cldwat_icritw = 2.0e-4 + cldwat_r3lcrit = 10.0e-6 +/ +&clubb_his_nl + clubb_history = .false. + clubb_rad_history = .false. +/ +&clubb_param_nl + clubb_c1 = 2.4 + clubb_c11 = 0.70 + clubb_c11b = 0.20 + clubb_c11c = 0.85 + clubb_c14 = 2.5D0 + clubb_c1b = 2.8 + clubb_c1c = 0.75 + clubb_c2rt = 1.75D0 + clubb_c6rtb = 7.50 + clubb_c6rtc = 0.50 + clubb_c6thlb = 7.50 + clubb_c6thlc = 0.50 + clubb_c8 = 5.2 + clubb_c_k10 = 0.35 + clubb_c_k10h = 0.35 + clubb_gamma_coef = 0.12D0 + clubb_gamma_coefb = 0.28D0 + clubb_gamma_coefc = 1.2 + clubb_mu = 0.0005 + clubb_wpxp_l_thresh = 100.0D0 +/ +&clubbpbl_diff_nl + clubb_cloudtop_cooling = .false. + clubb_expldiff = .true. + clubb_ice_deep = 14.e-6 + clubb_ice_sh = 50.e-6 + clubb_ipdf_call_placement = 2 + clubb_liq_deep = 8.e-6 + clubb_liq_sh = 10.e-6 + clubb_rainevap_turb = .false. + clubb_rnevap_effic = 1.2D0 + clubb_stabcorrect = .false. + clubb_timestep = 300.0D0 + clubb_tk1 = 268.15D0 + clubb_tk2 = 238.15D0 + clubb_use_sgv = .true. + relvar_fix = .true. +/ +&cospsimulator_nl + cosp_lite = .true. + docosp = .true. +/ +&ctl_nl + cubed_sphere_map = 2 + dt_remap_factor = 2 + dt_tracer_factor = 6 + hypervis_order = 2 + hypervis_scaling = 0 + hypervis_subcycle = 1 + hypervis_subcycle_q = 6 + hypervis_subcycle_tom = 1 + integration = "explicit" + mesh_file = '/dev/null' + nu = 1.0e15 + nu_div = -1.0 + nu_p = -1.0 + nu_q = -1.0 + nu_s = -1.0 + nu_top = 2.5e5 + qsplit = -1 + rsplit = -1 + se_ftype = 2 + se_limiter_option = 9 + se_ne = 30 + se_nsplit = -1 + se_partmethod = 4 + se_topology = "cube" + se_tstep = 300 + statefreq = 480 + theta_advect_form = 1 + theta_hydrostatic_mode = .true. + transport_alg = 12 + tstep_type = 5 + vert_remap_q_alg = 10 + vthreads = 1 +/ +&dust_nl + dust_emis_fact = 1.50D0 + soil_erod_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/dst/dst_1.9x2.5_c090203.nc' +/ +&gw_drag_nl + effgw_beres = 0.35 + effgw_cm = 1.D0 + effgw_oro = 0.375 + fcrit2 = 1.0 + frontgfc = 1.25D-15 + gw_convect_hcf = 10.0 + gw_dc = 2.5D0 + gw_drag_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/gw/newmfspectra40_dc25.nc' + pgwv = 32 + tau_0_ubc = .true. + taubgnd = 2.5D-3 +/ +&hetfrz_classnuc_nl + hist_hetfrz_classnuc = .false. +/ +&linoz_nl + linoz_psc_t = 197.5 +/ +&mam_mom_nl + mam_mom_cycle_yr = 1 + mam_mom_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mam/marine_BGC/' + mam_mom_datatype = 'CYCLICAL' + mam_mom_filename = 'monthly_macromolecules_0.1deg_bilinear_latlon_year01_merge_date.nc' + mam_mom_fixed_tod = 0 + mam_mom_fixed_ymd = 0 + mam_mom_mixing_state = 3 + mam_mom_specifier = 'chla:CHL1','mpoly:TRUEPOLYC','mprot:TRUEPROTC','mlip:TRUELIPC' +/ +µ_mg_nl + ice_sed_ai = 500.0 + micro_mg_berg_eff_factor = 0.7D0 + micro_mg_dcs = 195.D-6 + micro_mg_dcs_tdep = .true. + micro_mg_num_steps = 1 + micro_mg_precip_frac_method = 'in_cloud' + micro_mg_sub_version = 0 + micro_mg_version = 2 + micro_mincdnc = 10.D6 + microp_uniform = .false. +/ +µp_aero_nl + microp_aero_wsub_scheme = 1 + microp_aero_wsubmin = 0.001D0 +/ +&modal_aer_opt_nl + water_refindex_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/water_refindex_rrtmg_c080910.nc' +/ +&nucleate_ice_nl + nucleate_ice_subgrid = 1.2D0 + so4_sz_thresh_icenuc = 0.080e-6 + use_preexisting_ice = .false. +/ +&phys_ctl_nl + cam_chempkg = 'linoz_mam4_resus_mom_soag' + cam_physpkg = 'default' + cld_macmic_num_steps = 6 + cld_sed = 1.0D0 + conv_water_in_rad = 1 + convproc_do_aer = .true. + convproc_do_gas = .false. + convproc_method_activate = 2 + crm_accel_factor = 0 + crm_accel_uv = .false. + deep_scheme = 'ZM' + demott_ice_nuc = .true. + do_aerocom_ind3 = .false. + do_clubb_sgs = .true. + do_tms = .false. + eddy_scheme = 'CLUBB_SGS' + export_gustiness = .false. + fix_g1_err_ndrop = .true. + history_aero_optics = .true. + history_aerosol = .true. + history_amwg = .true. + history_budget = .false. + history_clubb = .false. + history_eddy = .false. + history_vdiag = .false. + history_verbose = .false. + history_waccm = .false. + ieflx_opt = 3 + linearize_pbl_winds = .false. + liqcf_fix = .true. + macrop_scheme = 'CLUBB_SGS' + mam_amicphys_optaa = 1 + mg_prc_coeff_fix = .true. + micro_do_icesupersat = .false. + micro_mg_accre_enhan_fac = 1.75D0 + microp_scheme = 'MG' + mmf_microphysics_scheme = 'sam1mom' + mmf_orientation_angle = 90.0 + mmf_vt_wn_max = 0 + n_so4_monolayers_pcage = 8.0D0 + prc_coef1 = 30500.0D0 + prc_exp = 3.19D0 + prc_exp1 = -1.40D0 + radiation_scheme = 'rrtmg' + regen_fix = .true. + resus_fix = .true. + rrtmg_temp_fix = .true. + shallow_scheme = 'CLUBB_SGS' + srf_flux_avg = 0 + ssalt_tuning = .true. + use_crm_accel = .false. + use_ecpp = .false. + use_gw_convect = .true. + use_gw_energy_fix = .true. + use_gw_front = .true. + use_gw_oro = .true. + use_hetfrz_classnuc = .true. + use_mmf = .false. + use_mmf_vt = .false. + use_subcol_microp = .false. + waccmx_opt = 'off' +/ +&prescribed_volcaero_nl + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc' + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_filetype = 'VOLC_CMIP6' + prescribed_volcaero_type = 'CYCLICAL' +/ +&rad_cnst_nl + icecldoptics = 'mitchell' + iceopticsfile = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/iceoptics_c080917.nc' + liqcldoptics = 'gammadist' + liqopticsfile = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/F_nwvl200_mu20_lam50_res64_t298_c080428.nc' + mode_defs = 'mam4_mode1:accum:=', 'A:num_a1:N:num_c1:num_mr:+', + 'A:so4_a1:N:so4_c1:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', 'A:pom_a1:N:pom_c1:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a1:N:soa_c1:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:bc_a1:N:bc_c1:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:dst_a1:N:dst_c1:dust:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', 'A:ncl_a1:N:ncl_c1:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a1:N:mom_c1:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode2:aitken:=', + 'A:num_a2:N:num_c2:num_mr:+', 'A:so4_a2:N:so4_c2:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:soa_a2:N:soa_c2:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:ncl_a2:N:ncl_c2:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a2:N:mom_c2:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode3:coarse:=', + 'A:num_a3:N:num_c3:num_mr:+', 'A:dst_a3:N:dst_c3:dust:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', + 'A:ncl_a3:N:ncl_c3:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', 'A:so4_a3:N:so4_c3:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:bc_a3:N:bc_c3:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', 'A:pom_a3:N:pom_c3:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a3:N:soa_c3:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:mom_a3:N:mom_c3:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', + 'mam4_mode4:primary_carbon:=', 'A:num_a4:N:num_c4:num_mr:+', + 'A:pom_a4:N:pom_c4:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', 'A:bc_a4:N:bc_c4:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:mom_a4:N:mom_c4:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc' + rad_climate = 'A:Q:H2O', 'N:O2:O2', 'N:CO2:CO2', + 'A:O3:O3', 'N:N2O:N2O', 'N:CH4:CH4', + 'N:CFC11:CFC11', 'N:CFC12:CFC12', 'M:mam4_mode1:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode1_rrtmg_aeronetdust_c141106.nc', + 'M:mam4_mode2:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode2_rrtmg_c130628.nc', 'M:mam4_mode3:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode3_rrtmg_aeronetdust_c141106.nc', 'M:mam4_mode4:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode4_rrtmg_c130628.nc' +/ +&radiation_nl + use_rad_dt_cosz = .true. +/ +&ref_pres_nl + clim_modal_aero_top_press = 1.D-4 + do_molec_press = 0.1D0 + molec_diff_bot_press = 50.D0 + trop_cloud_top_press = 1.D2 +/ +&solar_inparm + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850-2299_input4MIPS_c20181106.nc' + solar_data_type = 'SERIAL' + solar_htng_spctrl_scl = .true. +/ +&spmd_dyn_inparm + dyn_npes = 5400 +/ +&subcol_nl + subcol_scheme = 'off' +/ +&tropopause_nl + tropopause_climo_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc' +/ +&uwshcu_nl + uwshcu_rpen = 5.0 +/ +&vert_diff_nl + diff_cnsrv_mass_check = .false. + do_iss = .true. + eddy_lbulk_max = 40.D3 + eddy_leng_max = 40.D3 + eddy_max_bot_pressure = 100.D3 + eddy_moist_entrain_a2l = 30.D0 + kv_freetrop_scale = 1.D0 + kv_top_pressure = 0.D0 + kv_top_scale = 1.D0 +/ +&wetdep_inparm + gas_wetdep_list = 'H2O2','H2SO4','SO2' +/ +&zmconv_nl + zmconv_alfa = 0.14D0 + zmconv_c0_lnd = 0.0020 + zmconv_c0_ocn = 0.0020 + zmconv_cape_cin = 1 + zmconv_dmpdz = -0.7e-3 + zmconv_ke = 5.0E-6 + zmconv_mx_bot_lyr_adj = 1 + zmconv_tau = 3600 + zmconv_tiedke_add = 0.8D0 + zmconv_tp_fac = 2.0D0 + zmconv_trigdcape_ull = .true. +/ diff --git a/user_nl_files/HISTsmbb-notUsed/historical-forcing-files b/user_nl_files/HISTsmbb-notUsed/historical-forcing-files new file mode 100644 index 0000000..48659dd --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/historical-forcing-files @@ -0,0 +1,30 @@ + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_elev_1850-2014_c180205.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_soag_elev_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_elev_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_elev_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_elev_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_elev_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_elev_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + fstrat_list = ' ' + linoz_data_file = 'linoz1850-2015_2010JPL_CMIP6_10deg_58km_c20171109.nc' + linoz_data_path = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub' + linoz_data_type = 'INTERP_MISSING_MONTHS' + rsf_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc' + season_wes_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc' + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_1.9x2.5_L26_1850-2015_c20181106.nc' + tracer_cnst_filelist = '' + diff --git a/user_nl_files/HISTsmbb-notUsed/old/README.cesmBB b/user_nl_files/HISTsmbb-notUsed/old/README.cesmBB new file mode 100644 index 0000000..c66258f --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/old/README.cesmBB @@ -0,0 +1,25 @@ + + 'bc_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_bc_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'bc_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'DMS -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_DMS_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'DMS -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_DMS_other_surface_1750-2015_0.9x1.25_c20170322.nc', + 'num_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'num_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'num_a2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_bc_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_pom_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'pom_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_pom_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'pom_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SO2_anthro-ag-ship-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SO2_anthro-ene_surface_1750-2015_0.9x1.25_c20170616.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SO2_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'so4_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'so4_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'so4_a2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SOAGx1.5_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SOAGx1.5_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SOAGx1.5_biogenic_surface_1750-2015_0.9x1.25_c20170322.nc' + + diff --git a/user_nl_files/HISTsmbb-notUsed/old/README.smbb b/user_nl_files/HISTsmbb-notUsed/old/README.smbb new file mode 100644 index 0000000..20d3ca4 --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/old/README.smbb @@ -0,0 +1,11 @@ +-rw-r--r-- 1 nanr ccsm1 656158916 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_so4_a1_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 9725840 Apr 7 12:23 emissions-cmip6_smoothed_pom_a4_anthro_surface_1920cycling_ne30_c20211028.nc +-rw-r--r-- 1 nanr ccsm1 9726016 Apr 7 12:23 emissions-cmip6_smoothed_num_pom_a4_anthro_surface_1920cycling_ne30_c20211028.nc +-rw-r--r-- 1 nanr ccsm1 656158760 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_pom_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158880 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_bc_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158716 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_so4_a1_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158712 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SOAGx1.5_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158668 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SO2_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158748 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_DMS_bb_surface_mol_175001-210101_ne30_c20201016.nc +-rw-r--r-- 1 nanr ccsm1 656158896 Apr 7 12:23 emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_pom_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc + diff --git a/user_nl_files/HISTsmbb-notUsed/old/atm_in b/user_nl_files/HISTsmbb-notUsed/old/atm_in new file mode 100644 index 0000000..dc3a7a7 --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/old/atm_in @@ -0,0 +1,411 @@ +&aerosol_nl + aer_drydep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + aer_wetdep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + seasalt_emis_scale = 0.6 + sol_factb_interstitial = 0.1D0 + sol_facti_cloud_borne = 1.0D0 + sol_factic_interstitial = 0.4D0 + sscav_tuning = .true. +/ +&cam_history_nl + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + fincl1 = 'extinct_sw_inp', 'extinct_lw_bnd7', 'extinct_lw_inp', 'CLD_CAL', 'TREFMNAV', 'TREFMXAV', 'IEFLX', 'ZMDT', 'ZMDQ', + 'TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + fincl2 = 'FLUT', 'PRECT', 'U200', 'V200', 'U850', 'V850', 'Z500', 'OMEGA500', 'UBOT', 'VBOT', 'TREFHT', 'TREFHTMN:M', + 'TREFHTMX:X', 'QREFHT', 'TS', 'PS', 'TMQ', 'TUQ', 'TVQ', 'TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', + 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', + 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', + 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT', 'PRECC', 'PRECTMX:X', 'PSL', 'RHREFHT', 'U10', 'Z200', 'QRS', + 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', + 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700', + 'T500', 'T010', 'TBOT', 'Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010', 'TROPF_P', + 'TROPF_T', 'TROPF_Z' + fincl3 = 'PSL', 'T200', 'T500', 'U850', 'V850', 'UBOT', 'VBOT', 'TREFHT', 'Z700', 'TBOT:M', 'FLDS', 'FSDS', 'PRECT', + 'PS', 'QREFHT', 'TS', 'TMQ', 'U10', 'Z200:I', 'Z500:I', 'TTQ:I', 'TUQ:I', 'TVQ:I', 'Q:I', 'T:I', 'U:I', + 'V:I', 'Z3:I' + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + fincl6 = 'CLDTOT_ISCCP', 'MEANCLDALB_ISCCP', 'MEANTAU_ISCCP', 'MEANPTOP_ISCCP', 'MEANTB_ISCCP', 'CLDTOT_CAL', + 'CLDTOT_CAL_LIQ', 'CLDTOT_CAL_ICE', 'CLDTOT_CAL_UN', 'CLDHGH_CAL', 'CLDHGH_CAL_LIQ', 'CLDHGH_CAL_ICE', + 'CLDHGH_CAL_UN', 'CLDMED_CAL', 'CLDMED_CAL_LIQ', 'CLDMED_CAL_ICE', 'CLDMED_CAL_UN', 'CLDLOW_CAL', + 'CLDLOW_CAL_LIQ', 'CLDLOW_CAL_ICE', 'CLDLOW_CAL_UN' + fincl7 = 'O3','PS','TROP_P' + mfilt = 1,30,120,120,240,30,1 + nhtfrq = 0,-24,-6,-6,-3,-24,0 +/ +&cam_inparm + bnd_topo = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/topo/USGS-gtopo30_ne30np4pg2_16xdel2.c20200108.nc' + dtime = 1800 + ncdata = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/inic/homme/cami_mam3_Linoz_ne30np4_L72_c160214.nc' + phys_chnk_fdim_max = 30 + phys_chnk_fdim_mult = 2 + phys_loadbalance = 2 + print_energy_errors = .false. + raytau0 = 5.0D0 +/ +&chem_inparm + chlorine_loading_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/Linoz_Chlorine_Loading_CMIP6_0003-2017_c20171114.nc' + chlorine_loading_type = 'SERIAL' + clim_soilw_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/clim_soilw.nc' + depvel_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/depvel_monthly.nc' + depvel_lnd_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/regrid_vegetation.nc' + drydep_srf_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mam/atmsrf_ne30pg2_200129.nc' + exo_coldens_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/phot/exo_coldens.nc' + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_elev_1850-2014_c180205.nc', + 'SOAG -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_soag_elev_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_elev_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_elev_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_elev_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_elev_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_elev_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + fstrat_list = ' ' + linoz_data_file = 'linoz1850-2015_2010JPL_CMIP6_10deg_58km_c20171109.nc' + linoz_data_path = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub' + linoz_data_type = 'INTERP_MISSING_MONTHS' + rsf_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc' + season_wes_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc' + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + tracer_cnst_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_1.9x2.5_L26_1850-2015_c20181106.nc' + tracer_cnst_filelist = '' + tracer_cnst_specifier = 'cnst_O3:O3','OH','NO3','HO2' + tracer_cnst_type = 'INTERP_MISSING_MONTHS' + xactive_prates = .false. + xs_long_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/phot/temp_prs_GT200nm_JPL10_c130206.nc' +/ +&chem_surfvals_nl + bndtvghg = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP-1-2-0_Annual_Global_0000-2014_c20180105.nc' + co2vmr = 0.000001e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ +&cldfrc2m_nl + cldfrc2m_rhmaxi = 1.05D0 + cldfrc2m_rhmini = 0.80D0 +/ +&cldfrc_nl + cldfrc_dp1 = 0.018D0 + cldfrc_dp2 = 500.0D0 + cldfrc_freeze_dry = .true. + cldfrc_ice = .true. + cldfrc_icecrit = 0.93D0 + cldfrc_iceopt = 5 + cldfrc_premib = 700.0D2 + cldfrc_premit = 25000.0D0 + cldfrc_rhminh = 0.800D0 + cldfrc_rhminl = 0.950D0 + cldfrc_rhminl_adj_land = 0.100D0 + cldfrc_sh1 = 0.04D0 + cldfrc_sh2 = 500.0D0 +/ +&cldsed_nl + cldsed_ice_stokes_fac = 1.0D0 +/ +&cldwat_nl + cldwat_conke = 5.0e-6 + cldwat_icritc = 18.0e-6 + cldwat_icritw = 2.0e-4 + cldwat_r3lcrit = 10.0e-6 +/ +&clubb_his_nl + clubb_history = .false. + clubb_rad_history = .false. +/ +&clubb_param_nl + clubb_c1 = 2.4 + clubb_c11 = 0.70 + clubb_c11b = 0.20 + clubb_c11c = 0.85 + clubb_c14 = 2.5D0 + clubb_c1b = 2.8 + clubb_c1c = 0.75 + clubb_c2rt = 1.75D0 + clubb_c6rtb = 7.50 + clubb_c6rtc = 0.50 + clubb_c6thlb = 7.50 + clubb_c6thlc = 0.50 + clubb_c8 = 5.2 + clubb_c_k10 = 0.35 + clubb_c_k10h = 0.35 + clubb_gamma_coef = 0.12D0 + clubb_gamma_coefb = 0.28D0 + clubb_gamma_coefc = 1.2 + clubb_mu = 0.0005 + clubb_wpxp_l_thresh = 100.0D0 +/ +&clubbpbl_diff_nl + clubb_cloudtop_cooling = .false. + clubb_expldiff = .true. + clubb_ice_deep = 14.e-6 + clubb_ice_sh = 50.e-6 + clubb_ipdf_call_placement = 2 + clubb_liq_deep = 8.e-6 + clubb_liq_sh = 10.e-6 + clubb_rainevap_turb = .false. + clubb_rnevap_effic = 1.2D0 + clubb_stabcorrect = .false. + clubb_timestep = 300.0D0 + clubb_tk1 = 268.15D0 + clubb_tk2 = 238.15D0 + clubb_use_sgv = .true. + relvar_fix = .true. +/ +&cospsimulator_nl + cosp_lite = .true. + docosp = .true. +/ +&ctl_nl + cubed_sphere_map = 2 + dt_remap_factor = 2 + dt_tracer_factor = 6 + hypervis_order = 2 + hypervis_scaling = 0 + hypervis_subcycle = 1 + hypervis_subcycle_q = 6 + hypervis_subcycle_tom = 1 + integration = "explicit" + mesh_file = '/dev/null' + nu = 1.0e15 + nu_div = -1.0 + nu_p = -1.0 + nu_q = -1.0 + nu_s = -1.0 + nu_top = 2.5e5 + qsplit = -1 + rsplit = -1 + se_ftype = 2 + se_limiter_option = 9 + se_ne = 30 + se_nsplit = -1 + se_partmethod = 4 + se_topology = "cube" + se_tstep = 300 + statefreq = 480 + theta_advect_form = 1 + theta_hydrostatic_mode = .true. + transport_alg = 12 + tstep_type = 5 + vert_remap_q_alg = 10 + vthreads = 1 +/ +&dust_nl + dust_emis_fact = 1.50D0 + soil_erod_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/dst/dst_1.9x2.5_c090203.nc' +/ +&gw_drag_nl + effgw_beres = 0.35 + effgw_cm = 1.D0 + effgw_oro = 0.375 + fcrit2 = 1.0 + frontgfc = 1.25D-15 + gw_convect_hcf = 10.0 + gw_dc = 2.5D0 + gw_drag_file = '/global/cfs/cdirs/e3sm/inputdata/atm/waccm/gw/newmfspectra40_dc25.nc' + pgwv = 32 + tau_0_ubc = .true. + taubgnd = 2.5D-3 +/ +&hetfrz_classnuc_nl + hist_hetfrz_classnuc = .false. +/ +&linoz_nl + linoz_psc_t = 197.5 +/ +&mam_mom_nl + mam_mom_cycle_yr = 1 + mam_mom_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mam/marine_BGC/' + mam_mom_datatype = 'CYCLICAL' + mam_mom_filename = 'monthly_macromolecules_0.1deg_bilinear_latlon_year01_merge_date.nc' + mam_mom_fixed_tod = 0 + mam_mom_fixed_ymd = 0 + mam_mom_mixing_state = 3 + mam_mom_specifier = 'chla:CHL1','mpoly:TRUEPOLYC','mprot:TRUEPROTC','mlip:TRUELIPC' +/ +µ_mg_nl + ice_sed_ai = 500.0 + micro_mg_berg_eff_factor = 0.7D0 + micro_mg_dcs = 195.D-6 + micro_mg_dcs_tdep = .true. + micro_mg_num_steps = 1 + micro_mg_precip_frac_method = 'in_cloud' + micro_mg_sub_version = 0 + micro_mg_version = 2 + micro_mincdnc = 10.D6 + microp_uniform = .false. +/ +µp_aero_nl + microp_aero_wsub_scheme = 1 + microp_aero_wsubmin = 0.001D0 +/ +&modal_aer_opt_nl + water_refindex_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/water_refindex_rrtmg_c080910.nc' +/ +&nucleate_ice_nl + nucleate_ice_subgrid = 1.2D0 + so4_sz_thresh_icenuc = 0.080e-6 + use_preexisting_ice = .false. +/ +&phys_ctl_nl + cam_chempkg = 'linoz_mam4_resus_mom_soag' + cam_physpkg = 'default' + cld_macmic_num_steps = 6 + cld_sed = 1.0D0 + conv_water_in_rad = 1 + convproc_do_aer = .true. + convproc_do_gas = .false. + convproc_method_activate = 2 + crm_accel_factor = 0 + crm_accel_uv = .false. + deep_scheme = 'ZM' + demott_ice_nuc = .true. + do_aerocom_ind3 = .false. + do_clubb_sgs = .true. + do_tms = .false. + eddy_scheme = 'CLUBB_SGS' + export_gustiness = .false. + fix_g1_err_ndrop = .true. + history_aero_optics = .true. + history_aerosol = .true. + history_amwg = .true. + history_budget = .false. + history_clubb = .false. + history_eddy = .false. + history_vdiag = .false. + history_verbose = .false. + history_waccm = .false. + ieflx_opt = 3 + linearize_pbl_winds = .false. + liqcf_fix = .true. + macrop_scheme = 'CLUBB_SGS' + mam_amicphys_optaa = 1 + mg_prc_coeff_fix = .true. + micro_do_icesupersat = .false. + micro_mg_accre_enhan_fac = 1.75D0 + microp_scheme = 'MG' + mmf_microphysics_scheme = 'sam1mom' + mmf_orientation_angle = 90.0 + mmf_vt_wn_max = 0 + n_so4_monolayers_pcage = 8.0D0 + prc_coef1 = 30500.0D0 + prc_exp = 3.19D0 + prc_exp1 = -1.40D0 + radiation_scheme = 'rrtmg' + regen_fix = .true. + resus_fix = .true. + rrtmg_temp_fix = .true. + shallow_scheme = 'CLUBB_SGS' + srf_flux_avg = 0 + ssalt_tuning = .true. + use_crm_accel = .false. + use_ecpp = .false. + use_gw_convect = .true. + use_gw_energy_fix = .true. + use_gw_front = .true. + use_gw_oro = .true. + use_hetfrz_classnuc = .true. + use_mmf = .false. + use_mmf_vt = .false. + use_subcol_microp = .false. + waccmx_opt = 'off' +/ +&prescribed_volcaero_nl + prescribed_volcaero_datapath = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/volc' + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_1850-2014_v3_c20171205.nc' + prescribed_volcaero_filetype = 'VOLC_CMIP6' + prescribed_volcaero_type = 'SERIAL' +/ +&rad_cnst_nl + icecldoptics = 'mitchell' + iceopticsfile = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/iceoptics_c080917.nc' + liqcldoptics = 'gammadist' + liqopticsfile = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/F_nwvl200_mu20_lam50_res64_t298_c080428.nc' + mode_defs = 'mam4_mode1:accum:=', 'A:num_a1:N:num_c1:num_mr:+', + 'A:so4_a1:N:so4_c1:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', 'A:pom_a1:N:pom_c1:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a1:N:soa_c1:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:bc_a1:N:bc_c1:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:dst_a1:N:dst_c1:dust:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', 'A:ncl_a1:N:ncl_c1:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a1:N:mom_c1:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode2:aitken:=', + 'A:num_a2:N:num_c2:num_mr:+', 'A:so4_a2:N:so4_c2:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:soa_a2:N:soa_c2:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:ncl_a2:N:ncl_c2:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a2:N:mom_c2:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode3:coarse:=', + 'A:num_a3:N:num_c3:num_mr:+', 'A:dst_a3:N:dst_c3:dust:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', + 'A:ncl_a3:N:ncl_c3:seasalt:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', 'A:so4_a3:N:so4_c3:sulfate:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:bc_a3:N:bc_c3:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', 'A:pom_a3:N:pom_c3:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a3:N:soa_c3:s-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:mom_a3:N:mom_c3:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', + 'mam4_mode4:primary_carbon:=', 'A:num_a4:N:num_c4:num_mr:+', + 'A:pom_a4:N:pom_c4:p-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', 'A:bc_a4:N:bc_c4:black-c:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:mom_a4:N:mom_c4:m-organic:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc' + rad_climate = 'A:Q:H2O', 'N:O2:O2', 'N:CO2:CO2', + 'A:O3:O3', 'N:N2O:N2O', 'N:CH4:CH4', + 'N:CFC11:CFC11', 'N:CFC12:CFC12', 'M:mam4_mode1:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode1_rrtmg_aeronetdust_c141106.nc', + 'M:mam4_mode2:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode2_rrtmg_c130628.nc', 'M:mam4_mode3:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode3_rrtmg_aeronetdust_c141106.nc', 'M:mam4_mode4:/global/cfs/cdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode4_rrtmg_c130628.nc' +/ +&radiation_nl + use_rad_dt_cosz = .true. +/ +&ref_pres_nl + clim_modal_aero_top_press = 1.D-4 + do_molec_press = 0.1D0 + molec_diff_bot_press = 50.D0 + trop_cloud_top_press = 1.D2 +/ +&solar_inparm + solar_data_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/solar/Solar_1850-2299_input4MIPS_c20181106.nc' + solar_data_type = 'SERIAL' + solar_htng_spctrl_scl = .true. +/ +&spmd_dyn_inparm + dyn_npes = 5400 +/ +&subcol_nl + subcol_scheme = 'off' +/ +&tropopause_nl + tropopause_climo_file = '/global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc' +/ +&uwshcu_nl + uwshcu_rpen = 5.0 +/ +&vert_diff_nl + diff_cnsrv_mass_check = .false. + do_iss = .true. + eddy_lbulk_max = 40.D3 + eddy_leng_max = 40.D3 + eddy_max_bot_pressure = 100.D3 + eddy_moist_entrain_a2l = 30.D0 + kv_freetrop_scale = 1.D0 + kv_top_pressure = 0.D0 + kv_top_scale = 1.D0 +/ +&wetdep_inparm + gas_wetdep_list = 'H2O2','H2SO4','SO2' +/ +&zmconv_nl + zmconv_alfa = 0.14D0 + zmconv_c0_lnd = 0.0020 + zmconv_c0_ocn = 0.0020 + zmconv_cape_cin = 1 + zmconv_dmpdz = -0.7e-3 + zmconv_ke = 5.0E-6 + zmconv_mx_bot_lyr_adj = 1 + zmconv_tau = 3600 + zmconv_tiedke_add = 0.8D0 + zmconv_tp_fac = 2.0D0 + zmconv_trigdcape_ull = .true. +/ diff --git a/user_nl_files/HISTsmbb-notUsed/old/user_nl_eam b/user_nl_files/HISTsmbb-notUsed/old/user_nl_eam new file mode 100644 index 0000000..28ce4eb --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/old/user_nl_eam @@ -0,0 +1,70 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + srf_emis_specifier = 'DMS -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so2_surf_1850-2014_c180205.nc', + 'bc_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_bc_a4_surf_1850-2014_c180205.nc', + 'num_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a1_surf_1850-2014_c180205.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_surf_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a4_surf_1850-2014_c180205.nc', + 'pom_a4 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_pom_a4_surf_1850-2014_c180205.nc', + 'so4_a1 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a1_surf_1850-2014_c180205.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_surf_1850-2014_c180205.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + +people/nanr> ls e3sm/inputdata/atm/eam/chem/emis/emissions_ssp370-BB_smoothed/ +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_DMS_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SO2_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SOAGx1.5_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_bc_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_pom_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_so4_a1_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_pom_a4_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_so4_a1_bb_surface_mol_175001-210101_ne30_c20201016.nc +emissions-cmip6_smoothed_num_pom_a4_anthro_surface_1920cycling_ne30_c20211028.nc +emissions-cmip6_smoothed_pom_a4_anthro_surface_1920cycling_ne30_c20211028.nc + + + + 'bc_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_bc_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'bc_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'DMS -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_DMS_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'DMS -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_DMS_other_surface_1750-2015_0.9x1.25_c20170322.nc', + 'num_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'num_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'num_a2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_bc_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_pom_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'num_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_num_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'pom_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_pom_a4_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'pom_a4 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SO2_anthro-ag-ship-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SO2_anthro-ene_surface_1750-2015_0.9x1.25_c20170616.nc', + 'SO2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SO2_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'so4_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'so4_a1 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'so4_a2 -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20200924.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SOAGx1.5_anthro_surface_1750-2015_0.9x1.25_c20170608.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/emissions_ssp370-BB_smoothed/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_smoothed_SOAGx1.5_bb_surface_mol_175001-210101_0.9x1.25_c20201016.nc', + 'SOAG -> $INPUTDATA_ROOT/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SOAGx1.5_biogenic_surface_1750-2015_0.9x1.25_c20170322.nc' + + + diff --git a/user_nl_files/HISTsmbb-notUsed/user_nl_eam b/user_nl_files/HISTsmbb-notUsed/user_nl_eam new file mode 100644 index 0000000..d0d4b82 --- /dev/null +++ b/user_nl_files/HISTsmbb-notUsed/user_nl_eam @@ -0,0 +1,33 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + diff --git a/user_nl_files/SSP370smbb-notUsed/user_nl_eam b/user_nl_files/SSP370smbb-notUsed/user_nl_eam new file mode 100644 index 0000000..d0d4b82 --- /dev/null +++ b/user_nl_files/SSP370smbb-notUsed/user_nl_eam @@ -0,0 +1,33 @@ + !! h0, h1, h2, h3, h4, h5,h6 + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1, 30,120,120,240, 30, 1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + ! monthly (h0) A + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV','IEFLX','ZMDT','ZMDQ','TTEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' + ! daily (h1) A + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS','PS','TMQ','TUQ','TVQ','TOZ', 'FLDS','FLNS','FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT','PRECC','PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', 'U1000', 'U700', 'U500', 'U200', 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', 'T1000', 'T850', 'T700','T500','T010','TBOT','Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010','TROPF_P','TROPF_T','TROPF_Z' + ! 6hourly (h2) I + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M','FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS','TMQ','U10','Z200:I','Z500:I','TTQ:I','TUQ:I','TVQ:I','Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + ! 6hourly (h3) A + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + ! 3hourly (h4) A + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + ! daily (h5) A + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE','CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE','CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + ! monthly (h6) I + fincl7 = 'O3', 'PS', 'TROP_P' + + ext_frc_specifier = 'SO2 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so2_elev_1850-2100_c221016.nc', + 'SOAG -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_soag_elev_1850-2100_c221016.nc', + 'bc_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_bc_a4_elev_1850-2100_c221016.nc', + 'num_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a1_elev_1850-2100_c221016.nc', + 'num_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_num_a2_elev_1850-2014_c180205.nc', + 'num_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_num_a4_elev_1850-2100_c221016.nc', + 'pom_a4 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_pom_a4_elev_1850-2100_c221016.nc', + 'so4_a1 -> /global/cfs/cdirs/ccsm1/people/nanr/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30-smoothed/cmip6_ssp370_mam4_smoothed_so4_a1_elev_1850-2100_c221016.nc', + 'so4_a2 -> /global/cfs/cdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne30/cmip6_mam4_so4_a2_elev_1850-2014_c180205.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + + + diff --git a/user_nl_files/le-hist/README.hf-output b/user_nl_files/le-hist/README.hf-output new file mode 100644 index 0000000..a732a51 --- /dev/null +++ b/user_nl_files/le-hist/README.hf-output @@ -0,0 +1,43 @@ + +https://docs.google.com/document/d/1MU-c3Z0o9rZToCz7uQ8Z6pGzBRnDBYyzvRH8NXvXIk0/edit + +E3SM output needed for detailed MJO analysis as well as looking at heat-waves and MOV: + + +6-hourly: (preferred for TCs): +UBOT, VBOT, U10, Z200, Z500, PSL (Hui for TCs) +IVT, vIVT, uIVT, TMQ,U850,V850, PRECT, PS (just in case we want to look at ARs; Need mods in cam_diagnostics or otherwise full field U, V, and Q) + +Nan: needs source code for E3SM + + + +Daily Mean 2D: + +Done: OLR, PS, PSL, PRECT, evaporation/surface latent flux, surface sensible heat flux, surface longwave radiation, surface shortwave radiation, TOA shortwave radiation (Kai) +Done: FLNT = OLR +Done: LHFLX = latent surface heat flux +Done: SHFLX = surface sensible heat flux +Done: FLNS = net longwave @ the surface +Done: FSNS = net shortwave @ the surface +Done: FSNT = TOA shortwav + + +Done: 'TREFHT', 'TREFHTMN:M', 'TREFHTMX:X', 'PRECT', ‘PRECTMX’, ‘RHREFHT’ (Mari, Katie, Maria extremes) + +Done: FLUT, PRECT, U200, V200, U850, V850 (MJO diagnostics from MDTF package) + + +Daily Mean 3D: +U, V, omega, T, specific humidity, and geopotential height (Kai) +Get Kai to + +LE levels: + 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', + 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', + 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', + 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', + 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', + 'OMEGA1000:A', 'OMEGA850:A', 'OMEGA700:A', 'OMEGA500:A', 'OMEGA200:A', 'OMEGA100:A', 'OMEGA050:A', 'OMEGA010:A' + + diff --git a/user_nl_files/le-hist/h6 b/user_nl_files/le-hist/h6 new file mode 100644 index 0000000..15e7a66 --- /dev/null +++ b/user_nl_files/le-hist/h6 @@ -0,0 +1,14 @@ +UBOT +VBOT +U10 +Z200 +Z500 +PSL +IVT +vIVT +uIVT +TMQ +U850 +V850 +PRECT + diff --git a/user_nl_files/le-hist/h6s b/user_nl_files/le-hist/h6s new file mode 100644 index 0000000..292ef2d --- /dev/null +++ b/user_nl_files/le-hist/h6s @@ -0,0 +1,14 @@ + +IVT +PRECT +PSL +TMQ +U10 +U850 +UBOT +uIVT +V850 +VBOT +vIVT +Z200 +Z500 diff --git a/user_nl_files/le-hist/lists/dh6 b/user_nl_files/le-hist/lists/dh6 new file mode 100644 index 0000000..8afded4 --- /dev/null +++ b/user_nl_files/le-hist/lists/dh6 @@ -0,0 +1,13 @@ +PSL +TREFHT +FLDS +FSDS +PRECT +PS +QREFHT +Q +TS +T +U +V +Z3 diff --git a/user_nl_files/le-hist/lists/dh6s b/user_nl_files/le-hist/lists/dh6s new file mode 100644 index 0000000..06594c9 --- /dev/null +++ b/user_nl_files/le-hist/lists/dh6s @@ -0,0 +1,13 @@ +FLDS +FSDS +PRECT +PS +PSL +Q +QREFHT +T +TREFHT +TS +U +V +Z3 diff --git a/user_nl_files/le-hist/user_nl_cam-SamS b/user_nl_files/le-hist/user_nl_cam-SamS new file mode 100644 index 0000000..2590dc0 --- /dev/null +++ b/user_nl_files/le-hist/user_nl_cam-SamS @@ -0,0 +1,14 @@ +!! h0, h1,h2 + nhtfrq = 0,-24, -6 + mfilt = 1, 30,120,120,240,30 + avgflag_pertape = 'A','A','I','A','A','A' + + +! monthly (h0)(A) + fincl1 = 'IEFLX' + +! daily (h1) (A) + fincl2 = 'FLUT','TREFHTMN','TREFHTMX','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! 6hrly (h2) (I) + fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' diff --git a/user_nl_files/le-hist/user_nl_cam-mbr1 b/user_nl_files/le-hist/user_nl_cam-mbr1 new file mode 100644 index 0000000..e69f7e2 --- /dev/null +++ b/user_nl_files/le-hist/user_nl_cam-mbr1 @@ -0,0 +1,33 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + +! monthly (h0) + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', + 'UBOT:A','VBOT:A','U10:A', 'Z200:A', 'Z500:A', 'U200:A', 'V200:A', 'U850:A', 'V850:A', 'U050:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', +! 'UBOT:A', 'VBOT:A', 'U10:A', 'Z200:A', 'Z500:A', 'U200:A', 'V200:A', 'U850:A', 'V850:A' , +! 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', +! 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', +! 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', +! 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', +! 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) + fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'IVT:I','uIVT:I','vIVT:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) + fincl6 = 'PRECC:I', 'PRECT:I' diff --git a/user_nl_files/le-hist/user_nl_cam-mbr2-3 b/user_nl_files/le-hist/user_nl_cam-mbr2-3 new file mode 100644 index 0000000..ff6d6f6 --- /dev/null +++ b/user_nl_files/le-hist/user_nl_cam-mbr2-3 @@ -0,0 +1,33 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + +! monthly (h0) + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', + 'UBOT:A','VBOT:A','U10:A', 'Z200:A', 'Z500:A', 'U200:A', 'V200:A', 'U850:A', 'V850:A', 'U050:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', + 'UBOT:A', 'VBOT:A', 'U10:A', 'Z200:A', 'Z500:A', 'U200:A', 'V200:A', 'U850:A', 'V850:A' , + 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', + 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', + 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', + 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', + 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) +!! fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'IVT:I','uIVT:I','vIVT:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) +!! fincl6 = 'PRECC:I', 'PRECT:I' diff --git a/user_nl_files/le-v1-catalyst/user_nl_cam-mbr1 b/user_nl_files/le-v1-catalyst/user_nl_cam-mbr1 new file mode 100644 index 0000000..bba9361 --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_cam-mbr1 @@ -0,0 +1,49 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + + !xxxx nhtfrq = 0,-24,-6 + !xxxx mfilt = 1,30,120,120,240,30 + !xxxx avgflag_pertape = 'A','A','I','A','A','A' + !xxxx fincl1 = 'IEFLX' + !xxxx fincl2 = 'FLUT','TREFHTMN','TREFHTMX','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! monthly (h0) + fexcl1 = 'LINOZ_SZA','extinct_lw_bnd7','extinct_lw_inp','extinct_sw_inp','hstobie_linoz','LINOZ_DO3', + 'LINOZ_DO3_PSC','LINOZ_O3CLIM','LINOZ_O3COL','LINOZ_SSO3','UU','VQ','VU','VV','bc_a1_SRF', + 'bc_a3_SRF','bc_a4_SRF','chla','dst_a1_SRF','mlip','mom_a1_SRF','mom_a2_SRF','mom_a3_SRF', + 'mom_a4_SRF','mpoly','mprot','ncl_a1_SRF','ncl_a2_SRF','ncl_a3_SRF','num_a1_SRF','num_a2_SRF', + 'num_a3_SRF','num_a4_SRF','pom_a1_SRF','pom_a3_SRF','pom_a4_SRF','so4_a1_SRF','so4_a2_SRF','so4_a3_SRF', + 'soa_a1_SRF','soa_a2_SRF','soa_a3_SRF' + + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', 'U10:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', +! 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', +! 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', +! 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', +! 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', +! 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) + fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +!xxxx Sam: 6 hrly instant +!xxxx fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'IVT:I','uIVT:I','vIVT:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) + fincl6 = 'PRECC:I', 'PRECT:I' + +history_aerosol=.false. +ncdata= '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.i.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-catalyst/user_nl_cam-mbr2-3 b/user_nl_files/le-v1-catalyst/user_nl_cam-mbr2-3 new file mode 100644 index 0000000..18be439 --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_cam-mbr2-3 @@ -0,0 +1,50 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + + !SamStevensonRuns + !xxxx nhtfrq = 0,-24,-6 + !xxxx mfilt = 1,30,120,120,240,30 + !xxxx avgflag_pertape = 'A','A','I','A','A','A' + !xxxx fincl1 = 'IEFLX' + !xxxx fincl2 = 'FLUT','TREFHTMN','TREFHTMX','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! monthly (h0) + fexcl1 = 'LINOZ_SZA','extinct_lw_bnd7','extinct_lw_inp','extinct_sw_inp','hstobie_linoz','LINOZ_DO3', + 'LINOZ_DO3_PSC','LINOZ_O3CLIM','LINOZ_O3COL','LINOZ_SSO3','UU','VQ','VU','VV','bc_a1_SRF', + 'bc_a3_SRF','bc_a4_SRF','chla','dst_a1_SRF','mlip','mom_a1_SRF','mom_a2_SRF','mom_a3_SRF', + 'mom_a4_SRF','mpoly','mprot','ncl_a1_SRF','ncl_a2_SRF','ncl_a3_SRF','num_a1_SRF','num_a2_SRF', + 'num_a3_SRF','num_a4_SRF','pom_a1_SRF','pom_a3_SRF','pom_a4_SRF','so4_a1_SRF','so4_a2_SRF','so4_a3_SRF', + 'soa_a1_SRF','soa_a2_SRF','soa_a3_SRF' + + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', 'U10:A', + 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', + 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', + 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', + 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', + 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) +! fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +!xxxx Sam: 6 hrly instant +!xxxx fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'IVT:I','uIVT:I','vIVT:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) +! fincl6 = 'PRECC:I', 'PRECT:I' + +history_aerosol=.false. +ncdata= '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.i.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-catalyst/user_nl_clm b/user_nl_files/le-v1-catalyst/user_nl_clm new file mode 100644 index 0000000..5146f3f --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_clm @@ -0,0 +1,30 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! +! Include namelist variables for drv_flds_in ONLY if -megan and/or -drydep options +! are set in the CLM_NAMELIST_OPTS env variable. +! +! EXCEPTIONS: +! Set use_cndv by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting +! Set use_vichydro by the compset you use and the CLM_BLDNML_OPTS -vichydro setting +! Set use_cn by the compset you use and CLM_BLDNML_OPTS -bgc setting +! Set use_crop by the compset you use and CLM_BLDNML_OPTS -crop setting +! Set spinup_state by the CLM_BLDNML_OPTS -bgc_spinup setting +! Set irrigate by the CLM_BLDNML_OPTS -irrig setting +! Set co2_ppmv with CCSM_CO2_PPMV option +! Set dtime with L_NCPL option +! Set fatmlndfrc with LND_DOMAIN_PATH/LND_DOMAIN_FILE options +! Set finidat with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases +! (includes $inst_string for multi-ensemble cases) +! Set glc_grid with CISM_GRID option +! Set glc_smb with GLC_SMB option +! Set maxpatch_glcmec with GLC_NEC option +! Set glc_do_dynglacier with GLC_TWO_WAY_COUPLING env variable +!---------------------------------------------------------------------------------- + + check_finidat_year_consistency = .false. + hist_nhtfrq = 0, -24 + hist_fincl2 = 'FSNO', 'H2OSNO', 'QRUNOFF', 'RAIN', 'SNOW', 'SOILWATER_10CM', 'VOLR', 'FCEV', 'FCTR', 'FGEV', 'FGR', 'FSH', 'FSM', 'QOVER', 'QSNOMELT' + finidat = ' ' + finidat_interp_source = '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0200-01-01-00000.nc' diff --git a/user_nl_files/user_nl_cpl b/user_nl_files/le-v1-catalyst/user_nl_cpl similarity index 97% rename from user_nl_files/user_nl_cpl rename to user_nl_files/le-v1-catalyst/user_nl_cpl index 7646e5e..a209536 100644 --- a/user_nl_files/user_nl_cpl +++ b/user_nl_files/le-v1-catalyst/user_nl_cpl @@ -17,4 +17,3 @@ ! For example, rather than set username to 'foo' in user_nl_cpl, call ! ./xmlchange USER=foo !------------------------------------------------------------------------ -histaux_l2x1yrg = .true. diff --git a/user_nl_files/le-v1-catalyst/user_nl_mosart b/user_nl_files/le-v1-catalyst/user_nl_mosart new file mode 100644 index 0000000..77a98c6 --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_mosart @@ -0,0 +1,12 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! NOTE: namelist variable coupling_period CAN ONLY be changed by modifying the value +! of the xml variable ROF_NCPL in env_run.xml +! NOTE: if the xml variable ROF GRID in env_build.xml is set to 'null', then +! the RTM build-namelist will set do_rtm to .false. - and will ignore +! any change below +!---------------------------------------------------------------------------------- + + +finidat_rtm = '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.mosart.r.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-catalyst/user_nl_mpascice b/user_nl_files/le-v1-catalyst/user_nl_mpascice new file mode 100644 index 0000000..310ee9d --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_mpascice @@ -0,0 +1,17 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpassi_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpassi/streams.seaice +!---------------------------------------------------------------------------------- + diff --git a/user_nl_files/le-v1-catalyst/user_nl_mpaso b/user_nl_files/le-v1-catalyst/user_nl_mpaso new file mode 100644 index 0000000..8912b49 --- /dev/null +++ b/user_nl_files/le-v1-catalyst/user_nl_mpaso @@ -0,0 +1,22 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpaso_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpaso/streams.ocean_forward +!---------------------------------------------------------------------------------- + + config_btr_dt = '0000_00:01:00' + config_mom_del4 = 1.2e11 + config_use_mom_del2 = .true. + config_mom_del2 = 1000.0 + config_dt = '00:30:00' diff --git a/user_nl_files/le-v1-sam/user_nl_cam b/user_nl_files/le-v1-sam/user_nl_cam new file mode 100644 index 0000000..642edcc --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_cam @@ -0,0 +1,25 @@ +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value + + nhtfrq = 0, -24, -6 + mfilt = 1, 30, 120, 120, 240, 30 + avgflag_pertape = 'A', 'A', 'I', 'A', 'A', 'A' + + fexcl1 = 'LINOZ_SZA','extinct_lw_bnd7','extinct_lw_inp','extinct_sw_inp','hstobie_linoz', + 'LINOZ_DO3','LINOZ_DO3_PSC','LINOZ_O3CLIM','LINOZ_O3COL','LINOZ_SSO3','UU','VQ', + 'VU','VV','bc_a1_SRF','bc_a3_SRF','bc_a4_SRF','chla','dst_a1_SRF','mlip','mom_a1_SRF', + 'mom_a2_SRF','mom_a3_SRF','mom_a4_SRF','mpoly','mprot','ncl_a1_SRF','ncl_a2_SRF', + 'ncl_a3_SRF','num_a1_SRF','num_a2_SRF','num_a3_SRF','num_a4_SRF','pom_a1_SRF', + 'pom_a3_SRF','pom_a4_SRF','so4_a1_SRF','so4_a2_SRF','so4_a3_SRF','soa_a1_SRF','soa_a2_SRF','soa_a3_SRF' + +! monthly (averaged) + fincl1 = 'IEFLX' + +! daily (averaged) + fincl2 = 'FLUT','TREFHTMN:N','TREFHTMX:X','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! 6 hrly (instantaneous) + fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' + + history_aerosol=.false. + ncdata= '/compyfs/vanr302/E3SM_simulations/init/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.i.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-sam/user_nl_clm b/user_nl_files/le-v1-sam/user_nl_clm new file mode 100644 index 0000000..3a7c2f4 --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_clm @@ -0,0 +1,30 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! +! Include namelist variables for drv_flds_in ONLY if -megan and/or -drydep options +! are set in the CLM_NAMELIST_OPTS env variable. +! +! EXCEPTIONS: +! Set use_cndv by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting +! Set use_vichydro by the compset you use and the CLM_BLDNML_OPTS -vichydro setting +! Set use_cn by the compset you use and CLM_BLDNML_OPTS -bgc setting +! Set use_crop by the compset you use and CLM_BLDNML_OPTS -crop setting +! Set spinup_state by the CLM_BLDNML_OPTS -bgc_spinup setting +! Set irrigate by the CLM_BLDNML_OPTS -irrig setting +! Set co2_ppmv with CCSM_CO2_PPMV option +! Set dtime with L_NCPL option +! Set fatmlndfrc with LND_DOMAIN_PATH/LND_DOMAIN_FILE options +! Set finidat with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases +! (includes $inst_string for multi-ensemble cases) +! Set glc_grid with CISM_GRID option +! Set glc_smb with GLC_SMB option +! Set maxpatch_glcmec with GLC_NEC option +! Set glc_do_dynglacier with GLC_TWO_WAY_COUPLING env variable +!---------------------------------------------------------------------------------- + + check_finidat_year_consistency = .false. + hist_nhtfrq = 0, -24 + hist_fincl2 = 'FSNO', 'H2OSNO', 'QRUNOFF', 'RAIN', 'SNOW', 'SOILWATER_10CM', 'VOLR', 'FCEV', 'FCTR', 'FGEV', 'FGR', 'FSH', 'FSM', 'QOVER', 'QSNOMELT' + finidat = ' ' + finidat_interp_source = '/compyfs/vanr302/E3SM_simulations/init/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-sam/user_nl_cpl b/user_nl_files/le-v1-sam/user_nl_cpl new file mode 100644 index 0000000..a209536 --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_cpl @@ -0,0 +1,19 @@ +!------------------------------------------------------------------------ +! Users should ONLY USE user_nl_cpl to change namelists variables +! for namelist variables in drv_in (except for the ones below) and +! any keyword/values in seq_maps.rc +! Users should add ALL user specific namelist and seq_maps.rc changes below +! using the following syntax +! namelist_var = new_namelist_value +! or +! mapname = new_map_name +! For example to change the default value of ocn2atm_fmapname to 'foo' use +! ocn2atm_fmapname = 'foo' +! +! Note that some namelist variables MAY NOT be changed in user_nl_cpl - +! they are defined in a $CASEROOT xml file and must be changed with +! xmlchange. +! +! For example, rather than set username to 'foo' in user_nl_cpl, call +! ./xmlchange USER=foo +!------------------------------------------------------------------------ diff --git a/user_nl_files/le-v1-sam/user_nl_mosart b/user_nl_files/le-v1-sam/user_nl_mosart new file mode 100644 index 0000000..c24fe78 --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_mosart @@ -0,0 +1,12 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! NOTE: namelist variable coupling_period CAN ONLY be changed by modifying the value +! of the xml variable ROF_NCPL in env_run.xml +! NOTE: if the xml variable ROF GRID in env_build.xml is set to 'null', then +! the RTM build-namelist will set do_rtm to .false. - and will ignore +! any change below +!---------------------------------------------------------------------------------- + + +finidat_rtm = '/compyfs/vanr302/E3SM_simulations/init/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.mosart.r.0200-01-01-00000.nc' diff --git a/user_nl_files/le-v1-sam/user_nl_mpascice b/user_nl_files/le-v1-sam/user_nl_mpascice new file mode 100644 index 0000000..310ee9d --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_mpascice @@ -0,0 +1,17 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpassi_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpassi/streams.seaice +!---------------------------------------------------------------------------------- + diff --git a/user_nl_files/le-v1-sam/user_nl_mpaso b/user_nl_files/le-v1-sam/user_nl_mpaso new file mode 100644 index 0000000..8912b49 --- /dev/null +++ b/user_nl_files/le-v1-sam/user_nl_mpaso @@ -0,0 +1,22 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpaso_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpaso/streams.ocean_forward +!---------------------------------------------------------------------------------- + + config_btr_dt = '0000_00:01:00' + config_mom_del4 = 1.2e11 + config_use_mom_del2 = .true. + config_mom_del2 = 1000.0 + config_dt = '00:30:00' diff --git a/user_nl_files/le-v2-golaz/user_nl_cam b/user_nl_files/le-v2-golaz/user_nl_cam new file mode 100644 index 0000000..e53b81c --- /dev/null +++ b/user_nl_files/le-v2-golaz/user_nl_cam @@ -0,0 +1,81 @@ +!!! h0, h1, h2, h3, h4, h5, h6 +!!! f1, f2, f3, f4, f5, f6, f7 +!!pertape 'A','A','I','A','A','A','I' + + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + +=================================== +!! h0 monthly (A) V2-LE +=================================== + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + +NEW fincl1 ======================== + [v1-LE] 'IEFLX' + [CATALYST] 'ZMDT', 'ZMDQ', 'STEND_CLUBB', 'RVMTEND_CLUBB', 'MPDT', 'MPDQ', 'DCQ', 'DTCOND' +END fincl1 ======================== + +=================================== +!! h1 daily (A) +=================================== + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS', + 'PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', + 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', + 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + +ADD to fincl2 ======================== + [v1-LE] 'PRECC', 'FLNSC', 'FLNS' + + [CATALYST] 'PRECTMX:X','PSL','RHREFHT', 'U10', 'Z200', 'QRS', 'QRL', + 'Q1000', 'Q850', 'Q700', 'Q500', 'Q200', 'Q100', 'Q050', 'Q010', 'QBOT:A', + 'U1000', 'U700', 'U500', 'U200', + 'V1000', 'V700', 'V500', 'V100', 'V050', 'V010', 'VBOT', + 'T1000', 'T850', 'T700', 'T500', 'T010', 'TBOT', + 'Z1000', 'Z850', 'Z700', 'Z500', 'Z200', 'Z100', 'Z050', 'Z010', + [SourceMods] 'TROPF_P','TROPF_T','TROPF_Z', +END fincl2 ======================== + + +=================================== +!! h2 6hrly (I) + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' +=================================== + +!!! 2D +ADD to fincl3: + [v1-LE] 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'TS' + [CATALYST] 'TMQ','U10','Z200:I','Z500:I', + [SourceMods] 'IVT:I','uIVT:I','vIVT:I' + +!!! 3D + [v1-LE] 'Q', 'T', 'U', 'V', 'Z3' + +=================================== +!! h3 6hrly (A) + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' +=================================== + +=================================== +!! h4 3hrly (A) + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' +=================================== + +ADD 3 hrly : 'PRECC:I', 'PRECT:I' + +=================================== +!! h5 daily (A) + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE', + 'CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE', + 'CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' +=================================== + +=================================== +!! h6 monthly (I) + fincl7 = 'O3', 'PS', 'TROP_P' +=================================== + +! Additional retuning + clubb_tk1 = 268.15D0 + gw_convect_hcf = 10.0 diff --git a/user_nl_files/le-v2-golaz/user_nl_cam-golazWithFormat b/user_nl_files/le-v2-golaz/user_nl_cam-golazWithFormat new file mode 100644 index 0000000..5fe19db --- /dev/null +++ b/user_nl_files/le-v2-golaz/user_nl_cam-golazWithFormat @@ -0,0 +1,38 @@ +!!! h0, h1, h2, h3, h4, h5, h6 +!!! f1, f2, f3, f4, f5, f6, f7 +!!pertape 'A','A','I','A','A','A','I' + + nhtfrq = 0,-24, -6, -6, -3,-24, 0 + mfilt = 1,30,120,120,240,30,1 + avgflag_pertape = 'A','A','I','A','A','A','I' + fexcl1 = 'CFAD_SR532_CAL', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', 'LINOZ_O3COL', 'LINOZ_SSO3', 'hstobie_linoz' + +!! h0 monthly (A) + fincl1 = 'extinct_sw_inp','extinct_lw_bnd7','extinct_lw_inp','CLD_CAL', 'TREFMNAV', 'TREFMXAV' + +!! h1 daily (A) + fincl2 = 'FLUT','PRECT','U200','V200','U850','V850','Z500','OMEGA500','UBOT','VBOT','TREFHT','TREFHTMN:M','TREFHTMX:X','QREFHT','TS', + 'PS','TMQ','TUQ','TVQ','TOZ', 'FLDS', 'FLNS', 'FSDS', 'FSNS', 'SHFLX', 'LHFLX', 'TGCLDCWP', 'TGCLDIWP', 'TGCLDLWP', 'CLDTOT', + 'T250', 'T200', 'T150', 'T100', 'T050', 'T025', 'T010', 'T005', 'T002', 'T001', 'TTOP', 'U250', 'U150', 'U100', 'U050', 'U025', 'U010', + 'U005', 'U002', 'U001', 'UTOP', 'FSNT', 'FLNT' + +!! h2 6hrly (I) + fincl3 = 'PSL','T200','T500','U850','V850','UBOT','VBOT','TREFHT', 'Z700', 'TBOT:M' + +!! h3 6hrly (A) + fincl4 = 'FLUT','U200','U850','PRECT','OMEGA500' + +!! h4 3hrly (A) + fincl5 = 'PRECT','PRECC','TUQ','TVQ','QFLX','SHFLX','U90M','V90M' + +!! h5 daily (A) + fincl6 = 'CLDTOT_ISCCP','MEANCLDALB_ISCCP','MEANTAU_ISCCP','MEANPTOP_ISCCP','MEANTB_ISCCP','CLDTOT_CAL','CLDTOT_CAL_LIQ','CLDTOT_CAL_ICE', + 'CLDTOT_CAL_UN','CLDHGH_CAL','CLDHGH_CAL_LIQ','CLDHGH_CAL_ICE','CLDHGH_CAL_UN','CLDMED_CAL','CLDMED_CAL_LIQ','CLDMED_CAL_ICE', + 'CLDMED_CAL_UN','CLDLOW_CAL','CLDLOW_CAL_LIQ','CLDLOW_CAL_ICE','CLDLOW_CAL_UN' + +!! h6 monthly (I) + fincl7 = 'O3', 'PS', 'TROP_P' + +! Additional retuning + clubb_tk1 = 268.15D0 + gw_convect_hcf = 10.0 diff --git a/user_nl_files/ssp370-v1/fromSam/README.casesOnCori b/user_nl_files/ssp370-v1/fromSam/README.casesOnCori new file mode 100644 index 0000000..7286c1e --- /dev/null +++ b/user_nl_files/ssp370-v1/fromSam/README.casesOnCori @@ -0,0 +1,13 @@ +e3smv1-le/e3smv1.20TR_CMIP6.ne30_oECv3_ICG.LE-v1.0210.001> cd /global/homes/s/slgs/E3SM_cases/ +Directory: /global/homes/s/slgs/E3SM_cases +slgs/E3SM_cases> ls -ltr +total 125 +drwxr-xr-x 8 slgs slgs 4096 Dec 11 12:00 test +drwxr-xr-x 3 slgs slgs 512 Dec 11 15:20 cases_notworking +-rwxr-xr-x 1 slgs slgs 75318 Jan 7 11:03 casediff_ssp370vs585_en5 +drwxr-xr-x 9 slgs slgs 4096 Jan 8 00:10 20200403.en5.LE_ens_ssp585.ne30_oECv3_ICG.compy +drwxr-xr-x 9 slgs slgs 4096 Jan 16 12:02 20200403.en6.LE_ens_ssp370.ne30_oECv3_ICG.compy +drwxr-xr-x 9 slgs slgs 4096 Jan 16 19:05 20200403.en5.LE_ens_ssp370.ne30_oECv3_ICG.compy +drwxr-xr-x 9 slgs slgs 8192 Jan 20 09:45 20200706.init.290.en10.LE_ens_ssp370.ne30_oECv3_ICG.compy + +COMPSET: A_WCYCLSSP370_CMIP6 diff --git a/user_nl_files/ssp370-v1/fromSam/atm_in b/user_nl_files/ssp370-v1/fromSam/atm_in new file mode 100644 index 0000000..5fdb1a2 --- /dev/null +++ b/user_nl_files/ssp370-v1/fromSam/atm_in @@ -0,0 +1,364 @@ +&aerosol_nl + aer_drydep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + aer_wetdep_list = 'bc_a1', 'bc_a3', 'bc_a4', 'dst_a1', 'dst_a3', 'mom_a1', 'mom_a2', 'mom_a3', 'mom_a4', 'ncl_a1', 'ncl_a2', + 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', 'num_a4', 'pom_a1', 'pom_a3', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', + 'soa_a1', 'soa_a2', 'soa_a3' + seasalt_emis_scale = 0.85 + sol_factb_interstitial = 0.1D0 + sol_facti_cloud_borne = 1.0D0 + sol_factic_interstitial = 0.4D0 + sscav_tuning = .true. +/ +&cam_history_nl + fexcl1 = 'LINOZ_SZA', 'extinct_lw_bnd7', 'extinct_lw_inp', 'extinct_sw_inp', 'hstobie_linoz', 'LINOZ_DO3', 'LINOZ_DO3_PSC', 'LINOZ_O3CLIM', + 'LINOZ_O3COL', 'LINOZ_SSO3', 'UU', 'VQ', 'VU', 'VV', 'bc_a1_SRF', 'bc_a3_SRF', + 'bc_a4_SRF', 'chla', 'dst_a1_SRF', 'mlip', 'mom_a1_SRF', 'mom_a2_SRF', 'mom_a3_SRF', 'mom_a4_SRF', + 'mpoly', 'mprot', 'ncl_a1_SRF', 'ncl_a2_SRF', 'ncl_a3_SRF', 'num_a1_SRF', 'num_a2_SRF', 'num_a3_SRF', + 'num_a4_SRF', 'pom_a1_SRF', 'pom_a3_SRF', 'pom_a4_SRF', 'so4_a1_SRF', 'so4_a2_SRF', 'so4_a3_SRF', 'soa_a1_SRF', + 'soa_a2_SRF', 'soa_a3_SRF' + fincl1 = 'IEFLX:A', 'ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + fincl2 = 'FLUT:A', 'TREFHT:A', 'TREFHTMN:M', 'TREFHTMX:X', 'LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X', 'PRECC:A', 'FLNT:A', + 'FSNS:A', 'FSNT:A', 'FLNSC:A', 'FLNS:A', 'PS:A', 'PSL:A', 'FLNT:A', 'RHREFHT:A', 'QRS:A', 'QRL:A', + 'TROPF_P:A', 'TROPF_T:A', 'TROPF_Z:A', 'U10:A', 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', + 'Q050:A', 'Q010:A', 'QBOT:A', 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', + 'U010:A', 'UBOT:A', 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', + 'VBOT:A', 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', + 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A' + fincl4 = 'PSL:I', 'TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', 'TMQ:I', 'U10:I', 'U850:I', + 'V850:I', 'UBOT:I', 'VBOT:I', 'Z200:I', 'Z500:I', 'IVT:I', 'uIVT:I', 'vIVT:I' + fincl5 = 'Q:I','T:I','U:I','V:I','Z3:I' + mfilt = 1,365,73,73,20,73 + nhtfrq = 0,-24,-24,-6,-6,-3 +/ +&cam_inparm + bnd_topo = '/project/projectdirs/e3sm/inputdata/atm/cam/topo/USGS-gtopo30_ne30np4_16xdel2-PFC-consistentSGH.nc' + dtime = 1800 + ncdata = '20200706.init.290.en10.LE_ens.ne30_oECv3_ICG.compy.cam.i.2015-01-01-00000.nc' + phys_loadbalance = 2 + print_energy_errors = .false. + raytau0 = 5.0D0 + use_rad_dt_cosz = .true. +/ +&carma_nl + carma_model = 'none' +/ +&chem_inparm + chlorine_loading_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/Linoz_Chlorine_Loading_CMIP6_Hist_SSP370_0003-2503_c20210202.nc' + chlorine_loading_type = 'SERIAL' + clim_soilw_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/clim_soilw.nc' + depvel_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/depvel_monthly.nc' + depvel_lnd_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/regrid_vegetation.nc' + drydep_srf_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mam/atmsrf_ne30np4_110920.nc' + exo_coldens_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/phot/exo_coldens.nc' + ext_frc_specifier = 'SO2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_elev_2015-2100_c210216.nc', + 'SOAG -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_soag_elev_2015-2100_c210216.nc', + 'bc_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_elev_2015-2100_c210216.nc', + 'num_a1 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_elev_2015-2100_c210216.nc', + 'num_a2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_elev_2015-2100_c210216.nc', + 'num_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_elev_2015-2100_c210216.nc', + 'pom_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_elev_2015-2100_c210216.nc', + 'so4_a1 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_elev_2015-2100_c210216.nc', + 'so4_a2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_elev_2015-2100_c210216.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + fstrat_list = ' ' + linoz_data_file = 'linoz_1850-2500_CMIP6_Hist_SSP370_10deg_58km_c20210202.nc' + linoz_data_path = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub' + linoz_data_type = 'INTERP_MISSING_MONTHS' + rsf_file = '/project/projectdirs/e3sm/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc' + season_wes_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc' + srf_emis_specifier = 'DMS -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/DMSflux.1850-2100.1deg_latlon_conserv.POPmonthlyClimFromACES4BGC_c20160727.nc', + 'SO2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so2_surf_2015-2100_c210216.nc', + 'bc_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_bc_a4_surf_2015-2100_c210216.nc', + 'num_a1 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a1_surf_2015-2100_c210216.nc', + 'num_a2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a2_surf_2015-2100_c210216.nc', + 'num_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_num_a4_surf_2015-2100_c210216.nc', + 'pom_a4 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_pom_a4_surf_2015-2100_c210216.nc', + 'so4_a1 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a1_surf_2015-2100_c210216.nc', + 'so4_a2 -> /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/emis/CMIP6_SSP370_ne30/cmip6_ssp370_mam4_so4_a2_surf_2015-2100_c210216.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + tracer_cnst_datapath = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart_aero/oxid' + tracer_cnst_file = 'oxid_SSP370_1.9x2.5_L70_2015-2100_c20211006.nc' + tracer_cnst_filelist = '' + tracer_cnst_specifier = 'cnst_O3:O3','OH','NO3','HO2' + tracer_cnst_type = 'INTERP_MISSING_MONTHS' + xactive_prates = .false. + xs_long_file = '/project/projectdirs/e3sm/inputdata/atm/waccm/phot/temp_prs_GT200nm_JPL10_c130206.nc' +/ +&chem_surfvals_nl + bndtvghg = '/project/projectdirs/e3sm/inputdata/atm/cam/ggas/GHG_CMIP_SSP370-1-2-1_Annual_Global_2015-2500_c20210509.nc' + co2vmr = 379.000000e-6 + flbc_list = ' ' + scenario_ghg = 'RAMPED' +/ +&cldfrc2m_nl + cldfrc2m_rhmaxi = 1.05D0 + cldfrc2m_rhmini = 0.80D0 +/ +&cldfrc_nl + cldfrc_dp1 = 0.045D0 + cldfrc_dp2 = 500.0D0 + cldfrc_freeze_dry = .true. + cldfrc_ice = .true. + cldfrc_icecrit = 0.93D0 + cldfrc_iceopt = 5 + cldfrc_premib = 700.0D2 + cldfrc_premit = 25000.0D0 + cldfrc_rhminh = 0.800D0 + cldfrc_rhminl = 0.950D0 + cldfrc_rhminl_adj_land = 0.100D0 + cldfrc_sh1 = 0.04D0 + cldfrc_sh2 = 500.0D0 +/ +&cldsed_nl + cldsed_ice_stokes_fac = 1.0D0 +/ +&cldwat_nl + cldwat_conke = 5.0e-6 + cldwat_icritc = 18.0e-6 + cldwat_icritw = 2.0e-4 + cldwat_r3lcrit = 10.0e-6 +/ +&clubb_his_nl + clubb_history = .false. + clubb_rad_history = .false. +/ +&clubb_param_nl + clubb_c1 = 1.335 + clubb_c14 = 1.06D0 + clubb_c2rt = 1.75D0 + clubb_c8 = 4.3 + clubb_c_k10 = 0.3 + clubb_gamma_coef = 0.32 + clubb_wpxp_l_thresh = 60.0D0 +/ +&clubbpbl_diff_nl + clubb_cloudtop_cooling = .false. + clubb_expldiff = .true. + clubb_ice_deep = 16.e-6 + clubb_ice_sh = 50.e-6 + clubb_liq_deep = 8.e-6 + clubb_liq_sh = 10.e-6 + clubb_rainevap_turb = .false. + clubb_rnevap_effic = 1.2D0 + clubb_stabcorrect = .false. + clubb_timestep = 300.0D0 + clubb_tk1 = 268.15D0 + clubb_tk2 = 238.15D0 + relvar_fix = .true. +/ +&cospsimulator_nl + cosp_lite = .true. +/ +&ctl_nl + hypervis_order = 2 + hypervis_scaling = 0 + hypervis_subcycle = 3 + hypervis_subcycle_q = 1 + integration = "explicit" + mesh_file = '/dev/null' + nu = 1.0e15 + nu_div = 2.5e15 + nu_p = 1.0e15 + nu_q = -1.0 + nu_top = 2.5e5 + qsplit = 1 + rsplit = 3 + se_ftype = 2 + se_limiter_option = 8 + se_ne = 30 + se_nsplit = 2 + se_partmethod = 4 + se_phys_tscale = 0 + se_topology = "cube" + statefreq = 480 + tstep_type = 5 + vert_remap_q_alg = 1 + vthreads = 1 +/ +&dust_nl + dust_emis_fact = 2.05D0 + soil_erod_file = '/project/projectdirs/e3sm/inputdata/atm/cam/dst/dst_1.9x2.5_c090203.nc' +/ +&gw_drag_nl + effgw_beres = 0.4 + effgw_cm = 1.D0 + effgw_oro = 0.25 + fcrit2 = 1.0 + frontgfc = 1.25D-15 + gw_dc = 2.5D0 + gw_drag_file = '/project/projectdirs/e3sm/inputdata/atm/waccm/gw/newmfspectra40_dc25.nc' + pgwv = 32 + tau_0_ubc = .true. + taubgnd = 2.5D-3 +/ +&hetfrz_classnuc_nl + hist_hetfrz_classnuc = .false. +/ +&hkconv_nl + hkconv_c0 = 1.0e-4 + hkconv_cmftau = 1800.0D0 +/ +&mam_mom_nl + mam_mom_cycle_yr = 1 + mam_mom_datapath = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mam/marine_BGC/' + mam_mom_datatype = 'CYCLICAL' + mam_mom_filename = 'monthly_macromolecules_0.1deg_bilinear_latlon_year01_merge_date.nc' + mam_mom_fixed_tod = 0 + mam_mom_fixed_ymd = 0 + mam_mom_mixing_state = 3 + mam_mom_specifier = 'chla:CHL1','mpoly:TRUEPOLYC','mprot:TRUEPROTC','mlip:TRUELIPC' +/ +µ_mg_nl + ice_sed_ai = 500.0 + micro_mg_berg_eff_factor = 0.1D0 + micro_mg_dcs = 195.D-6 + micro_mg_dcs_tdep = .true. + micro_mg_num_steps = 1 + micro_mg_precip_frac_method = 'in_cloud' + micro_mg_sub_version = 0 + micro_mg_version = 2 + microp_uniform = .false. +/ +µp_aero_nl + microp_aero_wsub_scheme = 1 +/ +&modal_aer_opt_nl + water_refindex_file = '/project/projectdirs/e3sm/inputdata/atm/cam/physprops/water_refindex_rrtmg_c080910.nc' +/ +&nucleate_ice_nl + nucleate_ice_subgrid = 1.2D0 + so4_sz_thresh_icenuc = 0.05e-6 + use_preexisting_ice = .false. +/ +&phys_ctl_nl + cam_chempkg = 'linoz_mam4_resus_mom_soag' + cam_physpkg = 'cam5' + cld_macmic_num_steps = 6 + cld_sed = 1.0D0 + conv_water_in_rad = 1 + convproc_do_aer = .true. + convproc_do_gas = .false. + convproc_method_activate = 2 + deep_scheme = 'ZM' + demott_ice_nuc = .true. + do_aerocom_ind3 = .false. + do_clubb_sgs = .true. + do_tms = .false. + eddy_scheme = 'CLUBB_SGS' + fix_g1_err_ndrop = .true. + history_aero_optics = .true. + history_aerosol = .false. + history_amwg = .true. + history_budget = .false. + history_clubb = .false. + history_eddy = .false. + history_vdiag = .false. + history_verbose = .false. + history_waccm = .false. + ieflx_opt = 2 + liqcf_fix = .true. + macrop_scheme = 'CLUBB_SGS' + mam_amicphys_optaa = 1 + mg_prc_coeff_fix = .true. + micro_do_icesupersat = .false. + micro_mg_accre_enhan_fac = 1.5D0 + microp_scheme = 'MG' + n_so4_monolayers_pcage = 8.0D0 + prc_coef1 = 30500.0D0 + prc_exp = 3.19D0 + prc_exp1 = -1.2D0 + radiation_scheme = 'rrtmg' + regen_fix = .true. + resus_fix = .true. + rrtmg_temp_fix = .true. + shallow_scheme = 'CLUBB_SGS' + srf_flux_avg = 0 + ssalt_tuning = .true. + use_gw_convect = .true. + use_gw_front = .true. + use_gw_oro = .true. + use_hetfrz_classnuc = .true. + use_subcol_microp = .false. + waccmx_opt = 'off' +/ +&prescribed_volcaero_nl + prescribed_volcaero_cycle_yr = 1 + prescribed_volcaero_datapath = '/project/projectdirs/e3sm/inputdata/atm/cam/volc' + prescribed_volcaero_file = 'CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc' + prescribed_volcaero_filetype = 'VOLC_CMIP6' + prescribed_volcaero_type = 'CYCLICAL' +/ +&rad_cnst_nl + icecldoptics = 'mitchell' + iceopticsfile = '/project/projectdirs/e3sm/inputdata/atm/cam/physprops/iceoptics_c080917.nc' + liqcldoptics = 'gammadist' + liqopticsfile = '/project/projectdirs/e3sm/inputdata/atm/cam/physprops/F_nwvl200_mu20_lam50_res64_t298_c080428.nc' + mode_defs = 'mam4_mode1:accum:=', 'A:num_a1:N:num_c1:num_mr:+', + 'A:so4_a1:N:so4_c1:sulfate:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', 'A:pom_a1:N:pom_c1:p-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a1:N:soa_c1:s-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:bc_a1:N:bc_c1:black-c:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:dst_a1:N:dst_c1:dust:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/dust4_rrtmg_c090521.nc:+', 'A:ncl_a1:N:ncl_c1:seasalt:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a1:N:mom_c1:m-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode2:aitken:=', + 'A:num_a2:N:num_c2:num_mr:+', 'A:so4_a2:N:so4_c2:sulfate:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:soa_a2:N:soa_c2:s-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:ncl_a2:N:ncl_c2:seasalt:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:mom_a2:N:mom_c2:m-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', 'mam4_mode3:coarse:=', + 'A:num_a3:N:num_c3:num_mr:+', 'A:dst_a3:N:dst_c3:dust:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/dust4_rrtmg_c090521.nc:+', + 'A:ncl_a3:N:ncl_c3:seasalt:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', 'A:so4_a3:N:so4_c3:sulfate:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', + 'A:bc_a3:N:bc_c3:black-c:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', 'A:pom_a3:N:pom_c3:p-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a3:N:soa_c3:s-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:mom_a3:N:mom_c3:m-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc', + 'mam4_mode4:primary_carbon:=', 'A:num_a4:N:num_c4:num_mr:+', + 'A:pom_a4:N:pom_c4:p-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', 'A:bc_a4:N:bc_c4:black-c:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:mom_a4:N:mom_c4:m-organic:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/poly_rrtmg_c130816.nc' + rad_climate = 'A:Q:H2O', 'N:O2:O2', 'N:CO2:CO2', + 'A:O3:O3', 'N:N2O:N2O', 'N:CH4:CH4', + 'N:CFC11:CFC11', 'N:CFC12:CFC12', 'M:mam4_mode1:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode1_rrtmg_c130628.nc', + 'M:mam4_mode2:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode2_rrtmg_c130628.nc', 'M:mam4_mode3:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode3_rrtmg_c130628.nc', 'M:mam4_mode4:/project/projectdirs/e3sm/inputdata/atm/cam/physprops/mam4_mode4_rrtmg_c130628.nc' +/ +&ref_pres_nl + clim_modal_aero_top_press = 1.D-4 + do_molec_press = 0.1D0 + molec_diff_bot_press = 50.D0 + trop_cloud_top_press = 1.D2 +/ +&solar_inparm + solar_data_file = '/project/projectdirs/e3sm/inputdata/atm/cam/solar/Solar_1850-2299_input4MIPS_c20181106.nc' + solar_data_type = 'SERIAL' + solar_htng_spctrl_scl = .true. +/ +&spmd_dyn_inparm + dyn_npes = 5400 +/ +&subcol_nl + subcol_scheme = 'off' +/ +&tropopause_nl + tropopause_climo_file = '/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc' +/ +&uwshcu_nl + uwshcu_rpen = 5.0 +/ +&vert_diff_nl + diff_cnsrv_mass_check = .false. + do_iss = .true. + eddy_lbulk_max = 40.D3 + eddy_leng_max = 40.D3 + eddy_max_bot_pressure = 100.D3 + eddy_moist_entrain_a2l = 30.D0 + kv_freetrop_scale = 1.D0 + kv_top_pressure = 0.D0 + kv_top_scale = 1.D0 +/ +&wetdep_inparm + gas_wetdep_list = 'H2O2','H2SO4','SO2' +/ +&zmconv_nl + zmconv_c0_lnd = 0.007 + zmconv_c0_ocn = 0.007 + zmconv_cape_cin = 1 + zmconv_dmpdz = -0.7e-3 + zmconv_ke = 5.e-6 + zmconv_mx_bot_lyr_adj = 2 + zmconv_tau = 3600 + zmconv_tiedke_add = 0.8D0 +/ diff --git a/user_nl_files/ssp370-v1/fromSam/atm_in-cesm-cmip6.101 b/user_nl_files/ssp370-v1/fromSam/atm_in-cesm-cmip6.101 new file mode 100644 index 0000000..4d0ce32 --- /dev/null +++ b/user_nl_files/ssp370-v1/fromSam/atm_in-cesm-cmip6.101 @@ -0,0 +1,389 @@ +&aerosol_nl + aer_drydep_list = 'bc_a1', 'bc_a4', 'dst_a1', 'dst_a2', 'dst_a3', 'ncl_a1', 'ncl_a2', 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', + 'num_a4', 'pom_a1', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', 'soa_a1', 'soa_a2' + aer_wetdep_list = 'bc_a1', 'bc_a4', 'dst_a1', 'dst_a2', 'dst_a3', 'ncl_a1', 'ncl_a2', 'ncl_a3', 'num_a1', 'num_a2', 'num_a3', + 'num_a4', 'pom_a1', 'pom_a4', 'so4_a1', 'so4_a2', 'so4_a3', 'soa_a1', 'soa_a2' + modal_accum_coarse_exch = .true. + seasalt_emis_scale = 1.00D0 + sol_factb_interstitial = 0.1D0 + sol_facti_cloud_borne = 1.0D0 + sol_factic_interstitial = 0.4D0 +/ +&aircraft_emit_nl + aircraft_datapath = '/glade/p/cesmdata/cseg/inputdata/atm/cam/ggas' + aircraft_specifier = 'ac_CO2 -> emissions-cmip6_CO2_anthro_ac_ssp370_201401-210112_fv_0.9x1.25_c20190207.txt' + aircraft_type = 'SERIAL' +/ +&blj_nl + do_beljaars = .true. +/ +&cam_history_nl + fincl1 = 'ABSORB:A', 'ACTREL:A', 'AODABSdn:A', 'AODBCdn:A', 'AODdnDUST1:A', 'AODdnDUST2:A', 'AODdnDUST3:A', 'AODdnMODE1:A', + 'AODdnMODE2:A', 'AODdnMODE3:A', 'AODDUST2:A', 'AODDUST:A', 'AODNIRstdn:A', 'AODPOMdn:A', 'AODSO4dn:A', 'AODSOAdn:A', + 'AODSSdn:A', 'AODUVdn:A', 'AODUVstdn:A', 'AODVIS:A', 'AODVISdn:A', 'AODVISstdn:A', 'AQ_SO2:A', 'AREA:A', + 'AREI:A', 'AREL:A', 'bc_a1:A', 'bc_a1DDF:A', 'bc_a1SFWET:A', 'bc_a4:A', 'bc_a4DDF:A', 'bc_a4SFWET:A', + 'bc_c1:A', 'bc_c1DDF:A', 'bc_c1SFWET:A', 'bc_c4:A', 'bc_c4DDF:A', 'bc_c4SFWET:A', 'BROX:A', 'BROY:A', + 'BURDENBCdn', 'BURDENDUSTdn:A', 'BURDENPOMdn:A', 'BURDENSEASALTdn:A', 'BURDENSO4dn:A', 'BURDENSOAdn:A', 'CDNUMC:A', 'CLDICE:A', + 'CLDLIQ:A', 'CLDTOT:A', 'CLOUD:A', 'CLOX:A', 'CLOY:A', 'CME:A', 'CMFDQ:A', 'CMFMC:A', + 'CMFMCDZM:A', 'CO2:A', 'CONCLD:A', 'DF_H2O2:A', 'DF_H2SO4:A', 'DF_SO2:A', 'dgnumwet1:A', 'dgnumwet2:A', + 'dgnumwet3:A', 'DH2O2CHM:A', 'DMS:A', 'dry_deposition_NHx_as_N:A', 'dry_deposition_NOy_as_N:A', 'Dso4_a1CHM:A', 'Dso4_a2CHM:A', 'Dso4_a3CHM:A', + 'dst_a1:A', 'dst_a1DDF:A', 'dst_a1SFWET:A', 'dst_a2:A', 'dst_a2DDF:A', 'dst_a2SFWET:A', 'dst_a3:A', 'dst_a3DDF:A', + 'dst_a3SFWET:A', 'dst_c1:A', 'dst_c1DDF:A', 'dst_c1SFWET:A', 'dst_c2:A', 'dst_c2DDF:A', 'dst_c2SFWET:A', 'dst_c3:A', + 'dst_c3DDF:A', 'dst_c3SFWET:A', 'DTCORE:A', 'EVAPPREC:A', 'EVAPQZM:A', 'EVAPTZM:A', 'EXTINCTdn:A', 'EXTINCTNIRdn:A', + 'EXTINCTUVdn:A', 'EXTxASYMdn:A', 'FCTL:A', 'FLDS:A', 'FLDSC:A', 'FLNR:A', 'FLNS:A', 'FLNSC:A', + 'FLNT:A', 'FLNTC:A', 'FLUT:A', 'FLUTC:A', 'FREQI:A', 'FREQL:A', 'FREQZM:A', 'FSDS:A', + 'FSDSC:A', 'FSNR:A', 'FSNS:A', 'FSNSC:A', 'FSNT:A', 'FSNTC:A', 'FSNTOAC:A', 'FSUTOA:A', + 'GS_SO2:A', 'H2O2:A', 'H2O:A', 'H2SO4:A', 'H2SO4M_C:A', 'H2SO4_sfnnuc1:A', 'HCL_GAS:A', 'HNO3_GAS:A', + 'HNO3_NAT:A', 'HNO3_STS:A', 'HO2:A', 'ICEFRAC:A', 'jh2o2:A', 'KVH_CLUBB:A', 'LANDFRAC:A', 'LHFLX:A', + 'MASS:A', 'ncl_a1:A', 'ncl_a1DDF:A', 'ncl_a1SFWET:A', 'ncl_a2:A', 'ncl_a2DDF:A', 'ncl_a2SFWET:A', 'ncl_a3:A', + 'ncl_a3DDF:A', 'ncl_a3SFWET:A', 'ncl_c1:A', 'ncl_c1DDF:A', 'ncl_c1SFWET:A', 'ncl_c2:A', 'ncl_c2DDF:A', 'ncl_c2SFWET:A', + 'ncl_c3:A', 'ncl_c3DDF:A', 'ncl_c3SFWET:A', 'NITROP_PD:A', 'NO3:A', 'NOX:A', 'NOY:A', 'num_a1:A', + 'num_a1_CLXF:A', 'num_a1DDF:A', 'num_a2:A', 'num_a2_CLXF:A', 'num_a2DDF:A', 'num_a2_sfnnuc1:A', 'num_a3:A', 'num_a3DDF:A', + 'num_a4:A', 'num_a4DDF:A', 'num_c1:A', 'num_c1DDF:A', 'num_c2:A', 'num_c2DDF:A', 'num_c3:A', 'num_c3DDF:A', + 'num_c4:A', 'num_c4DDF:A', 'NUMLIQ:A', 'O3:A', 'OH:A', 'OMEGA:A', 'OMEGAT:A', 'PBLH:A', + 'PDELDRY:A', 'PHIS:A', 'pom_a1:A', 'pom_a1DDF:A', 'pom_a1SFWET:A', 'pom_a4:A', 'pom_a4DDF:A', 'pom_a4SFWET:A', + 'pom_c1:A', 'pom_c1DDF:A', 'pom_c1SFWET:A', 'pom_c4:A', 'pom_c4DDF:A', 'pom_c4SFWET:A', 'PRECC:A', 'PRECT:A', + 'PS:A', 'PSL:A', 'PTEQ:A', 'PTTEND:A', 'Q:A', 'QFLX:A', 'QRAIN:A', 'QREFHT:A', + 'QRL:A', 'QRLC:A', 'QRS:A', 'QRSC:A', 'QSNOW:A', 'RAD_ICE:A', 'RAD_LNAT:A', 'RAD_SULFC:A', + 'REFF_AERO:A', 'RELHUM:A', 'RHREFHT:A', 'SAD_AERO:A', 'SAD_ICE:A', 'SAD_LNAT:A', 'SAD_SULFC:A', 'SAD_TROP:A', + 'SFbc_a4:A', 'SFDMS:A', 'SFdst_a1:A', 'SFdst_a2:A', 'SFdst_a3:A', 'SFncl_a1:A', 'SFncl_a2:A', 'SFncl_a3:A', + 'SFnum_a1:A', 'SFnum_a2:A', 'SFnum_a3:A', 'SFpom_a4:A', 'SFSO2:A', 'SFso4_a1:A', 'SFso4_a2:A', 'SHFLX:A', + 'SO2:A', 'SO2_CHML:A', 'SO2_CHMP:A', 'SO2_CLXF:A', 'SO2_XFRC:A', 'so4_a1:A', 'so4_a1_CHMP:A', 'so4_a1_CLXF:A', + 'so4_a1DDF:A', 'so4_a1_sfgaex1:A', 'so4_a1SFWET:A', 'so4_a2:A', 'so4_a2_CHMP:A', 'so4_a2_CLXF:A', 'so4_a2DDF:A', 'so4_a2_sfgaex1:A', + 'so4_a2_sfnnuc1:A', 'so4_a2SFWET:A', 'so4_a3:A', 'so4_a3DDF:A', 'so4_a3_sfgaex1:A', 'so4_a3SFWET:A', 'so4_c1:A', 'so4_c1AQH2SO4:A', + 'so4_c1AQSO4:A', 'so4_c1DDF:A', 'so4_c1SFWET:A', 'so4_c2:A', 'so4_c2AQH2SO4:A', 'so4_c2AQSO4:A', 'so4_c2DDF:A', 'so4_c2SFWET:A', + 'so4_c3:A', 'so4_c3AQH2SO4:A', 'so4_c3AQSO4:A', 'so4_c3DDF:A', 'so4_c3SFWET:A', 'SOLIN:A', 'SOLLD:A', 'SOLSD:A', + 'SSAVIS:A', 'SST:A', 'T:A', 'TAQ:A', 'TAUBLJX:A', 'TAUBLJY:A', 'TAUGWX:A', 'TAUGWY:A', + 'TAUX:A', 'TAUY:A', 'TBRY:A', 'TCLY:A', 'TGCLDIWP:A', 'TGCLDLWP:A', 'TMDMS:A', 'TMQ:A', + 'TMSO2:A', 'TMso4_a1:A', 'TMso4_a2:A', 'TMso4_a3:A', 'TOT_CLD_VISTAU:A', 'TOTH:A', 'TREFHT:A', 'TREFHTMN:M', + 'TREFHTMX:X', 'TROP_P:A', 'TROP_T:A', 'TROP_Z:A', 'TS:A', 'TSMN:M', 'TSMX:X', 'TTEND_TOT:A', + 'TTGWORO:A', 'U10:A', 'U:A', 'UTGWORO:A', 'UU:A', 'V:A', 'VD01:A', 'VV:A', + 'WD_H2O2:A', 'WD_H2SO4:A', 'WD_SO2:A', 'wet_deposition_NHx_as_N:A', 'wet_deposition_NOy_as_N:A', 'Z3:A', 'ZMDQ:A', 'ZMDT:A', + 'ZMMTT:A', 'ZMMU:A', 'MSKtem', 'PS', 'PSL', 'VTHzm', 'UVzm', 'UWzm', + 'Uzm', 'Vzm', 'THzm', 'Wzm', 'PHIS' + fincl2 = 'ACTNL:A', 'ACTREL:A', 'BURDENBCdn:A', 'BURDENDUSTdn:A', 'BURDENPOMdn:A', 'BURDENSEASALTdn:A', 'BURDENSO4dn:A', 'BURDENSOAdn:A', 'CDNUMC:A', + 'CLDICE:A', 'CLDLIQ:A', 'CLDTOT:A', 'CLOUD:A', 'CMFMC:A', 'CMFMCDZM:A', 'FCTL:A', 'FLDS:A', 'FLDSC:A', + 'FLNR:A', 'FLNS:A', 'FLNSC:A', 'FLNT:A', 'FLNTC:A', 'FLUT:A', 'FLUTC:A', 'FSDS:A', 'FSDSC:A', + 'FSNR:A', 'FSNS:A', 'FSNSC:A', 'FSNTOA:A', 'FSNTOAC:A', 'LHFLX:A', 'MASS:A', 'OMEGA:A', 'OMEGA500:A', + 'PBLH:A', 'PDELDRY:A', 'PRECC:A', 'PRECT:A', 'PS:A', 'PSL:A', 'Q:A', 'QREFHT:A', 'QSNOW:A', + 'RELHUM:A', 'RHREFHT:A', 'SHFLX:A', 'SOLIN:A', 'SOLLD:A', 'SOLSD:A', 'T:A', 'T500:A', 'T700:A', + 'T850:A', 'TAUBLJX:A', 'TAUBLJY:A', 'TAUGWX:A', 'TAUGWY:A', 'TAUX:A', 'TAUY:A', 'TGCLDIWP:A', 'TGCLDLWP:A', + 'TMQ:A', 'TREFHT:A', 'TREFHTMN:M', 'TREFHTMX:X', 'TS:A', 'TSMN:M', 'TSMX:X', 'U:A', 'U10:A', + 'UTGWORO:A', 'V:A', 'Z3:A', 'Z500:A' + fincl7 = 'MSKtem','PS','PSL','VTHzm','UVzm','UWzm','Uzm','Vzm','THzm','Wzm','PHIS' + mfilt = 1,5,20,40,120,240,365,73,365 + ndens = 2,2,2,2,2,2,1,1,1 + nhtfrq = 0,-24,-6,-3,-1,1,-24,-120,-240 +/ +&cam_initfiles_nl + bnd_topo = '/glade/p/cesmdata/cseg/inputdata/atm/cam/topo/fv_0.9x1.25_nc3000_Nsw042_Nrs008_Co060_Fi001_ZR_sgh30_24km_GRNL_c170103.nc' + ncdata = 'b.e21.BHIST.f09_g17.CMIP6-historical.010_v2.cam.i.2015-01-01-00000.nc' + use_topo_file = .true. +/ +&check_energy_nl + print_energy_errors = .false. +/ +&chem_inparm + chem_use_chemtrop = .true. + clim_soilw_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/dvel/clim_soilw.nc' + depvel_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/dvel/depvel_monthly.nc' + depvel_lnd_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/dvel/regrid_vegetation.nc' + exo_coldens_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/phot/exo_coldens.nc' + ext_frc_specifier = 'H2O -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/elev/H2OemissionCH4oxidationx2_3D_L70_1849-2101_CMIP6ensAvg_SSP3-7.0_c190403.nc', + 'num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a1_anthro-ene_vertical_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_a1_so4_contvolcano_vertical_850-5000_0.9x1.25_c20170724.nc', + 'num_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_num_a2_so4_contvolcano_vertical_850-5000_0.9x1.25_c20170724.nc', + 'SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_SO2_contvolcano_vertical_850-5000_0.9x1.25_c20170724.nc', + 'so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a1_anthro-ene_vertical_mol_175001-210101_0.9x1.25_c20190222.nc', + 'so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_so4_a1_contvolcano_vertical_850-5000_0.9x1.25_c20170724.nc', + 'so4_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/CMIP6_emissions_1750_2015/emissions-cmip6_so4_a2_contvolcano_vertical_850-5000_0.9x1.25_c20170724.nc' + ext_frc_type = 'INTERP_MISSING_MONTHS' + fstrat_list = ' ' + rsf_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/phot/RSF_GT200nm_v3.0_c080811.nc' + season_wes_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/dvel/season_wes.nc' + srf_emis_specifier = 'bc_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_bc_a4_anthro_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'bc_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'DMS -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_DMS_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'DMS -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-SSP_DMS_other_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_bc_a4_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_bc_a4_anthro_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_pom_a4_anthro_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'num_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_num_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'pom_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_pom_a4_anthro_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'pom_a4 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_pom_a4_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SO2_anthro-ag-ship-res_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SO2_anthro-ene_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'SO2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SO2_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a1_anthro-ag-ship_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'so4_a1 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a1_bb_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'so4_a2 -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_so4_a2_anthro-res_surface_mol_175001-210101_0.9x1.25_c20190222.nc', + 'SOAG -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SOAGx1.5_anthro_surface_mol_175001-210101_0.9x1.25_c20200403.nc', + 'SOAG -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp370/emissions-cmip6-ScenarioMIP_IAMC-AIM-ssp370-1-1_SOAGx1.5_bb_surface_mol_175001-210101_0.9x1.25_c20200403.nc', + 'SOAG -> /glade/p/cesmdata/cseg/inputdata/atm/cam/chem/emis/emissions_ssp/emissions-cmip6-SOAGx1.5_biogenic_surface_mol_175001-210101_0.9x1.25_c20190329.nc' + srf_emis_type = 'INTERP_MISSING_MONTHS' + tracer_cnst_datapath = '/glade/p/cesmdata/cseg/inputdata/atm/cam/tracer_cnst' + tracer_cnst_file = 'tracer_cnst_halons_3D_L70_1849-2101_CMIP6ensAvg_SSP3-7.0_c190403.nc' + tracer_cnst_filelist = '' + tracer_cnst_specifier = 'O3','OH','NO3','HO2' + tracer_cnst_type = 'INTERP_MISSING_MONTHS' + xactive_prates = .false. + xs_long_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/phot/temp_prs_GT200nm_JPL10_c140624.nc' +/ +&chem_surfvals_nl + flbc_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_2014-2500_CMIP6_SSP370_0p5degLat_GlobAnnAvg_c190301.nc' + flbc_list = 'CO2','CH4','N2O','CFC11eq','CFC12' + flbc_type = 'SERIAL' + scenario_ghg = 'CHEM_LBC_FILE' +/ +&circ_diag_nl + do_circulation_diags = .true. +/ +&cldfrc2m_nl + cldfrc2m_do_subgrid_growth = .true. + cldfrc2m_rhmaxi = 1.0D0 + cldfrc2m_rhmaxis = 1.0D0 + cldfrc2m_rhmini = 0.80D0 + cldfrc2m_rhminis = 1.0D0 +/ +&cldfrc_nl + cldfrc_dp1 = 0.10D0 + cldfrc_dp2 = 500.0D0 + cldfrc_freeze_dry = .true. + cldfrc_ice = .true. + cldfrc_icecrit = 0.93D0 + cldfrc_iceopt = 5 + cldfrc_premib = 700.0D2 + cldfrc_premit = 75000.0D0 + cldfrc_rhminh = 0.800D0 + cldfrc_rhminl = 0.950D0 + cldfrc_rhminl_adj_land = 0.000D0 + cldfrc_sh1 = 0.04D0 + cldfrc_sh2 = 500.0D0 +/ +&clubb_his_nl + clubb_history = .false. + clubb_rad_history = .false. +/ +&clubb_params_nl + clubb_beta = 2.4 + clubb_c11 = 0.7D0 + clubb_c11b = 0.35D0 + clubb_c14 = 2.2D0 + clubb_c2rt = 1.0 + clubb_c2rtthl = 1.3 + clubb_c2thl = 1.0 + clubb_c7 = 0.5 + clubb_c7b = 0.5 + clubb_c8 = 4.2 + clubb_c_k10 = 0.5 + clubb_c_k10h = 0.3 + clubb_do_liqsupersat = .false. + clubb_gamma_coef = 0.308 + clubb_l_lscale_plume_centered = .false. + clubb_l_use_ice_latent = .false. + clubb_lambda0_stability_coef = 0.04 + clubb_mult_coef = 1.0D0 + clubb_skw_denom_coef = 0.0 +/ +&clubbpbl_diff_nl + clubb_cloudtop_cooling = .false. + clubb_expldiff = .true. + clubb_rainevap_turb = .false. + clubb_rnevap_effic = 1.0D0 + clubb_stabcorrect = .false. + clubb_timestep = 300.0D0 +/ +&co2_cycle_nl + co2_flag = .true. + co2_readflux_aircraft = .true. + co2_readflux_fuel = .true. + co2flux_fuel_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/ggas/emissions-cmip6_CO2_anthro_surface_ScenarioMIP_IAMC-AIM-ssp370_201401-210112_fv_0.9x1.25_c20190207.nc' +/ +&conv_water_nl + conv_water_frac_limit = 0.01d0 + conv_water_in_rad = 1 +/ +&dust_nl + dust_emis_fact = 0.70D0 + soil_erod_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/dst/dst_source2x2tunedcam6-2x2-04062017.nc' +/ +&dyn_fv_inparm + fv_del2coef = 3.e+5 + fv_div24del2flag = 4 + fv_fft_flt = 1 + fv_filtcw = 0 + fv_nspltvrm = 2 +/ +&gw_drag_nl + effgw_rdg_beta = 1.0D0 + effgw_rdg_beta_max = 1.0D0 + fcrit2 = 1.0 + gw_apply_tndmax = .true. + gw_dc = 0.D0 + gw_dc_long = 0.D0 + gw_limit_tau_without_eff = .false. + gw_lndscl_sgh = .true. + gw_oro_south_fac = 1.d0 + gw_prndl = 0.5D0 + n_rdg_beta = 10 + pgwv = 0 + pgwv_long = 0 + rdg_beta_cd_llb = 1.0D0 + tau_0_ubc = .false. + trpd_leewv_rdg_beta = .false. + use_gw_rdg_beta = .true. + use_gw_rdg_gamma = .false. +/ +&gw_rdg_nl + gw_rdg_c_betamax_ds = 0.0d0 + gw_rdg_c_betamax_sm = 2.0d0 + gw_rdg_c_gammamax = 2.0d0 + gw_rdg_do_adjust_tauoro = .true. + gw_rdg_do_backward_compat = .false. + gw_rdg_do_divstream = .true. + gw_rdg_do_smooth_regimes = .false. + gw_rdg_fr_c = 1.0D0 + gw_rdg_frx0 = 2.0d0 + gw_rdg_frx1 = 3.0d0 + gw_rdg_orohmin = 0.01d0 + gw_rdg_orom2min = 0.1d0 + gw_rdg_orostratmin = 0.002d0 + gw_rdg_orovmin = 1.0d-3 +/ +µ_mg_nl + micro_do_sb_physics = .false. + micro_mg_adjust_cpt = .false. + micro_mg_berg_eff_factor = 1.0D0 + micro_mg_dcs = 500.D-6 + micro_mg_num_steps = 1 + micro_mg_precip_frac_method = 'in_cloud' + micro_mg_sub_version = 0 + micro_mg_version = 2 +/ +&modal_aer_opt_nl + water_refindex_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/water_refindex_rrtmg_c080910.nc' +/ +&nucleate_ice_nl + nucleate_ice_incloud = .false. + nucleate_ice_strat = 1.0D0 + nucleate_ice_subgrid = 1.2D0 + nucleate_ice_subgrid_strat = 1.2D0 + nucleate_ice_use_troplev = .true. + use_preexisting_ice = .true. +/ +&phys_ctl_nl + cam_chempkg = 'trop_mam4' + cam_physpkg = 'cam6' + cld_macmic_num_steps = 3 + deep_scheme = 'ZM' + do_clubb_sgs = .true. + eddy_scheme = 'CLUBB_SGS' + history_aero_optics = .false. + history_aerosol = .false. + history_amwg = .true. + history_budget = .false. + history_chemistry = .true. + history_chemspecies_srf = .true. + history_clubb = .true. + history_dust = .false. + history_eddy = .false. + history_vdiag = .false. + history_waccm = .false. + history_waccmx = .false. + macrop_scheme = 'CLUBB_SGS' + microp_scheme = 'MG' + radiation_scheme = 'rrtmg' + shallow_scheme = 'CLUBB_SGS' + srf_flux_avg = 0 + use_gw_convect_dp = .false. + use_gw_convect_sh = .false. + use_gw_front = .false. + use_gw_front_igw = .false. + use_gw_oro = .false. + use_hetfrz_classnuc = .true. + use_simple_phys = .false. + use_subcol_microp = .false. + waccmx_opt = 'off' +/ +&prescribed_ozone_nl + prescribed_ozone_datapath = '/glade/p/cesmdata/cseg/inputdata/atm/cam/ozone_strataero' + prescribed_ozone_file = 'ozone_strataero_WACCM_L70_zm5day_18500101-21010201_CMIP6histEnsAvg_SSP370_c190403.nc' + prescribed_ozone_name = 'O3' + prescribed_ozone_type = 'SERIAL' +/ +&prescribed_strataero_nl + prescribed_strataero_datapath = '/glade/p/cesmdata/cseg/inputdata/atm/cam/ozone_strataero' + prescribed_strataero_file = 'ozone_strataero_WACCM_L70_zm5day_18500101-21010201_CMIP6histEnsAvg_SSP370_c190403.nc' + prescribed_strataero_type = 'SERIAL' + prescribed_strataero_use_chemtrop = .true. +/ +&qneg_nl + print_qneg_warn = 'summary' +/ +&rad_cnst_nl + icecldoptics = 'mitchell' + iceopticsfile = '/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/iceoptics_c080917.nc' + liqcldoptics = 'gammadist' + liqopticsfile = '/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/F_nwvl200_mu20_lam50_res64_t298_c080428.nc' + mode_defs = 'mam4_mode1:accum:=', 'A:num_a1:N:num_c1:num_mr:+', + 'A:so4_a1:N:so4_c1:sulfate:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', 'A:pom_a1:N:pom_c1:p-organic:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:soa_a1:N:soa_c1:s-organic:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', 'A:bc_a1:N:bc_c1:black-c:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc:+', + 'A:dst_a1:N:dst_c1:dust:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', 'A:ncl_a1:N:ncl_c1:seasalt:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc', + 'mam4_mode2:aitken:=', 'A:num_a2:N:num_c2:num_mr:+', + 'A:so4_a2:N:so4_c2:sulfate:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc:+', 'A:soa_a2:N:soa_c2:s-organic:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc:+', + 'A:ncl_a2:N:ncl_c2:seasalt:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', 'A:dst_a2:N:dst_c2:dust:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc', + 'mam4_mode3:coarse:=', 'A:num_a3:N:num_c3:num_mr:+', + 'A:dst_a3:N:dst_c3:dust:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc:+', 'A:ncl_a3:N:ncl_c3:seasalt:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc:+', + 'A:so4_a3:N:so4_c3:sulfate:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc', 'mam4_mode4:primary_carbon:=', + 'A:num_a4:N:num_c4:num_mr:+', 'A:pom_a4:N:pom_c4:p-organic:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/ocpho_rrtmg_c130709.nc:+', + 'A:bc_a4:N:bc_c4:black-c:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc' + rad_climate = 'A:Q:H2O', 'N:O2:O2', + 'N:CO2:CO2', 'N:ozone:O3', + 'N:N2O:N2O', 'N:CH4:CH4', + 'N:CFC11:CFC11', 'N:CFC12:CFC12', + 'M:mam4_mode1:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/mam4_mode1_rrtmg_aeronetdust_sig1.6_dgnh.48_c140304.nc', 'M:mam4_mode2:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/mam4_mode2_rrtmg_aitkendust_c141106.nc', + 'M:mam4_mode3:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/mam4_mode3_rrtmg_aeronetdust_sig1.2_dgnl.40_c150219.nc', 'M:mam4_mode4:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/mam4_mode4_rrtmg_c130628.nc', + 'N:VOLC_MMR1:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.6_mode1_c170214.nc', 'N:VOLC_MMR2:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.6_mode2_c170214.nc', + 'N:VOLC_MMR3:/glade/p/cesmdata/cseg/inputdata/atm/cam/physprops/volc_camRRTMG_byradius_sigma1.2_mode3_c170214.nc' +/ +&ref_pres_nl + clim_modal_aero_top_press = 1.D-4 + do_molec_press = 0.1D0 + molec_diff_bot_press = 50.D0 + trop_cloud_top_press = 1.D2 +/ +&solar_data_opts + solar_htng_spctrl_scl = .true. + solar_irrad_data_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/solar/SolarForcingCMIP6_18491230-22991231_c171031.nc' +/ +&spmd_fv_inparm + npr_yz = 64,18,18,64 +/ +&tms_nl + do_tms = .false. +/ +&tropopause_nl + tropopause_climo_file = '/glade/p/cesmdata/cseg/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc' +/ +&vert_diff_nl + diff_cnsrv_mass_check = .false. + do_iss = .true. +/ +&wetdep_inparm + gas_wetdep_list = 'H2O2','H2SO4','SO2' + gas_wetdep_method = 'NEU' +/ +&zmconv_nl + zmconv_c0_lnd = 0.0075D0 + zmconv_c0_ocn = 0.0300D0 + zmconv_ke = 5.0E-6 + zmconv_ke_lnd = 1.0E-5 + zmconv_microp = .false. + zmconv_momcd = 0.7000D0 + zmconv_momcu = 0.7000D0 + zmconv_num_cin = 1 + zmconv_org = .false. +/ diff --git a/user_nl_files/user_nl_cam b/user_nl_files/user_nl_cam deleted file mode 100644 index bcc0b84..0000000 --- a/user_nl_files/user_nl_cam +++ /dev/null @@ -1,69 +0,0 @@ -! Users should add all user specific namelist changes below in the form of -! namelist_var = new_namelist_value - -nhtfrq = 0, -24, -6, -3, -1, 1, -24,-120,-240 -mfilt = 1, 5, 20, 40, 120, 240, 365, 73, 365 -ndens = 2, 2, 2, 2, 2, 2, 1, 1, 1 - -fincl1 = 'ABSORB:A', 'ACTREL:A', 'AODABSdn:A', 'AODBCdn:A', 'AODdnDUST1:A', 'AODdnDUST2:A', - 'AODdnDUST3:A', 'AODdnMODE1:A', 'AODdnMODE2:A', 'AODdnMODE3:A', 'AODDUST2:A', 'AODDUST:A', 'AODNIRstdn:A', 'AODPOMdn:A', 'AODSO4dn:A', 'AODSOAdn:A', 'AODSSdn:A', - 'AODUVdn:A', 'AODUVstdn:A', 'AODVIS:A', 'AODVISdn:A', 'AODVISstdn:A', 'AQ_SO2:A', 'AREA:A', 'AREI:A', 'AREL:A', 'bc_a1:A', 'bc_a1DDF:A', - 'bc_a1SFWET:A', 'bc_a4:A', 'bc_a4DDF:A', 'bc_a4SFWET:A', 'bc_c1:A', 'bc_c1DDF:A', 'bc_c1SFWET:A', 'bc_c4:A', 'bc_c4DDF:A', - 'bc_c4SFWET:A', 'BROX:A', 'BROY:A', 'BURDENBCdn', 'BURDENDUSTdn:A', 'BURDENPOMdn:A', 'BURDENSEASALTdn:A', 'BURDENSO4dn:A', - 'BURDENSOAdn:A', 'CDNUMC:A', 'CLDICE:A', 'CLDLIQ:A', 'CLDTOT:A', 'CLOUD:A', 'CLOX:A', 'CLOY:A', 'CME:A', 'CMFDQ:A', 'CMFMC:A', - 'CMFMCDZM:A', 'CO2:A', 'CONCLD:A', 'DF_H2O2:A', 'DF_H2SO4:A', - 'DF_SO2:A', 'dgnumwet1:A', 'dgnumwet2:A', 'dgnumwet3:A', 'DH2O2CHM:A', 'DMS:A', 'dry_deposition_NHx_as_N:A', - 'dry_deposition_NOy_as_N:A', 'Dso4_a1CHM:A', 'Dso4_a2CHM:A', 'Dso4_a3CHM:A', 'dst_a1:A', 'dst_a1DDF:A', 'dst_a1SFWET:A', 'dst_a2:A', 'dst_a2DDF:A', - 'dst_a2SFWET:A', 'dst_a3:A', 'dst_a3DDF:A', 'dst_a3SFWET:A', 'dst_c1:A', 'dst_c1DDF:A', 'dst_c1SFWET:A', 'dst_c2:A', 'dst_c2DDF:A', 'dst_c2SFWET:A', - 'dst_c3:A', 'dst_c3DDF:A', 'dst_c3SFWET:A', 'DTCORE:A', 'EVAPPREC:A', - 'EVAPQZM:A', 'EVAPTZM:A', 'EXTINCTdn:A', 'EXTINCTNIRdn:A', 'EXTINCTUVdn:A', 'EXTxASYMdn:A', 'FCTL:A', - 'FLDS:A', 'FLDSC:A', 'FLNR:A', 'FLNS:A', 'FLNSC:A', 'FLNT:A', 'FLNTC:A', 'FLUT:A', 'FLUTC:A', 'FREQI:A', 'FREQL:A', 'FREQZM:A', 'FSDS:A', - 'FSDSC:A', 'FSNR:A', 'FSNS:A', 'FSNSC:A', 'FSNT:A', 'FSNTC:A', 'FSNTOAC:A', 'FSUTOA:A', - 'GS_SO2:A', 'H2O2:A', 'H2O:A', 'H2SO4:A', 'H2SO4M_C:A', 'H2SO4_sfnnuc1:A', - 'HCL_GAS:A', 'HNO3_GAS:A', 'HNO3_NAT:A', 'HNO3_STS:A', 'HO2:A', 'ICEFRAC:A','jh2o2:A','KVH_CLUBB:A', 'LANDFRAC:A', 'LHFLX:A', 'MASS:A', - 'ncl_a1:A', 'ncl_a1DDF:A', 'ncl_a1SFWET:A', 'ncl_a2:A', 'ncl_a2DDF:A', 'ncl_a2SFWET:A', 'ncl_a3:A', 'ncl_a3DDF:A', 'ncl_a3SFWET:A', 'ncl_c1:A', - 'ncl_c1DDF:A', 'ncl_c1SFWET:A', 'ncl_c2:A', 'ncl_c2DDF:A', 'ncl_c2SFWET:A', 'ncl_c3:A', 'ncl_c3DDF:A', 'ncl_c3SFWET:A', - 'NITROP_PD:A', 'NO3:A', 'NOX:A', 'NOY:A', - 'num_a1:A', 'num_a1_CLXF:A', 'num_a1DDF:A', 'num_a2:A', 'num_a2_CLXF:A', 'num_a2DDF:A', 'num_a2_sfnnuc1:A', 'num_a3:A', 'num_a3DDF:A', 'num_a4:A', - 'num_a4DDF:A', 'num_c1:A', 'num_c1DDF:A', 'num_c2:A', 'num_c2DDF:A', 'num_c3:A', 'num_c3DDF:A', 'num_c4:A', 'num_c4DDF:A', 'NUMLIQ:A', 'O3:A', - 'OH:A', 'OMEGA:A', 'OMEGAT:A', 'PBLH:A', 'PDELDRY:A', 'PHIS:A', 'pom_a1:A', 'pom_a1DDF:A', 'pom_a1SFWET:A', - 'pom_a4:A', 'pom_a4DDF:A', 'pom_a4SFWET:A', 'pom_c1:A', 'pom_c1DDF:A', 'pom_c1SFWET:A', 'pom_c4:A', 'pom_c4DDF:A', 'pom_c4SFWET:A', - 'PRECC:A', 'PRECT:A', 'PS:A', 'PSL:A', 'PTEQ:A', 'PTTEND:A', 'Q:A', 'QFLX:A', 'QRAIN:A', 'QREFHT:A', 'QRL:A', 'QRLC:A', 'QRS:A', - 'QRSC:A', 'QSNOW:A', 'RAD_ICE:A', 'RAD_LNAT:A', 'RAD_SULFC:A', 'REFF_AERO:A', 'RELHUM:A', 'RHREFHT:A', - 'SAD_AERO:A', 'SAD_ICE:A', 'SAD_LNAT:A', 'SAD_SULFC:A', 'SAD_TROP:A', 'SFbc_a4:A', - 'SFDMS:A', 'SFdst_a1:A', 'SFdst_a2:A', 'SFdst_a3:A', - 'SFncl_a1:A', 'SFncl_a2:A', 'SFncl_a3:A', 'SFnum_a1:A', 'SFnum_a2:A', - 'SFnum_a3:A', 'SFpom_a4:A', 'SFSO2:A', 'SFso4_a1:A', 'SFso4_a2:A', 'SHFLX:A', 'SO2:A', - 'SO2_CHML:A', 'SO2_CHMP:A', 'SO2_CLXF:A', 'SO2_XFRC:A', 'so4_a1:A', 'so4_a1_CHMP:A', 'so4_a1_CLXF:A', 'so4_a1DDF:A', 'so4_a1_sfgaex1:A', 'so4_a1SFWET:A', - 'so4_a2:A', 'so4_a2_CHMP:A', 'so4_a2_CLXF:A', 'so4_a2DDF:A', 'so4_a2_sfgaex1:A', 'so4_a2_sfnnuc1:A', 'so4_a2SFWET:A', 'so4_a3:A', - 'so4_a3DDF:A', 'so4_a3_sfgaex1:A', 'so4_a3SFWET:A', 'so4_c1:A', 'so4_c1AQH2SO4:A', 'so4_c1AQSO4:A', 'so4_c1DDF:A', 'so4_c1SFWET:A', 'so4_c2:A', 'so4_c2AQH2SO4:A', - 'so4_c2AQSO4:A', 'so4_c2DDF:A', 'so4_c2SFWET:A', 'so4_c3:A', 'so4_c3AQH2SO4:A', 'so4_c3AQSO4:A', 'so4_c3DDF:A', 'so4_c3SFWET:A', - 'SOLIN:A', 'SOLLD:A', 'SOLSD:A', 'SSAVIS:A', 'SST:A', - 'T:A', 'TAQ:A', 'TAUBLJX:A', 'TAUBLJY:A', 'TAUGWX:A', 'TAUGWY:A', 'TAUX:A', 'TAUY:A', 'TBRY:A', 'TCLY:A', - 'TGCLDIWP:A', 'TGCLDLWP:A', 'TMDMS:A', 'TMQ:A', 'TMSO2:A', - 'TMso4_a1:A', 'TMso4_a2:A', 'TMso4_a3:A', 'TOT_CLD_VISTAU:A', 'TOTH:A', 'TREFHT:A', 'TREFHTMN:A', 'TREFHTMX:A', 'TROP_P:A', - 'TROP_T:A', 'TROP_Z:A', 'TS:A', 'TSMN:M', 'TSMX:X', 'TTEND_TOT:A', 'TTGWORO:A', 'U10:A', 'U:A', 'UTGWORO:A', - 'UU:A', 'V:A', 'VD01:A', 'VV:A', 'WD_H2O2:A', 'WD_H2SO4:A', 'WD_SO2:A', 'wet_deposition_NHx_as_N:A', 'wet_deposition_NOy_as_N:A', - 'Z3:A', 'ZMDQ:A', 'ZMDT:A', 'ZMMTT:A', 'ZMMU:A', - 'MSKtem','PS','PSL','VTHzm','UVzm','UWzm','Uzm','Vzm','THzm','Wzm','PHIS' - ! 'dst_a1_SRF:A', 'dst_a3_SRF:A', 'AODABS:A', 'soa_a1:A', 'soa_a2:A', 'soa_c1:A', 'soa_c2:A', - ! 'soa_a1SFWET:A', 'soa_a2SFWET:A', 'soa_c1SFWET:A', 'soa_c2SFWET:A', 'soa_a1DDF:A', 'soa_a2DDF:A', 'soa_c1DDF:A', 'soa_c2DDF:A', 'bc_a4_CLXF:A', - ! 'pom_a4_CLXF:A', 'soa_a1_sfgaex1:A', 'soa_a2_sfgaex1:A', 'ADRAIN:A', 'ADSNOW:A', 'ANRAIN:A', 'ANSNOW:A', 'AQRAIN:A', 'AQSNOW:A', 'AQSO4_H2O2:A', - ! 'AQSO4_O3:A', 'AWNC:A', 'AWNI:A', 'CCN3:A', 'CLDHGH:A', 'CLDLOW:A', 'CLDMED:A', 'FICE:A', 'FREQR:A', 'FREQS:A', 'FSNTOA:A', 'FSNTOA:A', 'H2SO4_sfgaex1:A', - ! 'ICIMR:A', 'ICWMR:A', 'IWC:A', 'LWCF:A', 'PRECL:A', 'PRECSC:A', 'PRECSL:A', 'QT:A', 'RAINQM:A', 'RCM_CLUBB:A', 'RELVAR:A', 'RTP2_CLUBB:A', 'RTPTHLP_CLUBB:A', - ! 'SFSOAG:A', 'SNOWQM:A', 'SWCF:A', 'TGCLDCWP:A', 'UP2_CLUBB:A', 'UPWP_CLUBB:A', 'VP2_CLUBB:A', 'VPWP_CLUBB:A', 'VQ:A', 'VU:A', 'WP2_CLUBB:A', 'WP3_CLUBB:A', - ! 'WPRCP_CLUBB:A', 'WPRTP_CLUBB:A', 'WPTHLP_CLUBB:A', 'WPTHVP_CLUBB:A', 'WSUB:A', 'bc_a1_SRF:A', 'bc_a4_SRF:A', 'O3colAbove:A' - -fincl2 = 'ACTNL:A', 'ACTREL:A', 'BURDENBCdn:A', 'BURDENDUSTdn:A', 'BURDENPOMdn:A', 'BURDENSEASALTdn:A', 'BURDENSO4dn:A', 'BURDENSOAdn:A', - 'CDNUMC:A', 'CLDICE:A', 'CLDLIQ:A', 'CLDTOT:A', 'CLOUD:A', 'CMFMC:A', 'CMFMCDZM:A', 'FCTL:A', 'FLDS:A', 'FLDSC:A', 'FLNR:A', 'FLNS:A', 'FLNSC:A', - 'FLNT:A', 'FLNTC:A', 'FLUT:A', 'FLUTC:A', 'FSDS:A', 'FSDSC:A', 'FSNR:A', 'FSNS:A', 'FSNSC:A', 'FSNTOA:A', 'FSNTOAC:A', 'LHFLX:A', 'MASS:A', 'OMEGA:A', - 'OMEGA500:A', 'PBLH:A', 'PDELDRY:A', 'PRECC:A', 'PRECT:A', 'PS:A', 'PSL:A', 'Q:A', 'QREFHT:A', 'QSNOW:A', 'RELHUM:A', 'RHREFHT:A', 'SHFLX:A', - 'SOLIN:A', 'SOLLD:A', 'SOLSD:A', 'T:A', 'T500:A', 'T700:A', 'T850:A', 'TAUBLJX:A', 'TAUBLJY:A', 'TAUGWX:A', 'TAUGWY:A', 'TAUX:A', 'TAUY:A', - 'TGCLDIWP:A', 'TGCLDLWP:A', 'TMQ:A', 'TREFHT:A', 'TREFHTMN:A', 'TREFHTMX:A', 'TS:A', 'TSMN:M', 'TSMX:X', 'U:A', 'U10:A', 'UTGWORO:A', - 'V:A', 'Z3:A', 'Z500:A' - fincl7 = 'MSKtem','PS','PSL','VTHzm','UVzm','UWzm','Uzm','Vzm','THzm','Wzm','PHIS' - -do_circulation_diags = .true. -co2flux_fuel_file = '$DIN_LOC_ROOT/atm/cam/ggas/emissions-cmip6_CO2_anthro_surface_175001-201512_fv_0.9x1.25_c20181011.nc' -aircraft_specifier = 'ac_CO2 -> ac_CO2_filelist_175001-201512_fv_0.9x1.25_c20181011.txt' - diff --git a/user_nl_files/user_nl_cice b/user_nl_files/user_nl_cice deleted file mode 100644 index c644606..0000000 --- a/user_nl_files/user_nl_cice +++ /dev/null @@ -1,9 +0,0 @@ -!---------------------------------------------------------------------------------- -! Users should add all user specific namelist changes below in the form of -! namelist_var = new_namelist_value -! Note - that it does not matter what namelist group the namelist_var belongs to -!---------------------------------------------------------------------------------- - -histfreq = 'm','d','x','x','x' -histfreq_n = 1,1,1,1,1 -f_CMIP = 'mdxxx' diff --git a/user_nl_files/user_nl_clm b/user_nl_files/user_nl_clm deleted file mode 100644 index f337f2e..0000000 --- a/user_nl_files/user_nl_clm +++ /dev/null @@ -1,133 +0,0 @@ - -!---------------------------------------------------------------------------------- -! Users should add all user specific namelist changes below in the form of -! namelist_var = new_namelist_value -! -! EXCEPTIONS: -! Set use_cndv by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting -! Set use_vichydro by the compset you use and the CLM_BLDNML_OPTS -vichydro setting -! Set use_cn by the compset you use and CLM_BLDNML_OPTS -bgc setting -! Set use_crop by the compset you use and CLM_BLDNML_OPTS -crop setting -! Set spinup_state by the CLM_BLDNML_OPTS -bgc_spinup setting -! Set irrigate by the CLM_BLDNML_OPTS -irrig setting -! Set co2_ppmv with CCSM_CO2_PPMV option -! Set dtime with L_NCPL option -! Set fatmlndfrc with LND_DOMAIN_PATH/LND_DOMAIN_FILE options -! Set finidat with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases -! (includes inst_string for multi-ensemble cases) -! Set maxpatch_glcmec with GLC_NEC option -! Set glc_do_dynglacier with GLC_TWO_WAY_COUPLING env variable -!---------------------------------------------------------------------------------- - -!---------------------------------------------------------------------------------- -! Fix Antarctica bug -! Erik: add this b/c RUN_REFCASE hist.011 was not fixed for the wetland bug -!---------------------------------------------------------------------------------- -use_init_interp = .true. - -!---------------------------------------------------------------------------------- -! Settings from output_base -!---------------------------------------------------------------------------------- - -! h0 stream (monthly average, gridcell-level) -! h0 stream ICE variables and other variables inactive by default (monthly average, gridcell-level) -! Emon, Lmon, LIMon -hist_mfilt(1) = 1 -hist_dov2xy(1) = .true. -hist_nhtfrq(1) = 0 -hist_type1d_pertape(1) = ' ' -hist_fexcl1 += 'PCT_GLC_MEC','PCT_NAT_PFT' -hist_fincl1 += 'EFLX_LH_TOT_ICE', 'FIRE_ICE', 'FLDS_ICE', 'FSH_ICE', 'FSNO_ICE', 'FSR_ICE', 'QFLX_SUB_SNOW_ICE', 'QRUNOFF_ICE', 'QSNOFRZ_ICE', 'QSNOMELT_ICE', 'RAIN_ICE', 'SNOW_ICE', 'SNOWICE_ICE', 'SNOWLIQ_ICE', 'SNOTXMASS_ICE', 'TG_ICE', 'TOPO_COL_ICE', 'TSA_ICE', 'TSOI_ICE' - -! h1 stream (monthly average, finest sub-grid) -! Emon, Lmon -hist_mfilt(2) = 1 -hist_dov2xy(2) = .false. -hist_nhtfrq(2) = 0 -hist_type1d_pertape(2) = ' ' -hist_fincl2 += 'TLAI', 'TSA','TREFMNAV','TREFMXAV', 'BTRANMN', 'Vcmx25Z', 'FSH', 'VEGWP', 'FCTR', 'FCEV', 'FGEV', 'FIRE', 'FSR', 'FIRA', 'FSA', 'GSSUNLN', 'GSSHALN', 'TSKIN', 'H2OSOI' - -! h2 stream (monthly average, landunit-level) -! Emon -hist_mfilt(3) = 1 -hist_dov2xy(3) = .false. -hist_nhtfrq(3) = 0 -hist_type1d_pertape(3) = 'LAND' -hist_fincl3 += 'FSR', 'H2OSNO', 'Q2M', 'SNOWDP', 'TSA', 'TREFMNAV', 'TREFMXAV', 'TG', 'QRUNOFF', 'FSH', 'FIRE', 'FIRA', 'FGR', 'EFLX_LH_TOT', 'RH2M', 'TLAI', 'SOILWATER_10CM', 'TOTSOILLIQ', 'TOTSOILICE', 'U10', 'TSOI_10CM', 'QIRRIG', 'URBAN_HEAT', 'WASTEHEAT', 'TSKIN' - -! h3 stream (yearly average, gridcell-level) -! Eyr -hist_mfilt(4) = 1 -hist_dov2xy(4) = .true. -hist_nhtfrq(4) = -8760 -hist_type1d_pertape(4) = ' ' -hist_fincl4 += 'PCT_GLC_MEC', 'QICE_FORC', 'TSRF_FORC', 'TOPO_FORC', 'PCT_NAT_PFT','PCT_LANDUNIT','FSNO_ICE' - -! h4 stream (yearly average, landunit-level) -! Eyr landuse tiles -! (No fields in base, but other usermods add to this stream) -hist_mfilt(5) = 1 -hist_dov2xy(5) = .false. -hist_nhtfrq(5) = -8760 -hist_type1d_pertape(5) = 'LAND' - -!---------------------------------------------------------------------------------- -! Settings from output_bgc -!---------------------------------------------------------------------------------- - -! h0 stream (monthly average, gridcell-level) -hist_fexcl1 += 'SOIL1C_vr', 'SOIL1N_vr', 'SOIL2C_vr', 'SOIL2N_vr', 'SOIL3C_vr', 'SOIL3N_vr', 'SOILC_vr','SOILN_vr', 'CWDC_vr', 'LITR1C_vr', 'LITR2C_vr', 'LITR3C_vr', 'LITR1N_vr', 'LITR2N_vr', 'LITR3N_vr', 'CWDN_vr', 'SMIN_NO3_vr', 'CONC_O2_UNSAT', 'CONC_O2_SAT','SMIN_NH4_vr','SMINN_vr' -hist_fincl1 += 'LEAFC_TO_LITTER', 'FROOTC_TO_LITTER','LITR1C_TO_SOIL1C','LITR1N_TO_SOIL1N','LITR2C_TO_SOIL1C', 'LITR2N_TO_SOIL1N','LITR3C_TO_SOIL2C','LITR3N_TO_SOIL2N','DWT_WOOD_PRODUCTC_GAIN_PATCH' - -! h1 stream (monthly average, finest sub-grid) -hist_fincl2 += 'GPP', 'NPP', 'AGNPP', 'TOTVEGC', 'NPP_NUPTAKE', 'AR', 'HR', 'HTOP' - -! h2 stream (monthly average, landunit-level) -! TOT_WOODPRODC:I, CROPPROD1C:I, and NEE are not available at the landunit level -hist_fincl3 += 'GPP', 'NPP', 'AR', 'HR', 'DWT_CONV_CFLUX_PATCH', 'WOOD_HARVESTC', 'DWT_WOOD_PRODUCTC_GAIN_PATCH', 'SLASH_HARVESTC', 'COL_FIRE_CLOSS', 'DWT_SLASH_CFLUX', 'FROOTC:I', 'HTOP' - -! h3 stream (yearly average, gridcell-level) -hist_fincl4 += 'SOILC_vr', 'SOILN_vr', 'CWDC_vr', 'LITR1C_vr', 'LITR2C_vr', 'LITR3C_vr', 'LITR1N_vr', 'LITR2N_vr', 'LITR3N_vr','CWDN_vr', 'TOTLITC:I', 'TOT_WOODPRODC:I', 'TOTSOMC:I','TOTVEGC:I' - -! h4 stream (yearly average, landunit-level) -hist_fincl5 += 'TOTSOMC:I', 'TOTSOMC_1m:I', 'TOTECOSYSC:I', 'TOTVEGC:I', 'WOODC:I', 'TOTLITC:I', 'LIVECROOTC:I', 'DEADCROOTC:I', 'FROOTC:I' - -!---------------------------------------------------------------------------------- -! Settings from output_crop -!---------------------------------------------------------------------------------- - -! h0 stream (monthly average, gridcell-level) -hist_fexcl1 += 'PCT_CFT' - -! h1 stream (monthly average, finest sub-grid) -hist_fincl2 += 'GRAINC_TO_FOOD','NFERTILIZATION' - -! h2 stream (monthly average, landunit-level) -hist_fincl3 += 'GRAINC_TO_FOOD' - -! h3 stream (yearly average, gridcell-level) -hist_fincl4 += 'PCT_CFT','CROPPROD1C:I' - -!---------------------------------------------------------------------------------- -! Settings from cmip6_carbon_isotopes -!---------------------------------------------------------------------------------- - -use_c13 = .true. -use_c14 = .true. -use_c13_timeseries = .true. -use_c14_bombspike = .true. - -! h0 stream (monthly average, gridcell-level) -hist_fexcl1 += 'C14_SOILC_vr' -! h3 stream (yearly average, gridcell-level) -hist_fincl4 += 'C14_SOILC_vr' - -!---------------------------------------------------------------------------------- -! Settings from cmip6_glaciers_virtual_antarctica -!---------------------------------------------------------------------------------- - -! This differs from the default in that it turns on virtual columns over Antarctica -! This is desired so that we have the output needed to drive a later offline CISM Antarctica simulation -! However, this increases the cost of CLM by about 10% -glacier_region_behavior = 'single_at_atm_topo', 'virtual', 'virtual', 'virtual' - diff --git a/user_nl_files/v2-catalyst/README.v2-output b/user_nl_files/v2-catalyst/README.v2-output new file mode 100644 index 0000000..eed00f4 --- /dev/null +++ b/user_nl_files/v2-catalyst/README.v2-output @@ -0,0 +1 @@ +https://docs.google.com/document/d/1yQqXLuW2Ct-ZhkjiVEjaCQdWJxYT_kBcC2VNyqGaic0/edit diff --git a/user_nl_files/v2-catalyst/user_nl_cam-mbr1 b/user_nl_files/v2-catalyst/user_nl_cam-mbr1 new file mode 100644 index 0000000..8f1d61c --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_cam-mbr1 @@ -0,0 +1,49 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + + !xxxx nhtfrq = 0,-24,-6 + !xxxx mfilt = 1,30,120,120,240,30 + !xxxx avgflag_pertape = 'A','A','I','A','A','A' + !xxxx fincl1 = 'IEFLX' + !xxxx fincl2 = 'FLUT','TREFHTMN','TREFHTMX','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! monthly (h0) + fexcl1 = 'LINOZ_SZA','extinct_lw_bnd7','extinct_lw_inp','extinct_sw_inp','hstobie_linoz','LINOZ_DO3', + 'LINOZ_DO3_PSC','LINOZ_O3CLIM','LINOZ_O3COL','LINOZ_SSO3','UU','VQ','VU','VV','bc_a1_SRF', + 'bc_a3_SRF','bc_a4_SRF','chla','dst_a1_SRF','mlip','mom_a1_SRF','mom_a2_SRF','mom_a3_SRF', + 'mom_a4_SRF','mpoly','mprot','ncl_a1_SRF','ncl_a2_SRF','ncl_a3_SRF','num_a1_SRF','num_a2_SRF', + 'num_a3_SRF','num_a4_SRF','pom_a1_SRF','pom_a3_SRF','pom_a4_SRF','so4_a1_SRF','so4_a2_SRF','so4_a3_SRF', + 'soa_a1_SRF','soa_a2_SRF','soa_a3_SRF' + + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', 'U10:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', +! 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', +! 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', +! 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', +! 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', +! 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) + fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +!xxxx Sam: 6 hrly instant +!xxxx fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'TTQ:I','TUQ:I','TVQ:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) + fincl6 = 'PRECC:I', 'PRECT:I' + +history_aerosol=.false. +ncdata= '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.i.0200-01-01-00000.nc' diff --git a/user_nl_files/v2-catalyst/user_nl_cam-mbr2-3 b/user_nl_files/v2-catalyst/user_nl_cam-mbr2-3 new file mode 100644 index 0000000..b73c783 --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_cam-mbr2-3 @@ -0,0 +1,50 @@ +!! h0, h1, h2, h3, h4, h5 + nhtfrq = 0, -24, -24, -6, -6, -3 + mfilt = 1, 365, 73, 73, 20, 73 + + !SamStevensonRuns + !xxxx nhtfrq = 0,-24,-6 + !xxxx mfilt = 1,30,120,120,240,30 + !xxxx avgflag_pertape = 'A','A','I','A','A','A' + !xxxx fincl1 = 'IEFLX' + !xxxx fincl2 = 'FLUT','TREFHTMN','TREFHTMX','LHFLX', 'SHFLX', 'PRECL', 'PRECC', 'FLNSC', 'FLNS' + +! monthly (h0) + fexcl1 = 'LINOZ_SZA','extinct_lw_bnd7','extinct_lw_inp','extinct_sw_inp','hstobie_linoz','LINOZ_DO3', + 'LINOZ_DO3_PSC','LINOZ_O3CLIM','LINOZ_O3COL','LINOZ_SSO3','UU','VQ','VU','VV','bc_a1_SRF', + 'bc_a3_SRF','bc_a4_SRF','chla','dst_a1_SRF','mlip','mom_a1_SRF','mom_a2_SRF','mom_a3_SRF', + 'mom_a4_SRF','mpoly','mprot','ncl_a1_SRF','ncl_a2_SRF','ncl_a3_SRF','num_a1_SRF','num_a2_SRF', + 'num_a3_SRF','num_a4_SRF','pom_a1_SRF','pom_a3_SRF','pom_a4_SRF','so4_a1_SRF','so4_a2_SRF','so4_a3_SRF', + 'soa_a1_SRF','soa_a2_SRF','soa_a3_SRF' + + fincl1 = 'IEFLX:A','ZMDT:A', 'ZMDQ:A', 'STEND_CLUBB:A', 'RVMTEND_CLUBB:A', 'MPDT:A', 'MPDQ:A', 'DCQ:A', 'DTCOND:A' + +! daily 2D (h1) - Averaged + fincl2 = 'FLUT:A','TREFHT:A','TREFHTMN:M','TREFHTMX:X','LHFLX:A', 'SHFLX:A', 'PRECT:A', 'PRECTMX:X','PRECC:A', + 'FLNT:A','FSNS:A','FSNT:A','FLNSC:A', 'FLNS:A','PS:A', 'PSL:A','FLNT:A', 'RHREFHT:A', + 'QRS:A', 'QRL:A','TROPF_P:A','TROPF_T:A','TROPF_Z:A', 'U10:A', + 'Q1000:A', 'Q850:A', 'Q700:A', 'Q500:A', 'Q200:A', 'Q100:A', 'Q050:A', 'Q010:A', 'QBOT:A', + 'U1000:A', 'U850:A', 'U700:A', 'U500:A', 'U200:A', 'U100:A', 'U050:A', 'U010:A', 'UBOT:A', + 'V1000:A', 'V850:A', 'V700:A', 'V500:A', 'V200:A', 'V100:A', 'V050:A', 'V010:A', 'VBOT:A', + 'T1000:A', 'T850:A', 'T700:A', 'T500:A', 'T200:A', 'T100:A', 'T050:A', 'T010:A', 'TBOT:A', + 'Z1000:A', 'Z850:A', 'Z700:A', 'Z500:A', 'Z200:A', 'Z100:A', 'Z050:A', 'Z010:A', 'U050:A', + + +! daily Full-column (h2) - Averaged (Kai) +! fincl3 = 'U:A', 'V:A', 'OMEGA:A', 'T:A', 'Q:A','Z3:A' + +!xxxx Sam: 6 hrly instant +!xxxx fincl3 = 'PSL','TREFHT', 'FLDS', 'FSDS', 'PRECT', 'PS', 'QREFHT', 'Q', 'PS', 'QREFHT', 'TS', 'T', 'U', 'V', 'Z3' +! 6hrly 2D (h3) + fincl4 = 'PSL:I','TREFHT:I', 'FLDS:I', 'FSDS:I', 'PRECT:A', 'PS:I', 'QREFHT:I', 'TS:I', + 'TMQ:I','U10:I','U850:I','V850:I','UBOT:I','VBOT:I','Z200:I','Z500:I', + 'TTQ:I','TUQ:I','TVQ:I' + +! 6 hrly Full-column (h4) + fincl5 = 'Q:I', 'T:I', 'U:I', 'V:I', 'Z3:I' + +! 3 hrly 2D (h5) +! fincl6 = 'PRECC:I', 'PRECT:I' + +history_aerosol=.false. +ncdata= '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.i.0200-01-01-00000.nc' diff --git a/user_nl_files/v2-catalyst/user_nl_clm b/user_nl_files/v2-catalyst/user_nl_clm new file mode 100644 index 0000000..5146f3f --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_clm @@ -0,0 +1,30 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! +! Include namelist variables for drv_flds_in ONLY if -megan and/or -drydep options +! are set in the CLM_NAMELIST_OPTS env variable. +! +! EXCEPTIONS: +! Set use_cndv by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting +! Set use_vichydro by the compset you use and the CLM_BLDNML_OPTS -vichydro setting +! Set use_cn by the compset you use and CLM_BLDNML_OPTS -bgc setting +! Set use_crop by the compset you use and CLM_BLDNML_OPTS -crop setting +! Set spinup_state by the CLM_BLDNML_OPTS -bgc_spinup setting +! Set irrigate by the CLM_BLDNML_OPTS -irrig setting +! Set co2_ppmv with CCSM_CO2_PPMV option +! Set dtime with L_NCPL option +! Set fatmlndfrc with LND_DOMAIN_PATH/LND_DOMAIN_FILE options +! Set finidat with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases +! (includes $inst_string for multi-ensemble cases) +! Set glc_grid with CISM_GRID option +! Set glc_smb with GLC_SMB option +! Set maxpatch_glcmec with GLC_NEC option +! Set glc_do_dynglacier with GLC_TWO_WAY_COUPLING env variable +!---------------------------------------------------------------------------------- + + check_finidat_year_consistency = .false. + hist_nhtfrq = 0, -24 + hist_fincl2 = 'FSNO', 'H2OSNO', 'QRUNOFF', 'RAIN', 'SNOW', 'SOILWATER_10CM', 'VOLR', 'FCEV', 'FCTR', 'FGEV', 'FGR', 'FSH', 'FSM', 'QOVER', 'QSNOMELT' + finidat = ' ' + finidat_interp_source = '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.clm2.r.0200-01-01-00000.nc' diff --git a/user_nl_files/v2-catalyst/user_nl_cpl b/user_nl_files/v2-catalyst/user_nl_cpl new file mode 100644 index 0000000..a209536 --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_cpl @@ -0,0 +1,19 @@ +!------------------------------------------------------------------------ +! Users should ONLY USE user_nl_cpl to change namelists variables +! for namelist variables in drv_in (except for the ones below) and +! any keyword/values in seq_maps.rc +! Users should add ALL user specific namelist and seq_maps.rc changes below +! using the following syntax +! namelist_var = new_namelist_value +! or +! mapname = new_map_name +! For example to change the default value of ocn2atm_fmapname to 'foo' use +! ocn2atm_fmapname = 'foo' +! +! Note that some namelist variables MAY NOT be changed in user_nl_cpl - +! they are defined in a $CASEROOT xml file and must be changed with +! xmlchange. +! +! For example, rather than set username to 'foo' in user_nl_cpl, call +! ./xmlchange USER=foo +!------------------------------------------------------------------------ diff --git a/user_nl_files/v2-catalyst/user_nl_mosart b/user_nl_files/v2-catalyst/user_nl_mosart new file mode 100644 index 0000000..77a98c6 --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_mosart @@ -0,0 +1,12 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes below in the form of +! namelist_var = new_namelist_value +! NOTE: namelist variable coupling_period CAN ONLY be changed by modifying the value +! of the xml variable ROF_NCPL in env_run.xml +! NOTE: if the xml variable ROF GRID in env_build.xml is set to 'null', then +! the RTM build-namelist will set do_rtm to .false. - and will ignore +! any change below +!---------------------------------------------------------------------------------- + + +finidat_rtm = '/global/cscratch1/sd/nanr/archive/20180129.DECKv1b_piControl.ne30_oEC.edison/0200-01-01-00000/20180129.DECKv1b_piControl.ne30_oEC.edison.mosart.r.0200-01-01-00000.nc' diff --git a/user_nl_files/v2-catalyst/user_nl_mpascice b/user_nl_files/v2-catalyst/user_nl_mpascice new file mode 100644 index 0000000..310ee9d --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_mpascice @@ -0,0 +1,17 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpassi_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpassi/streams.seaice +!---------------------------------------------------------------------------------- + diff --git a/user_nl_files/v2-catalyst/user_nl_mpaso b/user_nl_files/v2-catalyst/user_nl_mpaso new file mode 100644 index 0000000..8912b49 --- /dev/null +++ b/user_nl_files/v2-catalyst/user_nl_mpaso @@ -0,0 +1,22 @@ +!---------------------------------------------------------------------------------- +! Users should add all user specific namelist changes after these comments +! in the form of +! namelist_var = new_namelist_value +! *** EXCEPT FOR *** +! 1. DO NOT CHANGE config_start_time, config_run_duration, config_stop_time, +! config_do_restart, config_Restart_timestamp_filename, config_calendar_type, +! config_set_restingThickness_to_IC, config_alter_ICs_for_pbcs +! 2. To preview the namelists, invoke $CASEROOT preview-namelists and look at +! $CASEROOT/CaseDocs/mpaso_in +! +! +! Users cannot use this file to configure streams. In order to configure streams, +! write a stream configuration file, and place it in +! SourceMods/src.mpaso/streams.ocean_forward +!---------------------------------------------------------------------------------- + + config_btr_dt = '0000_00:01:00' + config_mom_del4 = 1.2e11 + config_use_mom_del2 = .true. + config_mom_del2 = 1000.0 + config_dt = '00:30:00'