[draft] merging the changes/improvements in `feature/release3.0.3` #409

byoung-joo · 2025-12-17T19:12:45Z

Work in progress

Description

This PR contains various updates from release/3.0.3 branch. Below you can find a brief summary of it.

Adding a set of new observation-related yaml files under config/jedi/ObsPlugs
- It is worth to mention that now bufr-converted files will be used for MHS by default, rather than ncdiag-converted files.
Adding no outer loop update: true under background error, which initializes the B only one time. This fixes Skip the re-initialization of B #396.
The hybrid weights for static and ensemble B parts can be specified either from scalar values or from files. This can be controlled with variational.hybridBECWeightFromFile key, which has a default value as False. With this, config/jedi/applications/3dhybrid-allsky.yaml can be removed, which is essentially the same as config/jedi/applications/3dhybrid.yaml.
For the BUMP VertcalBalance, now the "global" vbal/sampling files will be used, instead of "local" files. There is no change in the result & runtime.
Adding new default meshes; 15 km and 15-3 km (many yamls under scenarios/defaults/ directory)
Adding the 30 km static B, 60 km LGETKF ensemble perturbation, 30 km LGETKF ensemble perturbation (see scenarios/defaults/variational.yaml)
Adding a set of scenarios, especially with a higher resolution or rttov-related ones.
While release/3.0.3 uses the crtm v2.4.x, this PR keeps the crtm v3 (as in current develop). Only updating the path for CRTMTABLES as /glade/campaign/mmm/parc/jban/pandac_common/crtm_coeffs_v3/, which also contains the coefficients files for active sensors.

### Description This PR updates data path for 30km externalanalyses for scenarios/3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC.yaml . Files modified: scenarios/3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC.yaml ### Tests completed #### Tier 1: - [x] 3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC

CMakeLists for new build

### Description This PR adds changes for assimilating GNSSRO bending angle (ROPP1D and MetOffice operators) with the NRL observation error. Previous error parameters using the Desroziers et al 2005 method are commented out for reference. Here, the built code is also updated to account for recent changes in mpas-jedi to pass geometric height as geopotential height for bending angle forward simulations. In addition, the `ObsSpaceV2-to-V3.yaml` file and scenario `3denvar_O30kmIE60km_WarmStart.yaml` are updated. ### Tests completed #### Tier 1: - [x] 3dvar_OIE120km_WarmStart - [x] 3denvar_OIE120km_IAU_WarmStart - [x] 3dvar_OIE120km_ColdStart - [x] 3dvar_O30kmIE60km_ColdStart - [x] 3denvar_O30kmIE60km_WarmStart - [x] eda_OIE120km_WarmStart - [x] getkf_OIE120km_WarmStart - [x] ForecastFromGFSAnalysesMPT #### Tier 2: - [x] 3denvar_O30kmIE60km_WarmStart (under scenario folder)

…AMSUA

3dhybrid scenario

…#333) ### Description After looking at Jim's weekly results today I realized I forgot to remove configurations for my experiment with allsky AMSUA that I had added in my local branch. This branch should not include these changes at this point and therefore this PR accomplishes that. By having allsky AMSUA in the benchmark, results from develop will produce degradation/improvements in some regions (corresponding to improvements/degradation when adding these obs), as shown in Jim's results. ### Issue closed None

predictor along with other predictors, for channels 8-10 and channel 13

#343) This PR adds MHS clear-sky assimilation using raw observations to replace the ncdiag files. - Use BennartzScatIndex check: References: Bennartz, R., A. Thoss, A. Dybbroe, and D. B. Michelson, 2002: Precipitation analysis using the AdvancedMicrowave Sounding Unit in support of nowcasting applications. Meteorol. Appl., 9,177–189, doi:10.1017/S1350482702002037. - Assimilate MHS Channel 5 within the range of 60S to 60N degrees. - Turn off MHS N19 Channel 3 due to poor quality. - Modified files: config/jedi/ObsPlugs/da/base/mhs_metop-a.yaml config/jedi/ObsPlugs/da/base/mhs_metop-b.yaml config/jedi/ObsPlugs/da/base/mhs_n18.yaml config/jedi/ObsPlugs/da/base/mhs_n19.yaml config/jedi/ObsPlugs/da/bias/mhs_metop-a.yaml config/jedi/ObsPlugs/da/bias/mhs_metop-b.yaml config/jedi/ObsPlugs/da/bias/mhs_n18.yaml config/jedi/ObsPlugs/da/bias/mhs_n19.yaml config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-a.yaml config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-b.yaml config/jedi/ObsPlugs/da/filtersWithBias/mhs_n18.yaml config/jedi/ObsPlugs/da/filtersWithBias/mhs_n19.yaml scenarios/3dhybrid_O30kmIE60km_SpecifiedEnsemble_VarBC.yaml scenarios/defaults/observations.yaml - Tests completed 3dhybrid_O30kmIE60km (conducted month-long cycling DA) 3denvar_OIE120km_WarmStart_VarBC (ran two cycles)

Description Add 'reduce obs spcace' function to the thinning section to reduce memory usage. With this function, those observations removed by thinning will not come to memory and will not be in the obsout_da files.

### Description I realized that last week I accidentally merged local changes into this branch. The changes are related to ABI/AHI data assimilation then I don't think it affects anybody's runs so far but these changes are not ready to be in this branch yet. I rebased the branch by deleting that specific commit (262d045). I apologize for the confusion.

…s from default list in forecast post (#350) ## Description Similar to PR [345](#345) to remove all jobs specification for the `forecast` entry in the scenario YAML files under the `scenarios` folder and `verifyobs` from the default list in the `post` for forecast. It also removes two scenarios: for abiahi and amsua, that were merged by mistake before.

### Description Previous PR [348](#348) didn't work. Restoring abi/ahi settings and removing additional yamls here.

* Enforce usage of internal error table for prepbufr files * Clean up specification of obs2ioda build

### Description This PR is analogous to [PR#368](#368) to update the build path to support the new [obs2ioda_v3 ](https://github.com/NCAR/obs2ioda) code (by Andy Stokely) and modify the `ObsToIODA.csh` script to account for the same. ### Tests completed #### Tier 2: - [x] GenerateObs

with changes related to VarBC/Harnisch-error model and rttov14 modified: initialize/data/Observations.py modified: initialize/framework/Build.py

new file: config/jedi/ObsPlugs/da/bias/amsua-cld_metop-a.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-cld_metop-b.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-cld_metop-c.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-cld_n15.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-cld_n18.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-cld_n19.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_metop-a.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_metop-b.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_metop-c.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n15.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n18.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n19.yaml

new file: tools/update_analysis_states.py

dual-res. mesh new file: scenarios/3denvar_O15kmI30km_AllSky.yaml new file: scenarios/3denvar_O15kmI30km_SpecifiedEnsemble_VarBC.yaml new file: scenarios/3dhybrid_O15-3kmI30km_SpecifiedEnsemble_VarBC.yaml new file: scenarios/3dhybrid_O15-3kmI30km_SpecifiedEnsemble_VarBC_New.yaml new file: scenarios/3dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC.yaml new file: scenarios/3dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml new file: scenarios/3dhybrid_O15kmI60km_SpecifiedEnsemble_VarBC_New.yaml new file: scenarios/3dhybrid_OIE30km_SpecifiedEnsemble_VarBC.yaml new file: scenarios/4dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml new file: scenarios/4dhybrid_O15kmI60km_SpecifiedEnsemble_VarBC_New.yaml new file: scenarios/GenerateGFSAnalyses_15km.yaml new file: scenarios/GenerateGFSAnalyses_15km3km.yaml

modified: config/jedi/ObsPlugs/enkf/ObsAnchors.yaml modified: config/jedi/applications/3dhybrid-allsky.yaml modified: config/jedi/applications/3dhybrid.yaml new file: config/jedi/applications/3dhybrid_v3.yaml new file: config/jedi/applications/4dhybrid_v3.yaml

modified: scenarios/defaults/enkf.yaml modified: scenarios/defaults/externalanalyses.yaml modified: scenarios/defaults/firstbackground.yaml modified: scenarios/defaults/forecast.yaml modified: scenarios/defaults/initic.yaml modified: scenarios/defaults/model.yaml modified: scenarios/defaults/variational.yaml

modified: bin/PrepJEDI.csh modified: bin/Variational.csh # use 'mpiexec -N' instead of 'mpiexec'

modified: initialize/applications/Variational.py modified: initialize/data/Model.py modified: initialize/framework/HPC.py

scenarios/3dhybrid-allsky_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml bin/PrepJEDI.csh scenarios/defaults/variational.yaml

modified: bin/PrepJEDI.csh new file: config/jedi/ObsPlugs/da/base/amsua-all_metop-a.yaml new file: config/jedi/ObsPlugs/da/base/amsua-all_metop-b.yaml new file: config/jedi/ObsPlugs/da/base/amsua-all_n15.yaml new file: config/jedi/ObsPlugs/da/base/amsua-all_n18.yaml new file: config/jedi/ObsPlugs/da/base/amsua-all_n19.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-all_metop-a.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-all_metop-b.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-all_n15.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-all_n18.yaml new file: config/jedi/ObsPlugs/da/bias/amsua-all_n19.yaml new file: config/jedi/ObsPlugs/da/filters/amsua-all_metop-a.yaml new file: config/jedi/ObsPlugs/da/filters/amsua-all_metop-b.yaml new file: config/jedi/ObsPlugs/da/filters/amsua-all_n15.yaml new file: config/jedi/ObsPlugs/da/filters/amsua-all_n18.yaml new file: config/jedi/ObsPlugs/da/filters/amsua-all_n19.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_metop-a.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_metop-b.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_n15.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_n18.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_n19.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/amsua-all_n19.yaml_cold-air-outbreak new file: scenarios/1_3dhyb_asr-ama_csr-mhs.yaml modified: scenarios/defaults/observations.yaml

new file: config/jedi/ObsPlugs/da/base/mhs-cld_metop-a.yaml new file: config/jedi/ObsPlugs/da/base/mhs-cld_metop-b.yaml new file: config/jedi/ObsPlugs/da/base/mhs-cld_n18.yaml new file: config/jedi/ObsPlugs/da/base/mhs-cld_n19.yaml new file: config/jedi/ObsPlugs/da/bias/mhs-cld_metop-a.yaml new file: config/jedi/ObsPlugs/da/bias/mhs-cld_metop-b.yaml new file: config/jedi/ObsPlugs/da/bias/mhs-cld_n18.yaml new file: config/jedi/ObsPlugs/da/bias/mhs-cld_n19.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/mhs-cld_metop-a.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/mhs-cld_metop-b.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/mhs-cld_n18.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/mhs-cld_n19.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-a.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-b.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_n18.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_n19.yaml new file: scenarios/2_3dhyb_asr-ama_asr-mhs.yaml modified: scenarios/defaults/observations.yaml

modified: config/jedi/ObsPlugs/da/base/mhs_n19_rttov.yaml

modified: config/jedi/ObsPlugs/da/bias/amsua_aqua_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/amsua_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/amsua_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/amsua_n15_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/amsua_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/amsua_n19_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/iasi_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/mhs_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/mhs_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/mhs_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/bias/mhs_n19_rttov.yaml

modified: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml new file: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml_88ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml new file: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml_88ch modified: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-a.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-a.yaml_88ch modified: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-b.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-b.yaml_88ch modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/defaults/firstbackground.yaml

modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml

modified: scenarios/3_3dhyb_asrama_asrmhs_atms.yaml modified: scenarios/4_4dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_iasi_rttovcpp.yaml

modified: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml new file: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml_66ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml new file: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml_66ch modified: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-a.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-a.yaml_66ch modified: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-b.yaml new file: config/jedi/ObsPlugs/da/filtersWithBias/iasi_metop-b.yaml_66ch new file: scenarios/3_3dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml new file: scenarios/3_4dhyb_asrama_asrmhs_atms.yaml new file: scenarios/3_4dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml new file: scenarios/4_3dhyb_asrama_asrmhs_atms_iasiQCmore.yaml

Conflicts: bin/GetObs.csh bin/ObsToIODA.csh bin/Variational.csh build/CMakeLists.txt config/jedi/ObsPlugs/da/bias/amsua-cld_metop-a.yaml config/jedi/ObsPlugs/da/bias/amsua-cld_metop-b.yaml config/jedi/ObsPlugs/da/bias/amsua-cld_n15.yaml config/jedi/ObsPlugs/da/bias/amsua-cld_n18.yaml config/jedi/ObsPlugs/da/bias/amsua-cld_n19.yaml config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_metop-a.yaml config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_metop-b.yaml config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n15.yaml config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n18.yaml config/jedi/ObsPlugs/da/filtersWithBias/amsua-cld_n19.yaml config/jedi/ObsPlugs/enkf/ObsAnchors.yaml config/mpas/variables.csh initialize/applications/EnKF.py initialize/framework/Build.py initialize/framework/HPC.py initialize/post/VerifyModel.py initialize/post/VerifyObs.py initialize/suites/GenerateExternalAnalyses.py scenarios/defaults/forecast.yaml scenarios/defaults/variational.yaml test/testinput/ForecastFromGFSAnalysesMPT.yaml

modified: scenarios/defaults/variational.yaml

modified: bin/PrepJEDI.csh modified: config/jedi/applications/3dhybrid-allsky.yaml modified: config/jedi/applications/3dhybrid.yaml modified: config/jedi/applications/3dvar.yaml

modified: bin/GetGDASAnalysisFromFTP.csh modified: bin/GetGFSAnalysisFromFTP.csh modified: bin/GetObs.csh modified: bin/PrepJEDI.csh

byoung-joo · 2025-12-17T20:49:51Z

Hi @liujake, I have two questions below.

From the discussion with Junmei, amsua-all_metop-a.yaml and other amsua-all_*.yaml files, which define both the sounding and imaging channels of AMSU-A, are not used in the scientific experiment scenarios. Do we still want to keep the corresponding code/changes and merge into develop ?
Due to the automatic data compression on Campaign storage, the observation files are copied to Scratch. We can live with observation files on Scratch in a short term. Do we want to keep those on Campaign or on Scratch (i.e., scenarios/defaults/observations.yaml)?

…paign modified: initialize/data/Observations.py modified: scenarios/0_3dhyb_csrama_csrmhs.yaml modified: scenarios/0_3dhyb_csrama_csrmhs_rttovcpp.yaml modified: scenarios/1_3dhyb_asrama_csrmhs.yaml modified: scenarios/2_3dhyb_asrama_asrmhs.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml modified: scenarios/3_4dhyb_asrama_asrmhs_atms.yaml modified: scenarios/3_4dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml modified: scenarios/3dhybrid-allsky_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml modified: scenarios/3dhybrid_O30kmIE60km_SpecifiedEnsemble_VarBC_allConfig.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasiQCmore.yaml modified: scenarios/4_4dhyb_asrama_asrmhs_atms_iasi.yaml renamed: scenarios/5_3denvar120km_csrama_csrmhs_crtm241.yaml -> scenarios/5_3denvar120km_csrama_csrmhs_crtm3.yaml modified: scenarios/5_3denvar120km_csrama_csrmhs_rttov14.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_iasi_rttovcpp.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_rttov12.yaml

are revised accordingly. One duplicated scenario was deleted. deleted: config/jedi/applications/3dhybrid-allsky.yaml deleted: config/jedi/applications/3dhybrid_v3.yaml modified: initialize/applications/Variational.py modified: scenarios/0_3dhyb_csrama_csrmhs.yaml modified: scenarios/0_3dhyb_csrama_csrmhs_rttovcpp.yaml modified: scenarios/1_3dhyb_asrama_csrmhs.yaml modified: scenarios/2_3dhyb_asrama_asrmhs.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml deleted: scenarios/3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC.yaml modified: scenarios/3dhybrid_O15-3kmI30km_SpecifiedEnsemble_VarBC_New.yaml renamed: scenarios/3dhybrid-allsky_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml -> scenarios/3dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC_AMSUA_ALLSKY.yaml renamed: scenarios/3dhybrid-allsky_O30kmIE60km_SpecifiedEnsemble_VarBC_AMSUA.yaml -> scenarios/3dhybrid_O30kmIE60km_SpecifiedEnsemble_VarBC_AMSUA_ALLSKY.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasiQCmore.yaml modified: scenarios/5_3denvar120km_csrama_csrmhs_crtm3.yaml modified: scenarios/5_3denvar120km_csrama_csrmhs_rttov14.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_iasi_rttovcpp.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_rttov12.yaml modified: scenarios/defaults/variational.yaml

configuration (i.e., scenarios/defaults/variational.yaml). modified: scenarios/0_3dhyb_csrama_csrmhs.yaml modified: scenarios/0_3dhyb_csrama_csrmhs_rttovcpp.yaml modified: scenarios/1_3dhyb_asrama_csrmhs.yaml modified: scenarios/2_3dhyb_asrama_asrmhs.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms.yaml modified: scenarios/3_3dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml modified: scenarios/3_4dhyb_asrama_asrmhs_atms.yaml modified: scenarios/3_4dhyb_asrama_asrmhs_atms_UpdateVarBC.yaml modified: scenarios/3denvar_O15kmI30km_AllSky.yaml modified: scenarios/3denvar_O15kmI30km_SpecifiedEnsemble_VarBC.yaml modified: scenarios/3dhybrid_O15-3kmI30km_SpecifiedEnsemble_VarBC.yaml modified: scenarios/3dhybrid_O15-3kmI30km_SpecifiedEnsemble_VarBC_New.yaml modified: scenarios/3dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC.yaml modified: scenarios/3dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC_AMSUA_ALLSKY.yaml modified: scenarios/3dhybrid_O30kmIE60km_SpecifiedEnsemble_VarBC_allConfig.yaml modified: scenarios/3dhybrid_OIE30km_SpecifiedEnsemble_VarBC.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasiQCmore.yaml modified: scenarios/4_4dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/4dhybrid_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_iasi_rttovcpp.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_rttov12.yaml

liujake · 2025-12-17T21:57:20Z

Hi @liujake, I have two questions below.

From the discussion with Junmei, amsua-all_metop-a.yaml and other amsua-all_*.yaml files, which define both the sounding and imaging channels of AMSU-A, are not used in the scientific experiment scenarios. Do we still want to keep the corresponding code/changes and merge into develop ?

Due to the automatic data compression on Campaign storage, the observation files are copied to Scratch. We can live with observation files on Scratch in a short term. Do we want to keep those on Campaign or on Scratch (i.e., scenarios/defaults/observations.yaml)?

Yes. Those yamls not used, but I think we can keep them for potential further investigation in the future. And I think some yaml files under 'filters' can be removed as we mostly use 'filtersWithBias'. For now, let us still point obs and other input datasets on Scratch to ease our testing later.

byoung-joo · 2025-12-17T22:02:29Z

Yes. Those yamls not used, but I think we can keep them for potential further investigation in the future. And I think some yaml files under 'filters' can be removed as we mostly use 'filtersWithBias'. For now, let us still point obs and other input datasets on Scratch to ease our testing later.

Sounds good! I will proceed and test one of the scientific scenario yamls soon!

0_3dhyb_csrama_csrmhs_rttovcpp.yaml for initial testing modified: 0_3dhyb_csrama_csrmhs.yaml modified: 0_3dhyb_csrama_csrmhs_rttovcpp.yaml

byoung-joo · 2025-12-18T19:05:09Z

Please see my first test below.

With this draft PR, I have run 0_3dhyb_csrama_csrmhs.yaml scenario for the 1st cycle. You can find the working directory in /glade/derecho/scratch/bjung/pandac/bjung_0_3dhyb_csrama_csrmhs/CyclingDA/2018041500 .

Runtime:   1083.92 sec,  Memory: total:  1136.96 Gb, per task: min =     2.18 Gb, max =     7.26 Gb

This configuration is consistent with
/glade/derecho/scratch/jban/pandac/jban_0_3dhyb_csrama_csrmhs/CyclingDA/2018041500 (of release/3.0.3).

Runtime:   1053.95 sec,  Memory: total:  1290.73 Gb, per task: min =     2.45 Gb, max =     7.59 Gb

The runtime is similar. Current PR seems to require slightly lesser memory.
The CostJo term and norm reduction look similar in general.

Conflicts: config/jedi/applications/3dhybrid-allsky.yaml config/jedi/applications/3dhybrid.yaml config/jedi/applications/3dvar.yaml config/jedi/applications/4dhybrid.yaml scenarios/defaults/variational.yaml

byoung-joo · 2025-12-19T03:18:16Z

Hi @junmeiban,

In the bias correction predictor, emissivityJacobian is used for clear-sky AMSU-A, while it is not used for cloudy AMSU-A channels. I'd like to check if this configuration is correct.

@juhyejuhye,
emissivityJacobian predictor is not used for RTTOV, perhaps it is not available from RTTOV. Is my understanding correct ?

modified: config/jedi/applications/3denvar.yaml

junmeiban · 2025-12-19T14:47:53Z

Hi @junmeiban,

In the bias correction predictor, emissivityJacobian is used for clear-sky AMSU-A, while it is not used for cloudy AMSU-A channels. I'd like to check if this configuration is correct.

@juhyejuhye, emissivityJacobian predictor is not used for RTTOV, perhaps it is not available from RTTOV. Is my understanding correct ?

@byoung-joo , Yes. We turn the emissJacobian predictor off because it is applied over land (see https://github.com/JCSDA-internal/ufo/blob/56fabf58044b0593599600e599683c5d075a7cbc/src/ufo/predictors/Emissivity.cc#L61 and also line 55)), while cloudy AMSU-A (for window channels) is conducted over water only. @liujake made a commit note here: 4deb438

byoung-joo · 2025-12-19T16:27:55Z

Thanks, @junmeiban.

byoung-joo · 2025-12-22T17:04:22Z

This PR is almost ready for review.
@juhyejuhye will push some rttov-related modification to this branch.

…nges in the develop branch 1. Remove "get values:" in rttov yamal files in base directory 2. Change variable name from " brightness_temperature_jacobian_surface_temperature" to " brightness_temperature_jacobian_skin_temperature_at_surface" in rttov yamal files in filtersWithBias directory On branch feature/release3.0.3_merging_develop Changes to be committed: modified: config/jedi/ObsPlugs/da/base/amsua_aqua_rttov.yaml modified: config/jedi/ObsPlugs/da/base/amsua_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/base/amsua_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/base/amsua_n15_rttov.yaml modified: config/jedi/ObsPlugs/da/base/amsua_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/base/amsua_n19_rttov.yaml modified: config/jedi/ObsPlugs/da/base/mhs_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/base/mhs_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/base/mhs_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/base/mhs_n19_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_aqua_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_n15_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/amsua_n19_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_n18_rttov.yaml modified: config/jedi/ObsPlugs/da/filtersWithBias/mhs_n19_rttov.yaml

modified: config/jedi/ObsPlugs/da/base/amsua-all_metop-a.yaml modified: config/jedi/ObsPlugs/da/base/amsua-all_metop-b.yaml modified: config/jedi/ObsPlugs/da/base/amsua-all_n15.yaml modified: config/jedi/ObsPlugs/da/base/amsua-all_n18.yaml modified: config/jedi/ObsPlugs/da/base/amsua-all_n19.yaml modified: config/jedi/ObsPlugs/da/base/amsua-cld_metop-c.yaml modified: config/jedi/ObsPlugs/da/base/amsua_n18_rttov12.yaml modified: config/jedi/ObsPlugs/da/base/atms_n20.yaml modified: config/jedi/ObsPlugs/da/base/atms_n21.yaml modified: config/jedi/ObsPlugs/da/base/atms_npp.yaml modified: config/jedi/ObsPlugs/da/base/cris-fsr_n20.yaml modified: config/jedi/ObsPlugs/da/base/cris-fsr_n21.yaml modified: config/jedi/ObsPlugs/da/base/cris-fsr_npp.yaml modified: config/jedi/ObsPlugs/da/base/cris_n20.yaml modified: config/jedi/ObsPlugs/da/base/cris_npp.yaml modified: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml_66ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-a.yaml_88ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-a_rttov.yaml modified: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml_66ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-b.yaml_88ch modified: config/jedi/ObsPlugs/da/base/iasi_metop-b_rttov.yaml modified: config/jedi/ObsPlugs/da/base/iasi_metop-c.yaml modified: config/jedi/ObsPlugs/da/base/mhs-cld_metop-a.yaml modified: config/jedi/ObsPlugs/da/base/mhs-cld_metop-b.yaml modified: config/jedi/ObsPlugs/da/base/mhs-cld_metop-c.yaml modified: config/jedi/ObsPlugs/da/base/mhs-cld_n18.yaml modified: config/jedi/ObsPlugs/da/base/mhs-cld_n19.yaml modified: config/jedi/ObsPlugs/da/base/mhs_metop-c.yaml modified: config/jedi/ObsPlugs/da/base/mhs_n18_rttov12.yaml modified: config/jedi/ObsPlugs/hofx/base/cris-fsr_n20.yaml modified: config/jedi/ObsPlugs/hofx/base/cris-fsr_n21.yaml modified: config/jedi/ObsPlugs/hofx/base/cris-fsr_npp.yaml modified: config/jedi/ObsPlugs/hofx/base/cris_n20.yaml modified: config/jedi/ObsPlugs/hofx/base/cris_npp.yaml modified: config/jedi/ObsPlugs/hofx/base/mhs_metop-c.yaml

junmeiban and others added 30 commits October 16, 2024 14:46

Change obs error for some GNSSRO files, update build, and

0daf7b3

CMakeLists for new build

Fix scenario scenarios/3denvar_O30kmIE60km_WarmStart.yaml

aa8a007

Update obsspace yaml

bf0b140

Changes for all-sky amsua

5be085b

Changes to scenario for all-sky amsua

da52d4a

Merge branch 'release/3.0.0' into feature/changes4GNSSRO_3.0.2_allsky…

552df14

…AMSUA

Remove blank space and comment out restart point, change back the

4ddfddf

3dhybrid scenario

Add yamls for using Harnish obs error as obs error model and varbc

262d045

predictor along with other predictors, for channels 8-10 and channel 13

Add "reduce obs space" in thining to reduce memory usage. (#347)

a539d79

Description Add 'reduce obs spcace' function to the thinning section to reduce memory usage. With this function, those observations removed by thinning will not come to memory and will not be in the obsout_da files.

Remove abi/ahi settings (#351)

574d838

### Description Previous PR [348](#348) didn't work. Restoring abi/ahi settings and removing additional yamls here.

Add missing InvariantStream to GenerateExternalAnalyses.py (#357)

fa0da6e

Use built-in error tables for PrepBUFR files (#366)

6f72498

* Enforce usage of internal error table for prepbufr files * Clean up specification of obs2ioda build

Use crtm-2.4.1's coeffs and mpas-bundle-3.0.3-based code build

7257bda

with changes related to VarBC/Harnisch-error model and rttov14 modified: initialize/data/Observations.py modified: initialize/framework/Build.py

This tool is to handle 15km-3km/15km DA analysis I/O issue

883f441

new file: tools/update_analysis_states.py

modified: bin/ExternalAnalysisToMPAS.csh

b881e75

modified: bin/PrepJEDI.csh modified: bin/Variational.csh # use 'mpiexec -N' instead of 'mpiexec'

modified: initialize/applications/EnKF.py

8dd352c

modified: initialize/applications/Variational.py modified: initialize/data/Model.py modified: initialize/framework/HPC.py

Modified for amsua allsky:

6aa7301

scenarios/3dhybrid-allsky_O15kmI30km_SpecifiedEnsemble_VarBC_New.yaml bin/PrepJEDI.csh scenarios/defaults/variational.yaml

liujake and others added 14 commits August 29, 2025 16:29

Fixed a bug in using a wrong rttov coef file.

1fc116b

modified: config/jedi/ObsPlugs/da/base/mhs_n19_rttov.yaml

beginning VarBC files link to 2018042412's ones in exp3

32ee260

modified: scenarios/4_3dhyb_asrama_asrmhs_atms_iasi.yaml

modified: scenarios/0_3dhyb_csrama_csrmhs_rttovcpp.yaml

c4bdf1d

modified: scenarios/3_3dhyb_asrama_asrmhs_atms.yaml modified: scenarios/4_4dhyb_asrama_asrmhs_atms_iasi.yaml modified: scenarios/6_3dhyb_csrama_csrmhs_iasi_rttovcpp.yaml

adding yaml files for amsua-cld_metop-c

fdc4c91

adding Observation Range Sanity Check for atms

956c390

Merge branch 'develop' into feature/release3.0.3_merging_develop

0fa6e03

update prefix

a2e97d1

modified: scenarios/defaults/variational.yaml

distinguish 2D and 3D variables in NICAS (covariance)

89629b7

modified: bin/PrepJEDI.csh modified: config/jedi/applications/3dhybrid-allsky.yaml modified: config/jedi/applications/3dhybrid.yaml modified: config/jedi/applications/3dvar.yaml

Merge branch 'develop' into feature/release3.0.3_merging_develop

63496d9

WIP: Remove unnecessary comment

9520d82

modified: bin/GetGDASAnalysisFromFTP.csh modified: bin/GetGFSAnalysisFromFTP.csh modified: bin/GetObs.csh modified: bin/PrepJEDI.csh

byoung-joo added 3 commits December 17, 2025 14:11

revise 0_3dhyb_csrama_csrmhs.yaml and

fe5f632

0_3dhyb_csrama_csrmhs_rttovcpp.yaml for initial testing modified: 0_3dhyb_csrama_csrmhs.yaml modified: 0_3dhyb_csrama_csrmhs_rttovcpp.yaml

Merge branch 'develop' into feature/release3.0.3_merging_develop

afec44d

Conflicts: config/jedi/applications/3dhybrid-allsky.yaml config/jedi/applications/3dhybrid.yaml config/jedi/applications/3dvar.yaml config/jedi/applications/4dhybrid.yaml scenarios/defaults/variational.yaml

activate no outer loop update for 3denvar.yaml

9656c5c

modified: config/jedi/applications/3denvar.yaml

juhyejuhye and others added 2 commits December 22, 2025 12:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[draft] merging the changes/improvements in `feature/release3.0.3` #409

[draft] merging the changes/improvements in `feature/release3.0.3` #409

Uh oh!

byoung-joo commented Dec 17, 2025 •

edited

Loading

Uh oh!

byoung-joo commented Dec 17, 2025 •

edited

Loading

Uh oh!

liujake commented Dec 17, 2025

Uh oh!

byoung-joo commented Dec 17, 2025

Uh oh!

byoung-joo commented Dec 18, 2025

Uh oh!

byoung-joo commented Dec 19, 2025

Uh oh!

junmeiban commented Dec 19, 2025

Uh oh!

byoung-joo commented Dec 19, 2025

Uh oh!

byoung-joo commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

[draft] merging the changes/improvements in feature/release3.0.3 #409

Are you sure you want to change the base?

[draft] merging the changes/improvements in feature/release3.0.3 #409

Uh oh!

Conversation

byoung-joo commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Work in progress

Description

Uh oh!

byoung-joo commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liujake commented Dec 17, 2025

Uh oh!

byoung-joo commented Dec 17, 2025

Uh oh!

byoung-joo commented Dec 18, 2025

Uh oh!

byoung-joo commented Dec 19, 2025

Uh oh!

junmeiban commented Dec 19, 2025

Uh oh!

byoung-joo commented Dec 19, 2025

Uh oh!

byoung-joo commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

[draft] merging the changes/improvements in `feature/release3.0.3` #409

[draft] merging the changes/improvements in `feature/release3.0.3` #409

byoung-joo commented Dec 17, 2025 •

edited

Loading

byoung-joo commented Dec 17, 2025 •

edited

Loading