-
Notifications
You must be signed in to change notification settings - Fork 43
Merge jcb-gdas into gdasapp #2031
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Add some yaml files
minor adjustments required by g-w JEDI ATM CI
and permanent changes to jcb-gdas client integration
…poses (#34) This PR modifies JCB client integration testing to allow for JCB apps to be subdirectories of larger repositories rather than stand-alone repositories. This will allow the merger of `jcb-gdas` into `gdasapp` while moving JCB client integration testing into the `gdasapp` repo on GitHub. A new optional key, 'app_subdir', is now permitted in `jcb_clients.yaml` that indicates the relative path of the JCB app in its parent repo. I tested these changes in two ways: 1. I created a `jcb-gdas` PR, NOAA-EMC/jcb-gdas#226 , setting the `jcb` repo branch for GitHub CI testing to this branch. GitHub CI passes successfully. 2. I created a `gdasapp` PR, NOAA-EMC/GDASApp#2031, that merged `jcb-gdas` into `gdasapp` as a subdirectory of that repo, and with the GitHub CI workflows for `jcb-gdas` into `gdasapp`. Again, I set the jcb` repo branch for GitHub CI testing to this branch, and it passes CI successly. --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
|
@DavidNew-NOAA : Thank you for opening this PR to move jcb-gdas into GDASApp. Two comments:
|
|
@DavidNew-NOAA and @CoryMartin-NOAA : The JCB client test is failing. Something in my merge of I installed |
|
Thanks @RussTreadon-NOAA , I'll take a look. My recent changes to jcb must have broken something |
|
WCOSS2 ctests The atm_jjob failures stem from the init jobs. The init jobs failed with the same error atm_jjob_var_init failed with atm_jjob_ens_init_split and atm_jjob_ens_init failed with
GDASApp A similar comment applies to Manually copy the observation atmosphere and atmosphere-letkf @DavidNew-NOAA : We need to update The current head of jcb-gdas @CoryMartin-NOAA are you aware of any other changes in jcb-gdas which we need to replicate in this PR? |
|
@RussTreadon-NOAA I managed to fix the jcb client integration test. I had commented out some lines in the github workflow script for testing and forgot to revert those changes. I'm going to need to redo this PR by the way since it's out of date with |
|
@DavidNew-NOAA . Thanks for the update. I reached out to @jianjunj and @bhuang95 for status updates on their jcb-gdas PRs. Once the path forward for their PRs is clear, we should inform developers that new jcb-gdas PRs will no longer be included in GDASApp. Developers with changes for jcb-gdas will need to open issues and PRs in the GDASApp after this PR is merged into GDASApp |
|
Thanks @RussTreadon-NOAA . Let's see what they say. It really won't be a lot of work for them to create new GDASApp PRs from their existing JCB-GDAS PRs, so I'm hoping they might volunteer to close the latter so we can move forward on this process of merging the two repositories. |
|
@RussTreadon-NOAA With all |
|
@RussTreadon-NOAA I was able to update this PR with the latest jcb-gdas develop changes, so this PR should be ready to go. |
RussTreadon-NOAA
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GDASApp test_atm_jjob_init jobs are failing for
30/144 Test #307: test_gdasapp_atm_jjob_var_init ...........................................***Failed 187.09 sec
31/144 Test #311: test_gdasapp_atm_jjob_ens_init ...........................................***Failed 202.91 sec
54/144 Test #313: test_gdasapp_atm_jjob_ens_init_split .....................................***Failed 68.20 sec
the same reason as before
wxflow.exceptions.WorkflowException: An error occurred while rendering JCB template for algorithm 3dvar:
Resolving templates for 3dvar failed with the following exception:
radiance_amsua_n19.yaml.j2
wxflow.exceptions.WorkflowException: An error occurred while rendering JCB template for algorithm local_ensemble_da_observer:
Resolving templates for local_ensemble_da_observer failed with the following exception:
radiance_amsua_n19.yaml.j2
parm/jcb-gdas/observations/atmosphere from this PR is missing the renamed observation files found in jcb-gdas observations/atmosphere
|
@RussTreadon-NOAA Dang. I was having trouble repeating the steps I made to create this PR. I may just need to debug those steps and start over. |
|
@DavidNew-NOAA . I interacted with Gemini to chart a way to replace the I created feature/test, cloned it on Dogwood, and executed the following commands returned by Gemini To confirm that history was preserved I ran the following test The history from e177e61 and thereafter agrees with what I see from https://github.com/NOAA-EMC/jcb-gdas/commits/develop/ I see the expected files in feature/test parm/jcb-gdas/observations/atmosphere. I not sure if this achieves what we want to do in the best way. I'll leave feature/test out there for you and other to examine. I'll delete it when we no longer need it. |
# Description This PR merges the jcb-gdas repository into gdasapp. Importantly, the git history is retained. The client integration testing for the jcb-gdas repo is moved into the GitHub workflows for this repo. This PR replaces #2031 # Companion PRs N/A # Issues # Automated CI tests to run in Global Workflow <!-- Which Global Workflow CI tests are required to adequately test this PR? --> - [X] atm_jjob <!-- JEDI atm single cycle DA !--> - [X] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !--> - [X] C96C48_hybatmsnowDA <!-- JEDI snow cycled DA !--> - [X] C96_gcafs_cycled <!-- JEDI aerosol cycled DA !--> - [X] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA !--> - [X] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !--> - [X] C96C48_ufsgsi_hybatmDA <!-- JEDI atm Var with GSI EnKF cycled DA !--> - [X] C96C48_hybatmDA <!-- GSI atm cycled DA !--> --------- Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> Co-authored-by: danholdaway <danholdaway@users.noreply.github.com> Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com> Co-authored-by: RussTreadon-NOAA <Russ.Treadon@noaa.gov> Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com> Co-authored-by: BrettHoover-NOAA <98188219+BrettHoover-NOAA@users.noreply.github.com> Co-authored-by: Brett Hoover <bhoover@orion-login-3.hpc.msstate.edu> Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: Emily Liu <eliu@hercules-login-2.hpc.msstate.edu> Co-authored-by: Azadeh Gholoubi <51101867+azadeh-gh@users.noreply.github.com> Co-authored-by: Azadeh Gholoubi <gholoubi@hercules-login-2.hpc.msstate.edu> Co-authored-by: Azadeh Gholoubi <gholoubi@hercules-login-3.hpc.msstate.edu> Co-authored-by: Azadeh Gholoubi <gholoubi@hercules-login-4.hpc.msstate.edu> Co-authored-by: JohnSteffen-NOAA <77806417+JohnSteffen-NOAA@users.noreply.github.com> Co-authored-by: Anna Shlyaeva <annash@ucar.edu> Co-authored-by: Jiarui Dong <Jiarui.Dong@noaa.gov> Co-authored-by: yuanxue2870 <136842224+yuanxue2870@users.noreply.github.com> Co-authored-by: Kevin Dougherty <69815622+kevindougherty-noaa@users.noreply.github.com> Co-authored-by: Cory Martin <Cory.R.Martin@gaea65.ncrc.gov> Co-authored-by: Anna Shlyaeva <anna.v.shlyaeva@noaa.gov> Co-authored-by: Yaping Wang <49168260+ypwang19@users.noreply.github.com> Co-authored-by: ypwang19 <yaping.wang@users.noreply.github.com> Co-authored-by: jianjunj <jianjun.jin@noaa.gov> Co-authored-by: Xuanli Li <101414760+XuanliLi-NOAA@users.noreply.github.com> Co-authored-by: Travis Elless <113720457+TravisElless-NOAA@users.noreply.github.com> Co-authored-by: AndrewEichmann-NOAA <58948505+AndrewEichmann-NOAA@users.noreply.github.com> Co-authored-by: john steffen <john.steffen@dlogin07.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: john steffen <john.steffen@dlogin08.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Nicholas Esposito <esposito@hercules-login-2.hpc.msstate.edu> Co-authored-by: Nicholas Esposito <esposito@hercules-login-4.hpc.msstate.edu> Co-authored-by: Nicholas Esposito <esposito@hercules-login-3.hpc.msstate.edu> Co-authored-by: john steffen <john.steffen@clogin09.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: guillaumevernieres <14031856+guillaumevernieres@users.noreply.github.com> Co-authored-by: Andy Tangborn <71304161+andytangborn@users.noreply.github.com> Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com> Co-authored-by: Tseganeh Z. Gichamo <zacctsega@yahoo.com> Co-authored-by: CoryMartin-NOAA <6354668+CoryMartin-NOAA@users.noreply.github.com> Co-authored-by: Bo Huang <40745309+bhuang95@users.noreply.github.com>
Description
This PR merges the jcb-gdas repository into gdasapp. Importantly, the git history is retained. The client integration testing for the jcb-gdas repo is moved into this the GitHub workflows for this repo.
Companion PRs
N/A
Issues
Automated CI tests to run in Global Workflow