Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set runtime shell requirements within Jenkins Pipeline #3171

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

TerrenceMcGuinness-NOAA
Copy link
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA commented Dec 16, 2024

Description

This PR updates the Jenkins Pipeline script to source gw_setup.sh from within the PR itself for when ever a shell command is issued that needs it.

Currently the runtime bash shell environment requirements for running the global-workflow (set by sourcing $HOMEgfs/workflow/gw_setup.sh) is established by the user's shell that makes the connection from the Java remote agent to the Jenkins Controller. This update simply sources gw_setup.sh directly during run time when ever needed instead.

NOTE: This does not apply for GitHub CLI and pyGitHub since they are not currently supported by Spack Stack and therefore not incorporated in to gw_setup.sh. These remaining dependances are the responsibility of the role account holders that manage the Jenkins Controller to Node Agent connections.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Copy link
Contributor

@AnilKumar-NOAA AnilKumar-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have added HOMEGFS to the runtime shell environment for gaea C5/C6 both in our epic role account .bashrc file as follows:

Added for Global Workflow

if [ -n "${HOMEGFS+x}" ] && [ -n "${HOMEGFS}" ]; then
source ${HOMEGFS}/workflow/gw_setup.sh
fi

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA marked this pull request as draft December 17, 2024 18:46
@TerrenceMcGuinness-NOAA
Copy link
Collaborator Author

@AnilKumar-NOAA Oh I see now that you were referring to the .bashrc in the roll account. In a Jenkins Pipeline the shell commands are not login shells but are issued from the shell environment from which the agent had been made to make the connection to the controller. Having said that, I will have to make some adjustments to the wrapper scripts so the needed modules can be set at runtime. Let me make some updates to the wrapper scripts in the CI framework and run some tests from a generic shell. In the meantime this PR has been reverted to a Draft

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA changed the title Set runtime HOMEgfs in Jenkins to support config.env Set runtime HOMEgfs in Jenkins to support BASH_ENV Dec 20, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion label Dec 20, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA marked this pull request as ready for review December 20, 2024 19:44
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 20, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 20, 2024
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 20, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 21, 2024
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Dec 21, 2024
@emcbot
Copy link

emcbot commented Dec 21, 2024

Checkout Failed on Orion in Build# 6: Could not perform submodule update

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 21, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed label Dec 21, 2024
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Dec 21, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress label Dec 23, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA marked this pull request as draft December 23, 2024 18:20
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA changed the title Set runtime HOMEgfs in Jenkins to support BASH_ENV Set runtime shell requirements within Jenkins Pipeline Dec 26, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion label Dec 26, 2024
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Dec 26, 2024
@emcbot
Copy link

emcbot commented Dec 26, 2024

CI Passed on Orion in Build# 1
Built and ran in directory /work2/noaa/stmp/CI/ORION/3171


Experiment C48_ATM_8bafe297 Completed 2 Cycles: *SUCCESS* at Thu Dec 26 11:50:34 AM CST 2024
Experiment C96_S2SWA_gefs_replay_ics_8bafe297 Completed 1 Cycles: *SUCCESS* at Thu Dec 26 12:10:03 PM CST 2024
Experiment C96_atm3DVar_8bafe297 Completed 3 Cycles: *SUCCESS* at Thu Dec 26 01:16:04 PM CST 2024
Experiment C96C48_hybatmDA_8bafe297 Completed 3 Cycles: *SUCCESS* at Thu Dec 26 01:28:23 PM CST 2024
Experiment C48_S2SW_8bafe297 Completed 2 Cycles: *SUCCESS* at Thu Dec 26 02:25:26 PM CST 2024
Experiment C48_S2SWA_gefs_8bafe297 Completed 1 Cycles: *SUCCESS* at Thu Dec 26 02:51:50 PM CST 2024

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA marked this pull request as ready for review December 26, 2024 21:31
… experment which should be its sole purpose, also did a which on gh incase it is not in ~/bin
export GH="${HOME}/bin/gh"
[[ -f "${GH}" ]] || echo "gh is not installed in ${HOME}/bin"
GH=$(command -v gh || echo "${HOME}/bin/gh")
[[ -f "${GH}" ]] || echo "ERROR: GitHub CLI (gh) not found. (exiting with error)"; exit 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is what I think you intend. Otherwise, exit 1will be called regardless of the result.

Suggested change
[[ -f "${GH}" ]] || echo "ERROR: GitHub CLI (gh) not found. (exiting with error)"; exit 1
[[ -f "${GH}" ]] || ( echo "ERROR: GitHub CLI (gh) not found. (exiting with error)"; exit 1 )

Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes to source gw_setup.sh are good.
A comment at the end of this PR re. GH might solve the shellnorm error. Please check.
Approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Issue related to CI/CD CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants