Make scheduler jobs configurable #503
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
documentation
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3099/Result ✅ SUCCESSBIRDHOUSE_DEPLOY_BRANCH : configurable-crontab DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/68/NOTEBOOK TEST RESULTS |
fmigneault
reviewed
Mar 7, 2025
| @@ -0,0 +1,3 @@ | |||
| if [ -z "${BIRDHOUSE_SCHEDULER_NOTEBOOKDEPLOY_JOB}${BIRDHOUSE_SCHEDULER_AUTODEPLOY_JOB}${BIRDHOUSE_SCHEDULER_LOGROTATE_JOB}${BIRDHOUSE_AUTODEPLOY_EXTRA_SCHEDULER_JOBS}" ] ; then | |||
| log WARN 'The scheduler job has been enabled but no jobs are configured to be run. Please enable some jobs or disable the scheduler' | |||
There was a problem hiding this comment.
Maybe use components/scheduler in the message to be explicit about what it referred here, just in case there are other "scheduler" logs happening somewhere else.
| Scheduler jobs can be enabled by sourcing specific ``*.env`` files in the ``components/scheduler`` | ||
| directory. The jobs that come included with the Birdhouse code are as follows: | ||
|
|
||
| * Enable automatic deployment: |
There was a problem hiding this comment.
Spacing for all of these does not seem to render correctly.
Probably needs a newline between the nested lists/code blocks.
I could see an anchor to specific subsections of the doc being useful here, so maybe consider using ^^^ header/separators here as well.
| . $COMPOSE_DIR/components/scheduler/logrotate_job.env | ||
|
|
||
| * Automatically update tutorial Jupyter notebooks: | ||
| * note that this only can be enabled if the ``jupyterhub`` component is also enabled |
There was a problem hiding this comment.
Should the dependency be applied dynamically when sourcing its env file?
This could be relevant if the sourced env makes use of a jupyterhub-specific config that must be sourced first.
Same concept of the below thredds cases.
Comment on lines
+2
to
+4
| ${BIRDHOUSE_SCHEDULER_NOTEBOOKDEPLOY_JOB} | ||
| ${BIRDHOUSE_SCHEDULER_AUTODEPLOY_JOB} | ||
| ${BIRDHOUSE_SCHEDULER_LOGROTATE_JOB} |
There was a problem hiding this comment.
I don't mind this approach particularly, but it seems somewhat a random combination whether specific scheduler config-vars are used vs the BIRDHOUSE_AUTODEPLOY_EXTRA_SCHEDULER_JOBS one.
Should the provided scheduler configs all use explicit variables, and BIRDHOUSE_AUTODEPLOY_EXTRA_SCHEDULER_JOBS would be left empty for users to easily extend with their private configs as they deem fit, without dealing with prepend/append other ones added dynamically?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
The scheduler component automatically enables three jobs (autodeploy, logrotate, notebookdeploy). If someone wants to use the scheduler component but does not want these jobs, there is no obvious way to disable any one of these jobs.
This change makes it possible to enable/disable jobs as required by the user and adds documentation to explain how to do this.
Breaking Change:
*.envfiles at the end of your local environment file(see documentation for further details)
What about... ?
'#'string?However, this is not obvious to the user and is unreliable since it is not documented.
Changes
Non-breaking changes
Breaking changes
Related Issue / Discussion
Additional Information
I would really like to use the scheduler for other things but I do not want to have to enable the autodeploy mechanism.
It seems like the scheduler component was designed with the autodeploy mechanism in mind since that is added by default but it could be much more useful if it was more configurable.
CI Operations
birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false