Cowbird jupyter e2e test #404
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cowbird-jupyter-e2e-test
…secure-data-proxy
github-actions
bot
added
ci/tests
This was referenced Nov 15, 2023
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2246/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// Infrastructure deployment failed. Instance has not been destroyed. @matprov |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2247/Result : success BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : test-cowbird-jupyter-access PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : false PAVICS_HOST : https://host-140-46.rdext.crim.ca Infrastructure deployment failed. Instance has not been destroyed. @matprov |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2248/Result : success BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : test-cowbird-jupyter-access PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : false PAVICS_HOST : https://host-140-20.rdext.crim.ca Infrastructure deployment failed. Instance has not been destroyed. @matprov |
fmigneault
reviewed
Nov 15, 2023
There was a problem hiding this comment.
Missing a new section in https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/optional-components/README.rst
This section should warn very explicitly that this component should not be used in non-test environments, as it opens public access for certain endpoints, defines admin-tokens for a JupyterHub user for which credentials are clearly visible in the script, and enforces use of root access for the test container. The component is for validation only. If used in a prod stack, it creates a security vulnerability.
birdhouse/components/cowbird/config/cowbird/config.yml.template
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/default.env
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/default.env
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/docker-compose-extra.yml
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/test_cowbird_jupyter.py
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/test_cowbird_jupyter.py
Outdated
Show resolved
Hide resolved
birdhouse/optional-components/test-cowbird-jupyter-access/test_cowbird_jupyter.py
Outdated
Show resolved
Hide resolved
tlvu
reviewed
Nov 16, 2023
birdhouse/optional-components/test-cowbird-jupyter-access/default.env
Outdated
Show resolved
Hide resolved
tlvu
reviewed
Nov 16, 2023
birdhouse/optional-components/test-cowbird-jupyter-access/default.env
Outdated
Show resolved
Hide resolved
tlvu
reviewed
Nov 16, 2023
|
|
||
| import json | ||
| import os | ||
| import requests |
There was a problem hiding this comment.
This script basically only need to make http requests, I think any pavics/workflow-tests is fine, no need for the specific version 210216.
And is this script a run once script or it has to continuously run and listen to connections?
If not, can we make it called only once by https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/scripts/bootstrap-instance-for-testsuite instead?
There was a problem hiding this comment.
Run once and forget.
However, the bootstrap script is not called by our Jenkins CI tests.
Only the PAVICS-E2E testall for running the notebooks are called on the instance swanned with this component added.
There was a problem hiding this comment.
However, the bootstrap script is not called by our Jenkins CI tests.
I am very surprised the boostrap script is called not call by your Jenkins CI tests. That boostrap entrypoint was specifically created for CI pipeline to call, so that new boostrap steps can be added transparently, without having to change the CI steps.
There was a problem hiding this comment.
The bootstrap entrypoint is called.
http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2255/console
[RESULT] bootstrap-instance-for-testsuite script finished. Exiting.
github-actions
bot
added
the
feature/WPS
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2280/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/40/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2284/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-133.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/42/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2285/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/43/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2288/Result : success BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/44/NOTEBOOK TEST RESULTS |
…cowbird-jupyter-e2e-test
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2289/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// Infrastructure deployment failed. Instance has not been destroyed. @matprov |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2290/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/46/NOTEBOOK TEST RESULTS |
|
@tlvu @fmigneault |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2293/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/47/NOTEBOOK TEST RESULTS |
fmigneault
approved these changes
Nov 21, 2023
tlvu
requested changes
Nov 21, 2023
There was a problem hiding this comment.
Since the boostrap entrypoint is called, see
http://daccs-jenkins.crim.ca/job/DACCS-iac-birdhouse/2255/console
[RESULT] bootstrap-instance-for-testsuite script finished. Exiting.
Please make it a new step in that entrypoint (a new script called by that entrypoint) instead of a new service just for a run-once and forget step.
My bad. Seems I was wrong. However, since the optional component doesn't only create permissions and test files, but also defines specific DockerSpawner configuration overrides needed to test the feature, I think a component would be required no matter what to integrate with other |
|
Same version is fine. |
tlvu
requested changes
Nov 22, 2023
| services: | ||
| jupyterhub: | ||
| environment: | ||
| TEST_COWBIRD_JUPYTERHUB_USERNAME: ${TEST_COWBIRD_JUPYTERHUB_USERNAME} |
There was a problem hiding this comment.
So this is the only change to other service (jupyterhub) than add the new service (test-cowbird-jupyter-access) just to be able to run the script once.
The jupyterhub service modification is not even necessary for Jenkins since Jenkins do not run on the PAVICS platform.
So I do not see why this bootstrap script can not be called by the bootstrap entrypoint instead of being in a new component that is not really a service.
Now I am a bit lost as to why Jenkins works properly since none of all the extra changes to jupyterhub component is ever taken by Jenkins. Aka, all the extra configs in JUPYTERHUB_CONFIG_OVERRIDE is not seen at all by Jenkins.
There was a problem hiding this comment.
Jenkins do not run on the PAVICS platform.
Just to clarify on this point, Jenkins do test against the PAVICS platform, but the Jupyter env runtime environment that the test notebooks runs on Jenkins is not spawned by the JupyterHub component, so any changes to the JupyterHub component is not seen/tested by Jenkins.
So if the test notebook works under Jenkins, it means the changes to the JupyterHub component might not be required. Maybe the new bootstrap script is all that is required, no need for JupyterHub config extra config changes.
Please either review the notebook or the necessity of JupyterHub extra configs.
There was a problem hiding this comment.
The way my test notebooks work is this :
-
The main notebook that is executed first is indeed not spawned by the JupyterHub component, but it does use the JupyterHub component. I highlighted the cells that use JuptyerHub here. It checks first that the test user exists on JupyterHub, gets its token, and then uses the
jhub-clientpackage to spawn a new JupyterLab with the usual DockerSpawner. So, the JupyterHub is indeed tested here. -
The second notebook is the notebook that is tested against the PAVICS platform, on the JupyterHub environment.
So, the changes to the JupyterHub component are used to add role permissions to be able to do the operations on the main notebook (creating a user, getting a user token), and the environment variables added to the DockerSpawner environment are used by the second notebook's test.
There was a problem hiding this comment.
@ChaamC Oh sorry I didn't have time to take a look at those notebooks. Did not realized the 1st notebook spawn the 2nd notebook so those new JupyterHub config are actually needed and exercised.
I guess we can keep this component that is not really a service right now to make things simple for you.
I guess, technically, we can have the default.env and the docker-compose-extra.yml with only the new added TEST_COWBIRD_JUPYTERHUB_USERNAME env var to the jupyterhub container. All the part about the new test-cowbird-jupyter-access new container can be a new boostrap script that is called only once.
Keeping this component that is not really a service, each time we restart the stack, that script that is meant to run once, will re-run again, any problem with that?
I understand this component will not be enabled on production so I don't care if ./pavics-compose.sh ps will probably report this test-cowbird-jupyter-access new container "Down" because it is not really service.
However, this component will more than likely be enabled permanently on our various test systems, just want to make sure it does not interfere negatively with other components or producing error when running more than once?
There was a problem hiding this comment.
@tlvu
Sadly, @ChaamC as already spent ~2x the amount of time originally planned for this month to complete this test validation, given how complex it can get with the double nesting of notebook executions, Jupyter API/kernel interactions, combined with Magpie/Cowbird permissions. Given the month is nearing the end, he will not have further time to refactor this.
Since there are little to no validation tests of the Jupyter/Cowbird interactions at the moment, I would like to suggest moving forward with this first implementation, and consider refactors if really judged necessary at a later point. IMO, given that part of the configuration must be done before the stack is started (Jupyter config/token), and other after (test user/permission setup), I believe that splitting them up into bootstrap/optional-component will only make this validation even more complicated than it already is. It will be much easier for someone to work on this feature if all configurations and scripts are combined under the same optional-component and executed roughly at the same time on startup.
I would like also to point out that restarting the stack should not restart that service each time if it is set to restart: "no". It will also not report that it is down, but that it did a successful "Exit 0". The only thing that @ChaamC could do as final check is that conflict conditions that could make the script fail are handled gracefully in case the service is forcefully restarted.
There was a problem hiding this comment.
@fmigneault @ChaamC I wasn't against leaving it as-is. I realized it's a bit late to change the design. I was just inquiring if any side-effects have been anticipated (ex: would re-run cause errors on force restart, anything else ...)
There was a problem hiding this comment.
So, if the test-cowbird-jupyter-access container from the optional-component is restarted, its script will get reexecuted, but the first thing the script does, apart from some variable assignations and some checks, is to try to create the test user on Magpie. If the user already exists, it raises an error and stops. Considering this, I don't think there should be any side effect if the stack is restarted, as it won't do anything if the test user already exists.
The effects brought by the script are :
- the creation of a test user Magpie
- creating some files in the test user's own workspace
- creating public and user-specific WPS outputs files in
/data/wps_outputs
birdhouse/optional-components/test-cowbird-jupyter-access/docker-compose-extra.yml
Outdated
Show resolved
Hide resolved
|
Adding @mishaschwartz to this review as he might be more aware of Cowbird than me. He might suggest other ways to make this work. |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2298/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// Infrastructure deployment failed. Instance has not been destroyed. @matprov |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2299/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : cowbird-jupyter-e2e-test PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/cowbird-jupyter-e2e-test/49/NOTEBOOK TEST RESULTS |
tlvu
approved these changes
Nov 29, 2023
…cowbird-jupyter-e2e-test
ChaamC
pushed a commit
to Ouranosinc/PAVICS-e2e-workflow-tests
that referenced
this pull request
Nov 30, 2023
# Overview Adds a new test to check the user's workspace in a JupyterLab environment that also uses Cowbird. It checks if the different expected folders are set up properly and if the user has the right permissions on the different test files. The test preparation is done with a script run by a new optional-component `test-cowbird-jupyter-access` found on birdhouse. Note that test is split in 2 notebooks : - `notebooks-auth/test_cowbird_jupyter.ipynb` : this notebook is the one executed on the container started by Jenkins. His responsability is to start a JupyterLab instance using the JupyterHub API. This instance will then run the actual tests. - `notebooks-auth/resources/user_test_cowbird_jupyter.ipynb` : this notebook is the one executed on the JupyterLab container and which contains the actual tests. ## Related Issue / Discussion - birdhouse PR [#404](bird-house/birdhouse-deploy#404)
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2327/Result : failure BIRDHOUSE_DEPLOY_BRANCH : cowbird-jupyter-e2e-test DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-46.rdext.crim.ca Infrastructure deployment failed. Instance has not been destroyed. @matprov |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
test-cowbird-jupyter-accessthat executes a script to set up a test user along with differenttest files. This component is used for the related e2e test from the
PAVICS-e2e-workflow-tests repo.
Fixes
Additional Information
PAVICS-e2e-workflow-tests: #131