nbgrader validate exit codes when run by a student

[meneguzzi]

Operating system

macOS Monterrey

nbgrader --version

Python version 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:13:44) [Clang 16.0.6 ]
nbgrader version 0.8.5

jupyterhub --version (if used with JupyterHub)

jupyter notebook --version

6.5.4

Expected behavior

nbgrader validate <student_notebook>;print $? should return (print here) a non-zero exit code if it fails any test

Actual behavior

The command returns 0 even when validation fails

Steps to reproduce the behavior

1. Create any NBGrader notebook with tests.
2. Release the notebook
3. run nbgrader validate <student_notebook>;print $?

For context, I'm trying to use nbgrader together with Github Classroom to run my course without having to configure a machine to receive nbgrader submissions, and allow students to get immediate feedback to their work when they commit.

[brichet]

Thanks @meneguzzi for opening this.

nbgrader validate <student_notebook>;print $? should return (print here) a non-zero exit code if it fails any test

Not sure about that, it will return error if an exception is raised, otherwise it returns the status (success or fails) of the validation, and the errors in cells.

Maybe you could parse the output, to look for VALIDATION FAILED or Success! Your notebook passes all the tests.

[meneguzzi]

Yes, my current hacky way of doing this is to use this command in the GitHub Classroom autograder:

unbuffer nbgrader validate AI-Jupyternotebook.ipynb | tee error.log; ! grep "VALIDATION FAILED" error.log

[meneguzzi]

For reference, this is important if you want to actually read where the errors were in Github action log. Having said that, I think ideally nbgrader validate should use the return codes in a meaningful way. This does not look like a very hard feature to implement.