feat(deps): add kubeflow-training to workbench images #826
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
andyatmiami
force-pushed
the
chore/add-kfto-python-sdk
branch
3 times, most recently
from
e09fa47 to
429e683
Compare
This commit adds `kubeflow-training[huggingface]` to the following workbench images:
- `./jupyter/datascience/ubi9-python-3.11`
- `./jupyter/pytorch/ubi9-python-3.11`
- `./jupyter/rocm/pytorch/ubi9-python-3.11`
- `./jupyter/trustyai/ubi9-python-3.11`
- `./codeserver/ubi9-python-3.11`
This outcome comes with a slew of caveats and disclaimers:
- Due to a dependency conflict, `codeflare-sdk==0.24.3` was **also** added to the following workbench images.
- `./jupyter/datascience/ubi9-python-3.11`
- `./jupyter/pytorch/ubi9-python-3.11`
- `./jupyter/rocm/pytorch/ubi9-python-3.11`
- `./jupyter/trustyai/ubi9-python-3.11`
- ⚠️ In what may be a "controversial" decision, `codeflare-sdk` was **NOT** updated on other workbench images. Since `0.24.3` was a "one-off" release to unblock the `kubeflow-training` inclusion - the thought process here is that normal "sync" procedures on the next official release will standardize the `codeflare-sdk` dependency across all workbench images. This allows us to restrict the testing effort of this commit.
- `jupyter/minmal/ubi9-python-3.11` was deliberately excluded from receiving `kubeflow-training` per discussions with team.
- Due to dependency conflicts discovered `tensorflow`-based workbench images,`kubeflow-training` has not been added to those workbench images at this time. This decision was agreed to by affect stakeholders. Core blocking issue can be seen here:
- onnx/tensorflow-onnx#2328
- Due to a dependency conflict, `transformers = "==4.38.0"` was **also** added to the`./jupyter/trustyai/ubi9-python-3.11` workbench image after discussion with the developer that last worked on the `trustyai` image. While it certainly must be tested, there was no strict requirement that necessitated pinning the `transformers` dependency to `4.36.2` - and the `huggingface` `extras` now introduces a `4.38.0` constraint for `transformers`.
Related-to: https://issues.redhat.com/browse/RHOAIENG-12822
andyatmiami
force-pushed
the
chore/add-kfto-python-sdk
branch
from
429e683 to
b2d3631
Compare
| ], | ||
| "markers": "python_version >= '3.8'", | ||
| "version": "==2.1.0" | ||
| "version": "==1.2.0" |
There was a problem hiding this comment.
Not immediately clear to me why this dependency got downgraded 🤔 I'm doing the pipenv install in a "controlled environment" (read: podman container running base/ubi9-python-3.11 )
Was initially updated here:
There was a problem hiding this comment.
|
/test all |
|
/test notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror |
|
/test notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror |
|
/test notebook-cuda-jupyter-ds-ubi9-python-3-11-pr-image-mirror |
|
/test notebooks-ubi9-e2e-tests |
|
@andyatmiami: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/test codeserver-notebook-e2e-tests |
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Description
This commit adds
kubeflow-training[huggingface]to the following workbench images:./jupyter/datascience/ubi9-python-3.11./jupyter/pytorch/ubi9-python-3.11./jupyter/rocm/pytorch/ubi9-python-3.11./jupyter/trustyai/ubi9-python-3.11./codeserver/ubi9-python-3.11This outcome comes with a slew of caveats and disclaimers:
codeflare-sdk==0.24.3was also added to the following workbench images../jupyter/datascience/ubi9-python-3.11./jupyter/pytorch/ubi9-python-3.11./jupyter/rocm/pytorch/ubi9-python-3.11./jupyter/trustyai/ubi9-python-3.11codeflare-sdkwas NOT updated on other workbench images. Since0.24.3was a "one-off" release to unblock thekubeflow-traininginclusion - the thought process here is that normal "sync" procedures on the next official release will standardize thecodeflare-sdkdependency across all workbench images. This allows us to restrict the testing effort of this commit.jupyter/minmal/ubi9-python-3.11was deliberately excluded from receivingkubeflow-trainingper discussions with team.tensorflow-based workbench images,kubeflow-traininghas not been added to those workbench images at this time. This decision was agreed to by affect stakeholders. Core blocking issue can be seen here:transformers = "==4.38.0"was also added to the./jupyter/trustyai/ubi9-python-3.11workbench image after discussion with the developer that last worked on thetrustyaiimage. While it certainly must be tested, there was no strict requirement that necessitated pinning thetransformersdependency to4.36.2- and thehuggingfaceextrasnow introduces a4.38.0constraint fortransformers.Related-to: https://issues.redhat.com/browse/RHOAIENG-12822
How Has This Been Tested?
Merge criteria: