Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix] Model uploader's jekins trigger parameter fix #402

Merged
merged 4 commits into from
Aug 8, 2024
Merged

[Fix] Model uploader's jekins trigger parameter fix #402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d6ec515
[Feature] Add a workflow parameter that model uploader can specific a…
conggguan Aug 6, 2024
6c59690
Merge branch 'opensearch-project:main' into main
conggguan Aug 7, 2024
d134be6
[Fix] To fix the Jekins trigger's wrong folder parameters bug.
conggguan Aug 7, 2024
208d343
[Comments] Add a comments for model uploader workflow's [Initiate fol…
conggguan Aug 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .github/workflows/model_uploader.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,25 @@ jobs:
echo "This workflow should only be triggered on 'main' branch"
exit 1
- name: Initiate folders
# This scripts init the folders path variables.
# 1. Retrieves the input model_id.
# 2. If upload_prefix is provided, constructs model_prefix using upload_prefix and model_source.
# - model_prefix: "ml-models/{model_source}/{upload_prefix}"
# 3. If upload_prefix is not provided, it constructs model_prefix using model_source and the prefix part of model_id.
# - The prefix part is the substring before the first '/' in model_id.
# Example:
# - Given model_id: "opensearch-project/opensearch-neural-sparse-encoding-v1"
# - model_prefix: "ml-models/{model_source}/opensearch-project"
# 4. Constructs model_folder and model_prefix_folder.
id: init_folders
run: |
model_id=${{ github.event.inputs.model_id }}
echo "model_folder=ml-models/${{github.event.inputs.model_source}}/${model_id}" >> $GITHUB_OUTPUT
if [[ -n "${{ github.event.inputs.upload_prefix }}" ]]; then
model_prefix="ml-models/${{ github.event.inputs.model_source }}/${{ github.event.inputs.upload_prefix }}"
else
model_prefix="ml-models/${{ github.event.inputs.model_source }}/${model_id%%/*}"
fi
echo "model_folder=$model_prefix/${model_id##*/}" >> $GITHUB_OUTPUT
Copy link
Collaborator

@dhrubo-os dhrubo-os Aug 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is ##* about? Can we add comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"##" is used to match the longest {*/}.
For example:

model_id="opensearch-project/opensearch-neural-sparse-encoding-v1/tree/main"
output="${model_id##*/}"
echo $output

will generate a main, because the ##*/ matched the "opensearch-project/opensearch-neural-sparse-encoding-v1/tree/" and deleted it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment in the code, what does the model_folder look like for both cases (if and else)

echo "model_prefix_folder=$model_prefix" >> $GITHUB_OUTPUT
- name: Initiate workflow_info
id: init_workflow_info
Expand Down Expand Up @@ -446,4 +456,4 @@ jobs:
version=${{ github.event.inputs.model_version }}
format=${{ github.event.inputs.tracing_format }}
jenkins_params="{\"BASE_DOWNLOAD_PATH\":\"$base_download_path\", \"VERSION\":\"$version\", \"FORMAT\":\"$format\"}"
sh utils/model_uploader/trigger_ml_models_release.sh $jenkins_trigger_token "$jenkins_params"
sh utils/model_uploader/trigger_ml_models_release.sh $jenkins_trigger_token "$jenkins_params"
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Removed pandas version pin in nox tests by @rawwar ([#368](https://github.com/opensearch-project/opensearch-py-ml/pull/368))
- Switch AL2 to AL2023 agent and DockerHub to ECR images in ml-models.JenkinsFile ([#377](https://github.com/opensearch-project/opensearch-py-ml/pull/377))
- Refactored validators in ML Commons' client([#385](https://github.com/opensearch-project/opensearch-py-ml/pull/385))
- Update model upload history - opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill (v.1.0.0)(TORCH_SCRIPT) by @dhrubo-os ([#400](https://github.com/opensearch-project/opensearch-py-ml/pull/400))

### Fixed
- Fix the wrong input parameter for model_uploader's base_download_path in jekins trigger.([#402](https://github.com/opensearch-project/opensearch-py-ml/pull/402))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] jenkins

- Enable make_model_config_json to add model description to model config file by @thanawan-atc in ([#203](https://github.com/opensearch-project/opensearch-py-ml/pull/203))
- Correct demo_ml_commons_integration.ipynb by @thanawan-atc in ([#208](https://github.com/opensearch-project/opensearch-py-ml/pull/208))
- Handle the case when the model max length is undefined in tokenizer by @thanawan-atc in ([#219](https://github.com/opensearch-project/opensearch-py-ml/pull/219))
Expand Down
3 changes: 1 addition & 2 deletions utils/model_uploader/upload_history/MODEL_UPLOAD_HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,4 @@ The following table shows sentence transformer model upload history.
|2023-08-31 15:22:19|@dhrubo-os|`sentence-transformers/msmarco-distilbert-base-tas-b`|1.0.2|TORCH_SCRIPT|N/A|N/A|6042401385|
|2023-09-13 18:03:32|@dhrubo-os|`sentence-transformers/distiluse-base-multilingual-cased-v1`|1.0.1|TORCH_SCRIPT|N/A|N/A|6178024517|
|2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|ONNX|N/A|N/A|6568285400|
|2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|TORCH_SCRIPT|N/A|N/A|6568285400|
|2024-08-06 12:42:00|@dhrubo-os|`opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill`|1.0.0|TORCH_SCRIPT|N/A|N/A|10271804648|
|2023-10-18 18:06:15|@dhrubo-os|`sentence-transformers/paraphrase-mpnet-base-v2`|1.0.0|TORCH_SCRIPT|N/A|N/A|6568285400|
10 changes: 0 additions & 10 deletions utils/model_uploader/upload_history/supported_models.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,5 @@
"Embedding Dimension": "N/A",
"Pooling Mode": "N/A",
"Workflow Run ID": "6568285400"
},
{
"Model Uploader": "@dhrubo-os",
"Upload Time": "2024-08-06 12:42:00",
"Model ID": "opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill",
"Model Version": "1.0.0",
"Model Format": "TORCH_SCRIPT",
"Embedding Dimension": "N/A",
"Pooling Mode": "N/A",
"Workflow Run ID": "10271804648"
}
]
Loading