[Draft] Automating Model tracing and uploading #193

thanawan-atc · 2023-07-17T19:05:45Z

Description

I implement a GitHub Action workflow that will automate model tracing and uploading process, with an initial focus on pre-trained sentence transformer models.

Issues Resolved

OpenSearch Team receives frequent requests from customers to incorporate additional models into our model hub. Moreover, the current manual process of tracing and uploading these models is laborious and time-consuming. As a result, there is a compelling need to automate this process to streamline model availability effectively.

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

codecov · 2023-07-17T22:28:18Z

Codecov Report

Merging #193 (d2afc36) into main (7622af5) will not change coverage.
The diff coverage is n/a.

❗ Current head d2afc36 differs from pull request most recent head 4d37394. Consider uploading reports for the commit 4d37394 to get more accurate results

@@           Coverage Diff           @@
##             main     #193   +/-   ##
=======================================
  Coverage   91.06%   91.06%           
=======================================
  Files          37       37           
  Lines        4052     4052           
=======================================
  Hits         3690     3690           
  Misses        362      362

…ert-base-tas-b) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

dhrubo-os · 2023-07-17T22:33:52Z

.github/CODEOWNERS

Is this necessary? I might need to talk with release team to give you write access in the repo.

No. I just made this changes for testing on my own repo. I will revert it back before the PR is merged.

dhrubo-os · 2023-07-17T22:34:43Z

.ci/run-repository.sh

Can we add some comments for all these if else conditions

dhrubo-os · 2023-07-17T22:36:07Z

.ci/run-repository.sh

@@ -33,7 +33,7 @@ docker build \
 echo -e "\033[1m>>>>> Run [opensearch-project/opensearch-py-ml container] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>\033[0m"


-if [[ "$IS_DOC" == "false" ]]; then
+if [[ "$TASK_TYPE" == "test" ]]; then


Let's add a comment.

dhrubo-os · 2023-07-17T23:14:11Z

.github/workflows/model_uploader.yml

+        required: true
+        type: choice
+        options:
+        - "BOTH"


How this will work? Will this create two separate workflow? How will we notify release team? One notification or two?

dhrubo-os · 2023-07-17T23:17:17Z

.github/workflows/model_uploader.yml

+      id-token: write
+      contents: read
+    strategy:
+      fail-fast: false


what is this for? May be add a comment why do we need this?

Yeah, after thinking about it, we can remove this. I initially followed the workflow for integration test.

dhrubo-os · 2023-07-18T18:10:36Z

.ci/run-repository.sh

+    python -m pip install pandas~=${PANDAS_VERSION}; 
+    python utils/model_uploader/model_autotracing.py ${MODEL_ID} ${MODEL_VERSION} ${TRACING_FORMAT} -ed ${EMBEDDING_DIMENSION} -pm ${POOLING_MODE}"


What do you think about adding as a nox session?

dhrubo-os · 2023-07-18T18:11:44Z

.github/workflows/build_deploy_doc.yml

@@ -20,7 +20,7 @@ jobs:
      - name: Checkout Repository
        uses: actions/checkout@v2
      - name: Integ ${{ matrix.cluster }} secured=${{ matrix.secured }} version=${{matrix.entry.opensearch_version}}
-        run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }} ${{ matrix.entry.opensearch_version }} true"
+        run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }} ${{ matrix.entry.opensearch_version }} doc"


Just wanted to make sure you tried this command in your end and it successfully generated the doc files?

Yes, it works as expected.

dhrubo-os · 2023-07-18T18:11:56Z

.github/workflows/integration.yml

@@ -18,7 +18,7 @@ jobs:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Integ ${{ matrix.cluster }} secured=${{ matrix.secured }} version=${{matrix.entry.opensearch_version}}
-        run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }} ${{ matrix.entry.opensearch_version }}"
+        run: "./.ci/run-tests ${{ matrix.cluster }} ${{ matrix.secured }} ${{ matrix.entry.opensearch_version }} test"


same as above

dhrubo-os · 2023-07-18T18:15:28Z

.github/workflows/model_uploader.yml

+        id: dryrun_model_uploading
+        run: |
+          aws s3 sync ./upload/ s3://opensearch-exp/ml-models/huggingface/sentence-transformers/ --dryrun
+          dryrun_output=$(aws s3 sync ./upload/ s3://opensearch-exp/ml-models/huggingface/sentence-transformers/ --dryrun)


We need to talk about the file path in our meeting.

dhrubo-os · 2023-07-18T18:51:10Z

utils/model_uploader/model_autotracing.py

+        assert False, f"Raised Exception in {model_format} model deployment: {e}"
+
+    # 3.) Check model status
+    try:


We don't need to check model task status also. IF model is deployed then we can start generating embedding.

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

…ansformers/msmarco-distilbert-base-tas-b) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

thanawan-atc added 17 commits July 13, 2023 13:46

Modify existing code to support Model Tracing

967f48e

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Change CODEOWNER temporarily for testing

35372ca

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Add model_uploader.yml with issue body formatted

ef97a5d

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update sh & yml files

d02b7d2

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Add Python Execution Files

b64b051

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Fix bug

65a57e4

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Fix bug

81f8652

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Fix bug

e8e8efa

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_uploader.yml

296c380

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_autotracing.py

d58d99f

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_uploader.yml

a909e09

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update run-repository.sh

3bd44fd

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_uploader.yml

982069c

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_uploader.yml

50c7a33

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Update model_uploader.yml

fb12aea

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Correct linting

e976553

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Revert CODEOWNERS

ed03ce1

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

thanawan-atc requested review from dhrubo-os, greaa-aws, ylwu-amzn, b4sjoo, jngz-es and rbhavna as code owners July 17, 2023 19:05

thanawan-atc added 4 commits July 17, 2023 15:07

Add upload history

df545b9

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Change CODEOWNERS temporarily for testing

f8038cc

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Revert CODEOWNERS

11052ce

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Change CODEOWNERS temporarily for testing

3d9749b

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

thanawan-atc and others added 2 commits July 17, 2023 22:35

Update MODEL_UPLOAD_HISTORY.md (sentence-transformers/msmarco-distilb…

44a611b

…ert-base-tas-b) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Rephrase wording

48974f8

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

dhrubo-os reviewed Jul 18, 2023

View reviewed changes

thanawan-atc and others added 11 commits July 18, 2023 19:27

Add comments to run-repository.sh

6c4e2cc

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Enable fail-fast

cb27d4b

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Remove model_uploader/__init__.py

63d7bd0

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Debug model_autotracing.py

516a37b

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Debug model_autotracing.py

844cb71

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Debug model_autotracing.py

93d0b3f

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Debug model_autotracing.py

0c1bc68

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Make register_and_deploy_sentence_transformer_model more concise

af560d6

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Debug register_and_deploy_sentence_transformer_model

bbd0214

Signed-off-by: Thanawan Atchariyachanvanit <[email protected]>

Merge remote-tracking branch 'upstream/main' into workflow-dev

d2afc36

GitHub Actions Workflow - Update MODEL_UPLOAD_HISTORY.md (sentence-tr…

4d37394

…ansformers/msmarco-distilbert-base-tas-b) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

thanawan-atc closed this Jul 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Automating Model tracing and uploading #193

[Draft] Automating Model tracing and uploading #193

thanawan-atc commented Jul 17, 2023 •

edited

Loading

codecov bot commented Jul 17, 2023 •

edited

Loading

dhrubo-os Jul 17, 2023

thanawan-atc Jul 18, 2023

dhrubo-os Jul 17, 2023

dhrubo-os Jul 17, 2023

dhrubo-os Jul 17, 2023

dhrubo-os Jul 17, 2023

thanawan-atc Jul 18, 2023

dhrubo-os Jul 18, 2023

dhrubo-os Jul 18, 2023

thanawan-atc Jul 19, 2023

dhrubo-os Jul 18, 2023

thanawan-atc Jul 19, 2023

dhrubo-os Jul 18, 2023

dhrubo-os Jul 18, 2023

		python -m pip install pandas~=${PANDAS_VERSION};
		python utils/model_uploader/model_autotracing.py ${MODEL_ID} ${MODEL_VERSION} ${TRACING_FORMAT} -ed ${EMBEDDING_DIMENSION} -pm ${POOLING_MODE}"

[Draft] Automating Model tracing and uploading #193

[Draft] Automating Model tracing and uploading #193

Conversation

thanawan-atc commented Jul 17, 2023 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Jul 17, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thanawan-atc commented Jul 17, 2023 •

edited

Loading

codecov bot commented Jul 17, 2023 •

edited

Loading