test: TC for Metric P0 nv_load_time per model #7697

indrajit96 · 2024-10-14T08:10:43Z

What does the PR do?

Test Case of model load time metrics

Checklist

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

Related PRs:

Core : triton-inference-server/core#397

Where should the reviewer start?

qa/L0_metrics/general_metrics_test.py

Test plan:

Added tests for

Normal Mode Model Load
Explicit Model Load
Explicit Model Unload

CI Pipeline ID: https://gitlab-master.nvidia.com/dl/dgx/tritonserver/-/pipelines/19322776

Background

Improve metrics in Triton

Added support for OTel context propagation --------- Co-authored-by: Markus Hennerbichler <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

This validates the change made to ../core wrt how model configuration mtime is handled.

* Run all cases wihh shm probe * Warmup test and then run multiple iterations * Log free shared memory on enter/exit of probe * Add shm probe to all tests * Add debug_str to shm_util * Refactor ensemble_io test, modify probe to check for growth rather than inequality * Improve stability of bls_tensor_lifecycle gpu memory tests * Add more visibility into failing model/case in python_unittest helper * [FIXME] Skip probe on certain subtests for now * [FIXME] Remove shm probe from test_restart on unhealthy stub * Start clean server run for each bls test case * Don't exit early on failure so logs can be properly collected * Restore bls test logic * Fix shm size compare * Print region name that leaked * Remove special handling on unittest * Remove debug str * Add enter and exit delay to shm leak probe --------- Co-authored-by: Ryan McCormick <[email protected]>

…ackend_python (#6823)

* Update trace_summery script * Remove GRPC_WAITREAD and Overhead

* Add gsutil cp retry helper function * Add max retry to GCS upload * Use simple sequential upload

* Handle empty output * Add test case for 0 dimension output * Fix up number of tests

* tensorrt-llm benchmarking test

* Update README and versions for 2.42.0 / 24.01 (#6789) * Update versions * Update README and versions for 2.42.0 / 24.01 * Fix documentaation genarion (#6801) * Ser version of sphix to 5.0 * Set verions 5.0.0 * Update README.md and versions post 24.01

…und (#6834) * Update miniconda version * Install pytest for different py version * Install pytest

* Add test for shutdown while loading * Fix intermittent failure on test_model_config_overwrite

Adding OpenTelemetry Batch Span Processor --------- Co-authored-by: Theo Clark <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

* Support Double-Type Infer/Response Parameters

* Base Python Backend Support for Windows

* Add unit test reports to L0_dlpack_multi_gpu * Add unit test reports to L0_warmup

* Add response statistics * Add L0_response_statistics * Enable http vs grpc statistics comparison * Add docs for response statistics protocol * Add more comments for response statistics test * Remove model name from config * Improve docs wordings * [Continue] Improve docs wordings * [Continue] Add more comments for response statistics test * [Continue 2] Improve docs wordings * Fix typo * Remove mentioning decoupled from docs * [Continue 3] Improve docs wordings * [Continue 4] Improve docs wordings Co-authored-by: Ryan McCormick <[email protected]> --------- Co-authored-by: Ryan McCormick <[email protected]>

* Switch to Python model for busyop test * Clean up * Address comment * Remove unused import

* Add cancellation into response statistics * Add test for response statistics cancel * Remove debugging print * Use is None comparison * Fix docs * Use default args None * Refactor RegisterModelStatistics()

)

…er each model reload (#7735)

Co-authored-by: Ryan McCormick <[email protected]>

Co-authored-by: Misha Chornyi <[email protected]>

yinggeh · 2024-10-31T16:53:57Z

docs/user_guide/metrics.md

+#### Load Time Per-Model
+The *Model Load Duration* reflects the time to load a model from storage into GPU/CPU in seconds.
+```
+# HELP nv_model_load_duration_secs Model load time in seconds


Do we need a sample output for a gauge metric?

qa/L0_metrics/general_metrics_test.py

qa/L0_metrics/test.sh

Co-authored-by: Ryan McCormick <[email protected]>

…ublished containers (#7759)

yinggeh · 2024-11-05T20:49:31Z

qa/L0_metrics/test.sh

 # Test 3 for explicit mode UNLOAD
 python3 -m pytest --junitxml="general_metrics_test.test_metrics_load_time_explicit_unload.report.xml" $CLIENT_PY::TestGeneralMetrics::test_metrics_load_time_explicit_unload >> $CLIENT_LOG 2>&1
 kill_server
+set -e

 # Test 4 for explicit mode LOAD and UNLOAD with multiple versions
 set +e
 CLIENT_PY="./general_metrics_test.py"


yinggeh · 2024-11-05T20:52:39Z

qa/L0_metrics/general_metrics_test.py

        print(f"Model '{model_name}' loaded successfully.")
-    else:
+    except AssertionError:


Do we want the test to pass if failed to load the model? If not, you should remove try...except...

Yes that's expected behaviour.
Models should load and unload. Else test should fail as subsequent metrics will be incorrect

If load or unload failure will result test to fail anyway, why not let it fail at the HTTP response code check instead of metrics check? This way people can easiler identify the root cause of job failure.

…am metric buckets (#7752)

…7770)

rmccorm4 · 2024-11-07T22:21:10Z

qa/L0_metrics/general_metrics_test.py

How come the core PR was merged way before this one finished? We currently have no ongoing tests for the merged feature on our nightly pipelines in core, right?

It was approved in parallel. A couple of days appart.
I was unable to get a CI passing due to other build issues.
And then @yinggeh added more comments after it was approved. Hence the delay.
Yes I will get this in ASAP after the trtllm Code freeze

oandreeva-nv and others added 30 commits January 17, 2024 10:32

Support for Context Propagation for OTel trace mode (#6785)

4a719e4

Added support for OTel context propagation --------- Co-authored-by: Markus Hennerbichler <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

Use current time when overwriting model configuration. (#6727)

87165b2

This validates the change made to ../core wrt how model configuration mtime is handled.

Added docs for otel context propagation (#6804)

7b06a37

Fix typos in trace.md (#6808)

b6e017e

Fix test_model_config_overwite in L0_lifecycle (#6818)

3e79b2a

Remove boost::filesystem (#6810)

3bff367

Generate unittest xml reports from L0_python_api (#6822)

bc71da0

Add unit test reports to L0_json, L0_metrics, L0_response_cache, L0_b…

6192c6e

…ackend_python (#6823)

Update trace summary script (#6758)

a514a05

* Update trace_summery script * Remove GRPC_WAITREAD and Overhead

Add gsutil upload retry helper function (#6817)

28f497c

* Add gsutil cp retry helper function * Add max retry to GCS upload * Use simple sequential upload

Add test for shutdown while unloading in background (#6835)

ddfdb2a

Handle 0 dimension output for generate endpoint (#6833)

56e4232

* Handle empty output * Add test case for 0 dimension output * Fix up number of tests

tensorrt-llm benchmarking test (#6771)

d98a59c

* tensorrt-llm benchmarking test

Use libmamba solver for L0_backend_python env test. Fix pytest not fo…

d0e2653

…und (#6834) * Update miniconda version * Install pytest for different py version * Install pytest

Add test for shutdown while loading model (#6837)

f92732d

* Add test for shutdown while loading * Fix intermittent failure on test_model_config_overwrite

Adding OpenTelemetry Batch Span Processor (#6842)

776e641

Adding OpenTelemetry Batch Span Processor --------- Co-authored-by: Theo Clark <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

Support Double-Type Inference Request/Response Parameters (#6755)

b0a495a

* Support Double-Type Infer/Response Parameters

Updating vllm version to 0.3.0 (#6858)

508929a

Python Backend Windows Support (#6830)

738c98f

* Base Python Backend Support for Windows

Add support for Oracle Cloud in deploy (#6850)

3d79568

Add link to TRTLLM metrics docs (#6874)

1df73dc

Add unit test reports to L0_dlpack_multi_gpu and L0_warmup (#6873)

4294cc6

* Add unit test reports to L0_dlpack_multi_gpu * Add unit test reports to L0_warmup

Set OV version to 2023.3.0 (#6880)

f078bfb

Fixing StringTo uint32_t used only by tracing (#6883)

80fc56c

Update 'main' to track development of 2.44.0 / 24.03 (#6892)

8a2a229

Fix busyop test for L0_memory_growth (#6900)

21a7fc5

* Switch to Python model for busyop test * Clean up * Address comment * Remove unused import

Add cancellation into response statistics (#6904)

60872b9

* Add cancellation into response statistics * Add test for response statistics cancel * Remove debugging print * Use is None comparison * Fix docs * Use default args None * Refactor RegisterModelStatistics()

rmccorm4 and others added 6 commits October 23, 2024 13:23

docs: Add support matrix for model parallelism in OpenAI Frontend (#7715

2f8de73

)

test: Add L0_additional_dependency_dirs (#7707)

dcfc6a0

test: Add small delay to L0_lifecycle test_load_new_model_version aft…

128f19a

…er each model reload (#7735)

Removing caching on windows. (#7717)

604b2aa

feat: Metrics Support in tritonfrontend (#7703)

4453fa3

Co-authored-by: Ryan McCormick <[email protected]>

build: RHEL8 Python Backend (#7744)

284e71d

Co-authored-by: Misha Chornyi <[email protected]>

yinggeh reviewed Oct 31, 2024

View reviewed changes

GuanLuo and others added 9 commits October 31, 2024 16:38

chore: ensure proper clean up in shared memory related tests (#7729)

3bfacf8

refactor: Include job id and nightly tag to results uploaded (#7751)

c7589f1

Update test script for TRT compatibility test to check for

0b724f2

Build: Update main branch post 24.10 release (#7754)

3b4fabd

ci: Adding tests for numpy>=2 (#7756)

67c59c8

Co-authored-by: Ryan McCormick <[email protected]>

Reapply "Change compute capability min value (#7708)" (#7757)

06b358a

build: Install tritonfrontend and tritonserver wheels by default in p…

8941e15

…ublished containers (#7759)

Fix model generation (#7764)

1f7a516

Review Comments Fixed

e83fcca

indrajit96 dismissed kthui’s stale review via f07f5ef November 4, 2024 23:47

indrajit96 added 2 commits November 4, 2024 15:49

Pre-Commit Fix

4d50a34

Merge branch 'main' into ibhosale_metrics_google

ffaa9b5

yinggeh reviewed Nov 5, 2024

View reviewed changes

yinggeh and others added 7 commits November 5, 2024 20:28

test: Test per-model metric customization and document custom histogr…

6191c67

…am metric buckets (#7752)

Extra assignment removed

3692bbc

Merge branch 'main' into ibhosale_metrics_google

65d79b3

fix: Fixing pip installation as a system package (#7768)

4725600

fix: Adding copyright support for .pyi files (#7769)

5f8f07b

fix: Skip copyrights check for "expected" files in L0_model_config (#…

0269a3c

…7770)

Update 'main' to track development of 2.53.0 / 24.12 (#7771)

51b304f

rmccorm4 reviewed Nov 7, 2024

View reviewed changes

Merge branch 'main' into ibhosale_metrics_google

4ab4c00

pvijayakrish force-pushed the ibhosale_metrics_google branch from 4764717 to 4ab4c00 Compare January 15, 2025 17:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: TC for Metric P0 nv_load_time per model #7697

test: TC for Metric P0 nv_load_time per model #7697

indrajit96 commented Oct 14, 2024 •

edited

Loading

yinggeh Oct 31, 2024

yinggeh Nov 5, 2024

yinggeh Nov 5, 2024

indrajit96 Nov 6, 2024

yinggeh Nov 7, 2024

rmccorm4 Nov 7, 2024

indrajit96 Nov 7, 2024

test: TC for Metric P0 nv_load_time per model #7697

Are you sure you want to change the base?

test: TC for Metric P0 nv_load_time per model #7697

Conversation

indrajit96 commented Oct 14, 2024 • edited Loading

What does the PR do?

Checklist

Commit Type:

Related PRs:

Where should the reviewer start?

Test plan:

Background

yinggeh Oct 31, 2024

Choose a reason for hiding this comment

yinggeh Nov 5, 2024

Choose a reason for hiding this comment

yinggeh Nov 5, 2024

Choose a reason for hiding this comment

indrajit96 Nov 6, 2024

Choose a reason for hiding this comment

yinggeh Nov 7, 2024

Choose a reason for hiding this comment

rmccorm4 Nov 7, 2024

Choose a reason for hiding this comment

indrajit96 Nov 7, 2024

Choose a reason for hiding this comment

indrajit96 commented Oct 14, 2024 •

edited

Loading