Skip to content

Commit dd43e50

Browse files
mlcommons-botmlcommons-bot
committed
Auto-merge updates from auto-update branch
2 parents 7424f30 + f4762a1 commit dd43e50

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+20645
-20645
lines changed

‎open/MLCommons/measurements/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/README.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ pip install -U cmind
1919

2020
cm rm cache -f
2121

22-
cm pull repo mlcommons@mlperf-automations --checkout=a90475d2de72bf0622cebe8d5ca8eb8c9d872fbd
22+
cm pull repo mlcommons@mlperf-automations --checkout=467517e4a572872046058e394a0d83512cfff38b
2323

2424
cm run script \
2525
--tags=app,mlperf,inference,generic,_nvidia,_retinanet,_tensorrt,_cuda,_valid,_r4.1-dev_default,_multistream \
@@ -71,7 +71,7 @@ cm run script \
7171
--env.CM_DOCKER_REUSE_EXISTING_CONTAINER=yes \
7272
--env.CM_DOCKER_DETACHED_MODE=yes \
7373
--env.CM_MLPERF_INFERENCE_RESULTS_DIR_=/home/arjun/gh_action_results/valid_results \
74-
--env.CM_DOCKER_CONTAINER_ID=f77641321894 \
74+
--env.CM_DOCKER_CONTAINER_ID=3216aa4729da \
7575
--env.CM_MLPERF_LOADGEN_COMPLIANCE_TEST=TEST01 \
7676
--add_deps_recursive.compiler.tags=gcc \
7777
--add_deps_recursive.coco2014-original.tags=_full \
@@ -129,7 +129,7 @@ Platform: RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config
129129
Model Precision: int8
130130

131131
### Accuracy Results
132-
`mAP`: `37.327`, Required accuracy for closed division `>= 37.1745`
132+
`mAP`: `37.324`, Required accuracy for closed division `>= 37.1745`
133133

134134
### Performance Results
135-
`Samples per query`: `5623507.0`
135+
`Samples per query`: `5613857.0`

‎open/MLCommons/measurements/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy_console.out

+30-30
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
[2024-12-25 01:40:51,607 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x2
2-
[2024-12-25 01:40:51,689 harness.py:249 INFO] The harness will load 2 plugins: ['build/plugins/NMSOptPlugin/libnmsoptplugin.so', 'build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so']
3-
[2024-12-25 01:40:51,689 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x2_TRT/retinanet/MultiStream
4-
[2024-12-25 01:40:51,689 __init__.py:46 INFO] Running command: ./build/bin/harness_default --plugins="build/plugins/NMSOptPlugin/libnmsoptplugin.so,build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so" --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=64 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=2 --map_path="data_maps/open-images-v6-mlperf/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/5860c00d55d14786/inference/mlperf.conf" --tensor_path="build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/00cdca2a511c4f33bad6658d0557db67.conf" --gpu_engines="./build/engines/RTX4090x2/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model retinanet --response_postprocess openimageeffnms
5-
[2024-12-25 01:40:51,689 __init__.py:53 INFO] Overriding Environment
1+
[2024-12-28 01:34:47,266 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x2
2+
[2024-12-28 01:34:47,342 harness.py:249 INFO] The harness will load 2 plugins: ['build/plugins/NMSOptPlugin/libnmsoptplugin.so', 'build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so']
3+
[2024-12-28 01:34:47,342 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x2_TRT/retinanet/MultiStream
4+
[2024-12-28 01:34:47,343 __init__.py:46 INFO] Running command: ./build/bin/harness_default --plugins="build/plugins/NMSOptPlugin/libnmsoptplugin.so,build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so" --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=64 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=2 --map_path="data_maps/open-images-v6-mlperf/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/5860c00d55d14786/inference/mlperf.conf" --tensor_path="build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/915a97073e504aeabc17038b264e8db8.conf" --gpu_engines="./build/engines/RTX4090x2/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model retinanet --response_postprocess openimageeffnms
5+
[2024-12-28 01:34:47,343 __init__.py:53 INFO] Overriding Environment
66
benchmark : Benchmark.Retinanet
77
buffer_manager_thread_count : 0
88
data_dir : /home/cmuser/CM/repos/local/cache/4db00c74da1e44c8/data
@@ -12,7 +12,7 @@ gpu_copy_streams : 1
1212
gpu_inference_streams : 1
1313
input_dtype : int8
1414
input_format : linear
15-
log_dir : /home/cmuser/CM/repos/local/cache/94a57f78972843c6/repo/closed/NVIDIA/build/logs/2024.12.25-01.40.50
15+
log_dir : /home/cmuser/CM/repos/local/cache/94a57f78972843c6/repo/closed/NVIDIA/build/logs/2024.12.28-01.34.46
1616
map_path : data_maps/open-images-v6-mlperf/val_map.txt
1717
mlperf_conf_path : /home/cmuser/CM/repos/local/cache/5860c00d55d14786/inference/mlperf.conf
1818
multi_stream_expected_latency_ns : 0
@@ -26,7 +26,7 @@ tensor_path : build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet
2626
test_mode : AccuracyOnly
2727
use_deque_limit : True
2828
use_graphs : True
29-
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/00cdca2a511c4f33bad6658d0557db67.conf
29+
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/915a97073e504aeabc17038b264e8db8.conf
3030
system_id : RTX4090x2
3131
config_name : RTX4090x2_retinanet_MultiStream
3232
workload_setting : WorkloadSetting(HarnessType.LWIS, AccuracyTarget.k_99, PowerSetting.MaxP)
@@ -40,27 +40,27 @@ power_limit : None
4040
cpu_freq : None
4141
&&&& RUNNING Default_Harness # ./build/bin/harness_default
4242
[I] mlperf.conf path: /home/cmuser/CM/repos/local/cache/5860c00d55d14786/inference/mlperf.conf
43-
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/00cdca2a511c4f33bad6658d0557db67.conf
43+
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/915a97073e504aeabc17038b264e8db8.conf
4444
Creating QSL.
4545
Finished Creating QSL.
4646
Setting up SUT.
47-
[I] [TRT] Loaded engine size: 73 MiB
48-
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 126, GPU 881 (MiB)
49-
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 128, GPU 891 (MiB)
47+
[I] [TRT] Loaded engine size: 72 MiB
48+
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 125, GPU 881 (MiB)
49+
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 127, GPU 891 (MiB)
5050
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +68, now: CPU 0, GPU 68 (MiB)
5151
[I] Device:0.GPU: [0] ./build/engines/RTX4090x2/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan has been successfully loaded.
52-
[I] [TRT] Loaded engine size: 73 MiB
52+
[I] [TRT] Loaded engine size: 72 MiB
5353
[W] [TRT] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
54-
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 161, GPU 624 (MiB)
55-
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 162, GPU 634 (MiB)
54+
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +6, GPU +10, now: CPU 159, GPU 625 (MiB)
55+
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 161, GPU 635 (MiB)
5656
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +69, now: CPU 0, GPU 137 (MiB)
5757
[I] Device:1.GPU: [0] ./build/engines/RTX4090x2/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan has been successfully loaded.
5858
[E] [TRT] 3: [runtime.cpp::~Runtime::401] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::401, condition: mEngineCounter.use_count() == 1 Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.)
59-
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 89, GPU 893 (MiB)
60-
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 90, GPU 901 (MiB)
59+
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 88, GPU 893 (MiB)
60+
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 88, GPU 901 (MiB)
6161
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +1, GPU +1528, now: CPU 1, GPU 1665 (MiB)
62-
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 90, GPU 636 (MiB)
63-
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 90, GPU 644 (MiB)
62+
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 89, GPU 637 (MiB)
63+
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 89, GPU 645 (MiB)
6464
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +1527, now: CPU 1, GPU 3192 (MiB)
6565
[I] Start creating CUDA graphs
6666
[I] Capture 2 CUDA graphs
@@ -69,7 +69,7 @@ Setting up SUT.
6969
[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: false
7070
Finished setting up SUT.
7171
Starting warmup. Running for a minimum of 5 seconds.
72-
Finished warmup. Ran for 5.14343s.
72+
Finished warmup. Ran for 5.14324s.
7373
Starting running actual test.
7474

7575
No warnings encountered during test.
@@ -87,34 +87,34 @@ Device Device:1.GPU processed:
8787
PerSampleCudaMemcpy Calls: 0
8888
BatchedCudaMemcpy Calls: 6196
8989
&&&& PASSED Default_Harness # ./build/bin/harness_default
90-
[2024-12-25 01:41:30,299 run_harness.py:166 INFO] Result: Accuracy run detected.
91-
[2024-12-25 01:41:30,299 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/94a57f78972843c6/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-openimages.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy/mlperf_log_accuracy.json --openimages-dir /home/cmuser/CM/repos/local/cache/4db00c74da1e44c8/preprocessed_data/open-images-v6-mlperf --output-file build/retinanet-results.json
90+
[2024-12-28 01:35:26,660 run_harness.py:166 INFO] Result: Accuracy run detected.
91+
[2024-12-28 01:35:26,660 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/94a57f78972843c6/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-openimages.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x2-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy/mlperf_log_accuracy.json --openimages-dir /home/cmuser/CM/repos/local/cache/4db00c74da1e44c8/preprocessed_data/open-images-v6-mlperf --output-file build/retinanet-results.json
9292
loading annotations into memory...
93-
Done (t=0.53s)
93+
Done (t=0.44s)
9494
creating index...
9595
index created!
9696
Loading and preparing results...
97-
DONE (t=17.92s)
97+
DONE (t=17.86s)
9898
creating index...
9999
index created!
100100
Running per image evaluation...
101101
Evaluate annotation type *bbox*
102-
DONE (t=133.92s).
102+
DONE (t=133.68s).
103103
Accumulating evaluation results...
104-
DONE (t=32.42s).
104+
DONE (t=33.01s).
105105
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.373
106106
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.522
107-
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.403
107+
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.404
108108
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022
109-
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.125
110-
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.413
109+
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.124
110+
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.412
111111
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.419
112112
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.599
113113
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.628
114-
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.082
114+
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.081
115115
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.344
116116
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.677
117-
mAP=37.327%
117+
mAP=37.324%
118118

119119
======================== Result summaries: ========================
120120

0 commit comments

Comments
 (0)