Skip to content

Commit

Permalink
Results from self hosted Github actions
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Feb 9, 2025
1 parent fcee210 commit a4d9682
Show file tree
Hide file tree
Showing 14 changed files with 465 additions and 465 deletions.
2 changes: 1 addition & 1 deletion ...ns/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
|---------------|------------|------------------------------------|--------------|-------------------|
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.404 | - |
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.409 | - |
4 changes: 2 additions & 2 deletions ...ion-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pip install -U mlcflow

mlc rm cache -f

mlc pull repo gateoverflow@mlperf-automations --checkout=0f5a86f9514172a0976090cdc91963b2b7eb4282
mlc pull repo gateoverflow@mlperf-automations --checkout=970e36703753fb9273ed7e593a7fc14ab84dd09c


```
Expand All @@ -43,4 +43,4 @@ Model Precision: fp32
`TOKENS_PER_SAMPLE`: `610.0`, Required accuracy for closed division `>= 265.005` and `<= 323.895`

### Performance Results
`Samples per second`: `0.40413`
`Samples per second`: `0.40942`
10 changes: 5 additions & 5 deletions ...on-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/accuracy_console.out
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
INFO:datasets:PyTorch version 2.6.0+cpu available.
Loading dataset...
Finished loading dataset.
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:01<00:23, 1.68s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:23, 1.82s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:22, 1.86s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.85s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.85s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.84s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.83s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:13, 1.97s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:18<00:15, 2.56s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:22<00:14, 2.85s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:25<00:11, 2.77s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:27<00:07, 2.56s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:29<00:04, 2.37s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:31<00:02, 2.33s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 1.73s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.11s/it]
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:01<00:23, 1.67s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:23, 1.84s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:22, 1.89s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.87s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.85s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.83s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.82s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:13, 1.95s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:18<00:14, 2.42s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:22<00:14, 2.87s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:24<00:11, 2.76s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:26<00:07, 2.53s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:28<00:04, 2.35s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:31<00:02, 2.39s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 1.78s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.11s/it]
INFO:Llama-70B-MAIN:Starting Benchmark run
/home/mlcuser/venv/mlc/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
Expand All @@ -22,7 +22,7 @@ IssueQuery started with 1 samples
IssueQuery done
Saving outputs to run_outputs/q0.pkl
Samples run: 1
BatchMaker time: 0.009028196334838867
Inference time: 773.6842555999756
Postprocess time: 0.015709400177001953
==== Total time: 773.7089931964874
BatchMaker time: 0.0083465576171875
Inference time: 743.4978594779968
Postprocess time: 0.024612903594970703
==== Total time: 743.530818939209
Expand Down
52 changes: 26 additions & 26 deletions .../gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/mlc-deps.mmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@ graph TD
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> pull,git,repo
get-mlperf-inference-utils,e341e5f86d8342e5 --> get,mlperf,inference,src
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> get,mlperf,inference,utils
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,python
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> detect,cpu
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,sys-utils-cm
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,python
get-generic-python-lib,94b62a682bc44791_(_torch_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_torch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_torch
get-generic-python-lib,94b62a682bc44791_(_torchvision_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_torchvision
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_torchvision
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,sys-utils-cm
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,python3
get-generic-python-lib,94b62a682bc44791_(_package.pyarrow_) --> get,python3
Expand All @@ -26,45 +26,45 @@ graph TD
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,generic-python-lib,_package.transformers
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> download-and-extract,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/open_orca
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
detect-cpu,586c8a43320142f7 --> detect,os
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,python
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,loadgen
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> generate,user-conf,mlperf,inference
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,loadgen
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,mlcommons,inference,src
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,mlcommons,inference,src
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.psutil
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.psutil
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.transformers
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.transformers
get-generic-python-lib,94b62a682bc44791_(_package.datasets_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.datasets
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.datasets
get-generic-python-lib,94b62a682bc44791_(_package.sentencepiece_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
get-generic-python-lib,94b62a682bc44791_(_package.protobuf_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.protobuf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.protobuf
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.accelerate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.accelerate
get-generic-python-lib,94b62a682bc44791_(_package.absl-py_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.absl-py
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.absl-py
get-generic-python-lib,94b62a682bc44791_(_package.evaluate_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.evaluate
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.evaluate
get-generic-python-lib,94b62a682bc44791_(_package.nltk_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.nltk
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.nltk
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.numpy
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.numpy
get-generic-python-lib,94b62a682bc44791_(_package.rouge-score_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
get-generic-python-lib,94b62a682bc44791_(_package.more-itertools_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
get-generic-python-lib,94b62a682bc44791_(_package.compressed_tensors_) --> get,python3
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
detect-cpu,586c8a43320142f7 --> detect,os
benchmark-program,19f369ef47084895 --> detect,cpu
benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_offline,_pytorch,_cpu,_llama2-70b-99,_bfloat16_) --> benchmark-mlperf
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> benchmark-mlperf
Binary file modified ...-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/mlc-deps.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit a4d9682

Please sign in to comment.