Merge branch 'mlcommons:mlperf-inference-results-scc24' into mlperf-i…

…nference-results-scc24
GATEOverflow · Oct 21, 2024 · 5b59cff · 5b59cff
2 parents 808c134 + c77698c
commit 5b59cff
Show file tree

Hide file tree

Showing 175 changed files with 10,869 additions and 142 deletions.
diff --git a/README.md b/README.md
diff --git a/open/Dreambrook_Team/code/bert-99/README.md b/open/Dreambrook_Team/code/bert-99/README.md
@@ -0,0 +1 @@
+TBD
diff --git a/...Dreambrook_Team/measurements/666dfb9811e4-reference-gpu-pytorch-cu124/README.md b/...Dreambrook_Team/measurements/666dfb9811e4-reference-gpu-pytorch-cu124/README.md
@@ -0,0 +1,3 @@
+| Model   | Scenario   |   Accuracy |   Throughput | Latency (in ms)   |
+|---------|------------|------------|--------------|-------------------|
+| bert-99 | offline    |     90.876 |       85.447 | -                 |
diff --git a/...reference-gpu-pytorch-cu124/bert-99/offline/666dfb9811e4-reference-gpu-pytorch-cu124.json b/...reference-gpu-pytorch-cu124/bert-99/offline/666dfb9811e4-reference-gpu-pytorch-cu124.json
@@ -0,0 +1,7 @@
+{
+  "starting_weights_filename": "https://armi.in/files/fp32/model.pytorch",
+  "retraining": "no",
+  "input_data_types": "fp32",
+  "weight_data_types": "fp32",
+  "weight_transformations": "none"
+}
diff --git a/...measurements/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline/README.md b/...measurements/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline/README.md
@@ -0,0 +1,54 @@
+This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
+
+*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
+
+## Host platform
+
+* OS version: Linux-6.1.112-1.el9.elrepo.x86_64-x86_64-with-glibc2.35
+* CPU version: x86_64
+* Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]
+* MLCommons CM version: 3.2.2
+
+## CM Run Command
+
+See [CM installation guide](https://docs.mlcommons.org/inference/install/).
+
+```bash
+pip install -U cmind
+
+cm rm cache -f
+
+cm pull repo mlcommons@cm4mlops --checkout=bcde11676cc734f2398028a039fc1c94ae9fa81a
+
+cm run script \
+	--tags=run-mlperf,inference,_r4.1-dev \
+	--model=bert-99 \
+	--implementation=reference \
+	--framework=pytorch \
+	--category=edge \
+	--scenario=Offline \
+	--execution_mode=valid \
+	--device=cuda \
+	--quiet
+```
+*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
+ you should simply reload mlcommons@cm4mlops without checkout and clean CM cache as follows:*
+
+```bash
+cm rm repo mlcommons@cm4mlops
+cm pull repo mlcommons@cm4mlops
+cm rm cache -f
+
+```
+
+## Results
+
+Platform: 666dfb9811e4-reference-gpu-pytorch-cu124
+
+Model Precision: fp32
+
+### Accuracy Results 
+`F1`: `90.87602`, Required accuracy for closed division `>= 89.96526`
+
+### Performance Results 
+`Samples per second`: `85.4467`
diff --git a/...easurements/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline/accuracy_console.out b/...easurements/666dfb9811e4-reference-gpu-pytorch-cu124/bert-99/offline/accuracy_console.out