-
Notifications
You must be signed in to change notification settings - Fork 95
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding index files for evaluation datasets
Previosly these files were stored in another repository. It is better to host them here to have a better correspondance between benchmark versions and dataset index files.
- Loading branch information
Showing
8 changed files
with
5,487 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
# Performance evaluation datasets | ||
|
||
This folder contains the index files for performance evaluation runs. Files udner this folder were previosly stored in https://github.com/eembc/energyrunner/tree/main/datasets | ||
|
||
# File Specifics | ||
|
||
| Model | Source | Dimensions | Min Accuracy | File Format | # Stimuli | | ||
| ----- | -------- | ---------- | ------------- | ---------------------- | --------- | | ||
| vww01 | COCO2014/val2017 | 96x96 | 80% | U8C3, RGB, where [0]=ulc and [9215]=lrc | 500 true, 500 false| | ||
| ic01 | CIFAR-10 | 32x32 | 85% | U8C3, RGB, where [0]=ulc and [1024]=lrc; this is different from original CiFAR-10 array which is 1024R, 1024G, 1024B | 200, 10 classes | | ||
| ad01 | ToyADMOS/car | n/a | AUC: 0.85 | Spectrogram, 5 slices, 128 freq. bins, FP32LE | 108 anomaly, 140 normal | | ||
| kws01 | Speech Commands v2 | n/a | 90% | Spectrogram, 49 frames x 10 MFCCs as INT8 | 1000 features, 12 classes | | ||
|
||
# Ground Truth Format | ||
|
||
For everything but anomaly detection, the y_labels.csv file format is: | ||
|
||
``` | ||
input file name,total number of classes,predicted class number | ||
``` | ||
|
||
For anomaly datection, the y_labels.csv file format specifies a sliding window for the input file | ||
|
||
``` | ||
input file name,total number of classes,predicted classes,window width (bytes),stride (bytes) | ||
``` | ||
|
||
Where number of classes is always 2 (anomaly, normal), and 0=normal, 1=anomaly. | ||
|
||
For each entry, the corresponding file must exist in the same directory as the ground-truths file. Copy those files from the dataset to the folder. For example, if you are using the image classification model, `ic01`, do this after completing training: | ||
|
||
``` Bash | ||
% cd tiny/v0.5/training/image_classification | ||
% mkdir -p ~/eembc/runner/benchmarks/ulp-mlperf/datasets/ic01/ | ||
% cp y_labels.c !$ | ||
% cp perf_dataset/* !$ | ||
``` | ||
|
||
Each model has it's own method for constructing the input files. Please refer to the [`training`](https://github.com/mlcommons/tiny/tree/master/benchmark/training) folder in the tiny repo the model you are interested in, the README's will explain more. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,248 @@ | ||
normal_id_01_00000003_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000013_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000023_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000033_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000043_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000053_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000063_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000073_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000083_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000093_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000103_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000113_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000123_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000133_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000143_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000153_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000163_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000173_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000183_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000193_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000203_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000213_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000223_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000233_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000243_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000253_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000263_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000273_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000283_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000293_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000303_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000313_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000323_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000333_hist_librosa.bin,2,0,2560,512 | ||
normal_id_01_00000343_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000003_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000013_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000023_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000033_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000043_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000053_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000063_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000073_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000083_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000093_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000103_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000113_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000123_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000133_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000143_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000153_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000163_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000173_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000183_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000193_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000203_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000213_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000223_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000233_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000243_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000253_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000263_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000273_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000283_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000293_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000303_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000313_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000323_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000333_hist_librosa.bin,2,0,2560,512 | ||
normal_id_02_00000343_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000003_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000013_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000023_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000033_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000043_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000053_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000063_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000073_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000083_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000093_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000103_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000113_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000123_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000133_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000143_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000153_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000163_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000173_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000183_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000193_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000203_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000213_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000223_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000233_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000243_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000253_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000263_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000273_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000283_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000293_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000303_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000313_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000323_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000333_hist_librosa.bin,2,0,2560,512 | ||
normal_id_03_00000343_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000003_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000013_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000023_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000033_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000043_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000053_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000063_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000073_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000083_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000093_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000103_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000113_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000123_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000133_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000143_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000153_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000163_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000173_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000183_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000193_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000203_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000213_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000223_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000233_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000243_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000253_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000263_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000273_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000283_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000293_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000303_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000313_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000323_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000333_hist_librosa.bin,2,0,2560,512 | ||
normal_id_04_00000343_hist_librosa.bin,2,0,2560,512 | ||
anomaly_id_01_00000003_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000013_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000023_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000033_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000043_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000053_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000063_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000073_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000083_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000093_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000103_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000113_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000123_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000133_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000143_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000153_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000163_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000173_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000183_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000193_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000203_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000213_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000223_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000233_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000243_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000253_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_01_00000263_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000003_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000013_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000023_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000033_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000043_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000053_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000063_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000073_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000083_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000093_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000103_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000113_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000123_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000133_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000143_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000153_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000163_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000173_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000183_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000193_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000203_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000213_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000223_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000233_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000243_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000253_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_02_00000263_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000003_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000013_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000023_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000033_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000043_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000053_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000063_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000073_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000083_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000093_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000103_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000113_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000123_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000133_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000143_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000153_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000163_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000173_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000183_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000193_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000203_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000213_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000223_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000233_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000243_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000253_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_03_00000263_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000003_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000013_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000023_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000033_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000043_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000053_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000063_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000073_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000083_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000093_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000103_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000113_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000123_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000133_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000143_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000153_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000163_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000173_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000183_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000193_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000203_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000213_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000223_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000233_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000243_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000253_hist_librosa.bin,2,1,2560,512 | ||
anomaly_id_04_00000263_hist_librosa.bin,2,1,2560,512 |
Oops, something went wrong.