Skip to content

Commit

Permalink
Adding index files for evaluation datasets
Browse files Browse the repository at this point in the history
Previosly these files were stored in another repository. It is better to
host them here to have a better correspondance between benchmark versions
and dataset index files.
  • Loading branch information
cskiraly committed Mar 28, 2022
1 parent a7ea804 commit 6dff252
Show file tree
Hide file tree
Showing 8 changed files with 5,487 additions and 0 deletions.
39 changes: 39 additions & 0 deletions benchmark/evaluation/datasets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Performance evaluation datasets

This folder contains the index files for performance evaluation runs. Files udner this folder were previosly stored in https://github.com/eembc/energyrunner/tree/main/datasets

# File Specifics

| Model | Source | Dimensions | Min Accuracy | File Format | # Stimuli |
| ----- | -------- | ---------- | ------------- | ---------------------- | --------- |
| vww01 | COCO2014/val2017 | 96x96 | 80% | U8C3, RGB, where [0]=ulc and [9215]=lrc | 500 true, 500 false|
| ic01 | CIFAR-10 | 32x32 | 85% | U8C3, RGB, where [0]=ulc and [1024]=lrc; this is different from original CiFAR-10 array which is 1024R, 1024G, 1024B | 200, 10 classes |
| ad01 | ToyADMOS/car | n/a | AUC: 0.85 | Spectrogram, 5 slices, 128 freq. bins, FP32LE | 108 anomaly, 140 normal |
| kws01 | Speech Commands v2 | n/a | 90% | Spectrogram, 49 frames x 10 MFCCs as INT8 | 1000 features, 12 classes |

# Ground Truth Format

For everything but anomaly detection, the y_labels.csv file format is:

```
input file name,total number of classes,predicted class number
```

For anomaly datection, the y_labels.csv file format specifies a sliding window for the input file

```
input file name,total number of classes,predicted classes,window width (bytes),stride (bytes)
```

Where number of classes is always 2 (anomaly, normal), and 0=normal, 1=anomaly.

For each entry, the corresponding file must exist in the same directory as the ground-truths file. Copy those files from the dataset to the folder. For example, if you are using the image classification model, `ic01`, do this after completing training:

``` Bash
% cd tiny/v0.5/training/image_classification
% mkdir -p ~/eembc/runner/benchmarks/ulp-mlperf/datasets/ic01/
% cp y_labels.c !$
% cp perf_dataset/* !$
```

Each model has it's own method for constructing the input files. Please refer to the [`training`](https://github.com/mlcommons/tiny/tree/master/benchmark/training) folder in the tiny repo the model you are interested in, the README's will explain more.
248 changes: 248 additions & 0 deletions benchmark/evaluation/datasets/ad01/y_labels.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
normal_id_01_00000003_hist_librosa.bin,2,0,2560,512
normal_id_01_00000013_hist_librosa.bin,2,0,2560,512
normal_id_01_00000023_hist_librosa.bin,2,0,2560,512
normal_id_01_00000033_hist_librosa.bin,2,0,2560,512
normal_id_01_00000043_hist_librosa.bin,2,0,2560,512
normal_id_01_00000053_hist_librosa.bin,2,0,2560,512
normal_id_01_00000063_hist_librosa.bin,2,0,2560,512
normal_id_01_00000073_hist_librosa.bin,2,0,2560,512
normal_id_01_00000083_hist_librosa.bin,2,0,2560,512
normal_id_01_00000093_hist_librosa.bin,2,0,2560,512
normal_id_01_00000103_hist_librosa.bin,2,0,2560,512
normal_id_01_00000113_hist_librosa.bin,2,0,2560,512
normal_id_01_00000123_hist_librosa.bin,2,0,2560,512
normal_id_01_00000133_hist_librosa.bin,2,0,2560,512
normal_id_01_00000143_hist_librosa.bin,2,0,2560,512
normal_id_01_00000153_hist_librosa.bin,2,0,2560,512
normal_id_01_00000163_hist_librosa.bin,2,0,2560,512
normal_id_01_00000173_hist_librosa.bin,2,0,2560,512
normal_id_01_00000183_hist_librosa.bin,2,0,2560,512
normal_id_01_00000193_hist_librosa.bin,2,0,2560,512
normal_id_01_00000203_hist_librosa.bin,2,0,2560,512
normal_id_01_00000213_hist_librosa.bin,2,0,2560,512
normal_id_01_00000223_hist_librosa.bin,2,0,2560,512
normal_id_01_00000233_hist_librosa.bin,2,0,2560,512
normal_id_01_00000243_hist_librosa.bin,2,0,2560,512
normal_id_01_00000253_hist_librosa.bin,2,0,2560,512
normal_id_01_00000263_hist_librosa.bin,2,0,2560,512
normal_id_01_00000273_hist_librosa.bin,2,0,2560,512
normal_id_01_00000283_hist_librosa.bin,2,0,2560,512
normal_id_01_00000293_hist_librosa.bin,2,0,2560,512
normal_id_01_00000303_hist_librosa.bin,2,0,2560,512
normal_id_01_00000313_hist_librosa.bin,2,0,2560,512
normal_id_01_00000323_hist_librosa.bin,2,0,2560,512
normal_id_01_00000333_hist_librosa.bin,2,0,2560,512
normal_id_01_00000343_hist_librosa.bin,2,0,2560,512
normal_id_02_00000003_hist_librosa.bin,2,0,2560,512
normal_id_02_00000013_hist_librosa.bin,2,0,2560,512
normal_id_02_00000023_hist_librosa.bin,2,0,2560,512
normal_id_02_00000033_hist_librosa.bin,2,0,2560,512
normal_id_02_00000043_hist_librosa.bin,2,0,2560,512
normal_id_02_00000053_hist_librosa.bin,2,0,2560,512
normal_id_02_00000063_hist_librosa.bin,2,0,2560,512
normal_id_02_00000073_hist_librosa.bin,2,0,2560,512
normal_id_02_00000083_hist_librosa.bin,2,0,2560,512
normal_id_02_00000093_hist_librosa.bin,2,0,2560,512
normal_id_02_00000103_hist_librosa.bin,2,0,2560,512
normal_id_02_00000113_hist_librosa.bin,2,0,2560,512
normal_id_02_00000123_hist_librosa.bin,2,0,2560,512
normal_id_02_00000133_hist_librosa.bin,2,0,2560,512
normal_id_02_00000143_hist_librosa.bin,2,0,2560,512
normal_id_02_00000153_hist_librosa.bin,2,0,2560,512
normal_id_02_00000163_hist_librosa.bin,2,0,2560,512
normal_id_02_00000173_hist_librosa.bin,2,0,2560,512
normal_id_02_00000183_hist_librosa.bin,2,0,2560,512
normal_id_02_00000193_hist_librosa.bin,2,0,2560,512
normal_id_02_00000203_hist_librosa.bin,2,0,2560,512
normal_id_02_00000213_hist_librosa.bin,2,0,2560,512
normal_id_02_00000223_hist_librosa.bin,2,0,2560,512
normal_id_02_00000233_hist_librosa.bin,2,0,2560,512
normal_id_02_00000243_hist_librosa.bin,2,0,2560,512
normal_id_02_00000253_hist_librosa.bin,2,0,2560,512
normal_id_02_00000263_hist_librosa.bin,2,0,2560,512
normal_id_02_00000273_hist_librosa.bin,2,0,2560,512
normal_id_02_00000283_hist_librosa.bin,2,0,2560,512
normal_id_02_00000293_hist_librosa.bin,2,0,2560,512
normal_id_02_00000303_hist_librosa.bin,2,0,2560,512
normal_id_02_00000313_hist_librosa.bin,2,0,2560,512
normal_id_02_00000323_hist_librosa.bin,2,0,2560,512
normal_id_02_00000333_hist_librosa.bin,2,0,2560,512
normal_id_02_00000343_hist_librosa.bin,2,0,2560,512
normal_id_03_00000003_hist_librosa.bin,2,0,2560,512
normal_id_03_00000013_hist_librosa.bin,2,0,2560,512
normal_id_03_00000023_hist_librosa.bin,2,0,2560,512
normal_id_03_00000033_hist_librosa.bin,2,0,2560,512
normal_id_03_00000043_hist_librosa.bin,2,0,2560,512
normal_id_03_00000053_hist_librosa.bin,2,0,2560,512
normal_id_03_00000063_hist_librosa.bin,2,0,2560,512
normal_id_03_00000073_hist_librosa.bin,2,0,2560,512
normal_id_03_00000083_hist_librosa.bin,2,0,2560,512
normal_id_03_00000093_hist_librosa.bin,2,0,2560,512
normal_id_03_00000103_hist_librosa.bin,2,0,2560,512
normal_id_03_00000113_hist_librosa.bin,2,0,2560,512
normal_id_03_00000123_hist_librosa.bin,2,0,2560,512
normal_id_03_00000133_hist_librosa.bin,2,0,2560,512
normal_id_03_00000143_hist_librosa.bin,2,0,2560,512
normal_id_03_00000153_hist_librosa.bin,2,0,2560,512
normal_id_03_00000163_hist_librosa.bin,2,0,2560,512
normal_id_03_00000173_hist_librosa.bin,2,0,2560,512
normal_id_03_00000183_hist_librosa.bin,2,0,2560,512
normal_id_03_00000193_hist_librosa.bin,2,0,2560,512
normal_id_03_00000203_hist_librosa.bin,2,0,2560,512
normal_id_03_00000213_hist_librosa.bin,2,0,2560,512
normal_id_03_00000223_hist_librosa.bin,2,0,2560,512
normal_id_03_00000233_hist_librosa.bin,2,0,2560,512
normal_id_03_00000243_hist_librosa.bin,2,0,2560,512
normal_id_03_00000253_hist_librosa.bin,2,0,2560,512
normal_id_03_00000263_hist_librosa.bin,2,0,2560,512
normal_id_03_00000273_hist_librosa.bin,2,0,2560,512
normal_id_03_00000283_hist_librosa.bin,2,0,2560,512
normal_id_03_00000293_hist_librosa.bin,2,0,2560,512
normal_id_03_00000303_hist_librosa.bin,2,0,2560,512
normal_id_03_00000313_hist_librosa.bin,2,0,2560,512
normal_id_03_00000323_hist_librosa.bin,2,0,2560,512
normal_id_03_00000333_hist_librosa.bin,2,0,2560,512
normal_id_03_00000343_hist_librosa.bin,2,0,2560,512
normal_id_04_00000003_hist_librosa.bin,2,0,2560,512
normal_id_04_00000013_hist_librosa.bin,2,0,2560,512
normal_id_04_00000023_hist_librosa.bin,2,0,2560,512
normal_id_04_00000033_hist_librosa.bin,2,0,2560,512
normal_id_04_00000043_hist_librosa.bin,2,0,2560,512
normal_id_04_00000053_hist_librosa.bin,2,0,2560,512
normal_id_04_00000063_hist_librosa.bin,2,0,2560,512
normal_id_04_00000073_hist_librosa.bin,2,0,2560,512
normal_id_04_00000083_hist_librosa.bin,2,0,2560,512
normal_id_04_00000093_hist_librosa.bin,2,0,2560,512
normal_id_04_00000103_hist_librosa.bin,2,0,2560,512
normal_id_04_00000113_hist_librosa.bin,2,0,2560,512
normal_id_04_00000123_hist_librosa.bin,2,0,2560,512
normal_id_04_00000133_hist_librosa.bin,2,0,2560,512
normal_id_04_00000143_hist_librosa.bin,2,0,2560,512
normal_id_04_00000153_hist_librosa.bin,2,0,2560,512
normal_id_04_00000163_hist_librosa.bin,2,0,2560,512
normal_id_04_00000173_hist_librosa.bin,2,0,2560,512
normal_id_04_00000183_hist_librosa.bin,2,0,2560,512
normal_id_04_00000193_hist_librosa.bin,2,0,2560,512
normal_id_04_00000203_hist_librosa.bin,2,0,2560,512
normal_id_04_00000213_hist_librosa.bin,2,0,2560,512
normal_id_04_00000223_hist_librosa.bin,2,0,2560,512
normal_id_04_00000233_hist_librosa.bin,2,0,2560,512
normal_id_04_00000243_hist_librosa.bin,2,0,2560,512
normal_id_04_00000253_hist_librosa.bin,2,0,2560,512
normal_id_04_00000263_hist_librosa.bin,2,0,2560,512
normal_id_04_00000273_hist_librosa.bin,2,0,2560,512
normal_id_04_00000283_hist_librosa.bin,2,0,2560,512
normal_id_04_00000293_hist_librosa.bin,2,0,2560,512
normal_id_04_00000303_hist_librosa.bin,2,0,2560,512
normal_id_04_00000313_hist_librosa.bin,2,0,2560,512
normal_id_04_00000323_hist_librosa.bin,2,0,2560,512
normal_id_04_00000333_hist_librosa.bin,2,0,2560,512
normal_id_04_00000343_hist_librosa.bin,2,0,2560,512
anomaly_id_01_00000003_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000013_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000023_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000033_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000043_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000053_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000063_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000073_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000083_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000093_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000103_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000113_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000123_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000133_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000143_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000153_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000163_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000173_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000183_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000193_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000203_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000213_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000223_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000233_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000243_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000253_hist_librosa.bin,2,1,2560,512
anomaly_id_01_00000263_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000003_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000013_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000023_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000033_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000043_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000053_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000063_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000073_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000083_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000093_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000103_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000113_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000123_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000133_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000143_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000153_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000163_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000173_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000183_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000193_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000203_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000213_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000223_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000233_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000243_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000253_hist_librosa.bin,2,1,2560,512
anomaly_id_02_00000263_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000003_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000013_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000023_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000033_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000043_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000053_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000063_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000073_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000083_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000093_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000103_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000113_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000123_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000133_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000143_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000153_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000163_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000173_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000183_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000193_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000203_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000213_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000223_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000233_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000243_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000253_hist_librosa.bin,2,1,2560,512
anomaly_id_03_00000263_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000003_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000013_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000023_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000033_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000043_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000053_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000063_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000073_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000083_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000093_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000103_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000113_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000123_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000133_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000143_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000153_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000163_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000173_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000183_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000193_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000203_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000213_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000223_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000233_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000243_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000253_hist_librosa.bin,2,1,2560,512
anomaly_id_04_00000263_hist_librosa.bin,2,1,2560,512
Loading

0 comments on commit 6dff252

Please sign in to comment.