Skip to content

Code for replicating Roboflow 100 benchmark results and programmatically downloading benchmark datasets

License

Notifications You must be signed in to change notification settings

roboflow/roboflow-100-benchmark

Repository files navigation

Roboflow Blog Roboflow Notebook YouTube

Note RF100 is part of the CVinW challenge!

Roboflow 100 📸: A Rich, Multi-Domain Object Detection Benchmark

rf100blog-mosaicthing

This repository implements the Roboflow 100 benchmark developed by Roboflow. It contains code to download the dataset and reproduce mAP values for YOLOv5 and YOLOv7 Fine-Tuning and GLIP Evaluation on 100 of Roboflow Universe datasets.

RF100 was sponsored with ❤️ by Intel

RF100 paper is on arxiv, if you used RF100 in your research, please consider citing us

@misc{2211.13523,
Author = {Floriana Ciaglia and Francesco Saverio Zuppichini and Paul Guerrie and Mark McQuade and Jacob Solawetz},
Title = {Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
Year = {2022},
Eprint = {arXiv:2211.13523},
}

RF100

RF100 contains the following datasets, carefully chosen from more than 90'000 datasets hosted on our universe hub. The datasets are splitted in 7 categories: Aerial, Videogames, Microscopic, Underwater, Documents, Electromagnetic and Real World.

Category Datasets Images Classes
Aerial 7 9683 24
Videogames 7 11579 88
Microscopic 11 13378 28
Underwater 5 18003 39
Documents 8 24813 90
Electromagnetic 12 36381 41
Real World 50 110615 495
Total 100 224,714 805

Note some datasets failed, only on 88

category yolov5 yolov7 yolov8 glip
aerial 0.636 0.504286 0.707286 0.0238571
documents 0.716125 0.7225 0.79775 0.018625
electromagnetic 0.7423 0.639 0.7899 0.0697
microscopic 0.650727 0.59166 0.747455 0.0230909
real world 0.781659 0.705276 0.842171 0.117049
underwater 0.56 0.6624 0.7024 0.1876
videogames 0.863333 0.783167 0.881 0.1625
Total 0.7071 0.6583 0.7811 0.0860

🚨 Head over the Appendix at the bottom to see samples from each dataset

We provide a notebook to help you using RF100 with PyTorch

Getting Started

First, clone this repo and go inside it.

git clone https://github.com/roboflow-ai/roboflow-100-benchmark.git
cd roboflow-100-benchmark
git submodule update --init --recursive

You will need an API key. RF100 can be accessed with any key from Roboflow, head over our doc to learn how to get one.

Then, export the key to your current shell

export ROBOFLOW_API_KEY=<YOUR_API_KEY>

Note: The datasets are taken from datasets_links.txt, you can modify that file to add/remove datasets.

Docker

The easiest and faster way to download RF100 is using docker and our Dockerfile.

NOTE Be sure to do the post process steps after you installed docker, we will read and write to shared volumes so your user should also be in the docker group.

If you have an NVIDIA GPU, be sure to also install nvidia docker

Be sure to have the ROBOFLOW_API_KEY in your env, then run it (we are passing our user so you shouldn't have permission errors on rf100 folder)

docker run --rm -it \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v $(pwd)/rf100:/workspace/rf100 \
    -v /etc/group:/etc/group:ro \
    -u "$(id -u):$(id -g)" \
    roboflow/rf100-download

roboflow/rf100-download is hosted on our docker hub

Internally, RF100 will downloaded to /app/rf100. You can also specify the format with the -f flag, by default coco is used.

docker run --rm -it \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/rf100:/workspace/rf100 \
    -v /etc/group:/etc/group:ro \
    -u "$(id -u):$(id -g)" \
    roboflow/rf100-download -f yolov5

If you want to build the container and not use the image on the hub, run

docker build -t rf100-download -f Dockerfile.rf100.download .

Local Env

To download RF100 in your local enviroment (python >=3.6), you need to install roboflow

pip install roboflow

Then,

chmod 770 ./scripts/download_datasets.sh
./scripts/download_datasets.sh
./scripts/download_datasets.sh -f yolov5 $ change format
./scripts/download_datasets.sh -l <path_to_my_location> change download location

Formats

Supported formats are

  • coco
  • yolov5 (used by YOLOv7 as well)

Reproduce Results

We will use docker to ensure the same enviroment is used.

First, build the container

docker build -t rf100-benchmark -f Dockerfile.rf100.benchmark .

Then, follow the guide for each model.

All results are stored inside ./runs.

Warning The run must be uninterrupted otherwise the downloaded datasets may be unfinished or corrupted

Warning If you have permission denied errors, you can either remove runs with sudo rm -rf ./runs or give permissions to everybody sudo chmod -R +x ./runs.

Warning Due to poor code support, random errors and time, we weren't able to run all the models on all datasets. Only 88 datasets were successfully benchmarked on all four models

YOLOv5 Fine-Tuning

Note, we will map the current folder to the container file system to persist data

mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./yolov5-benchmark/train.sh

YOLOv7 Fine-Tuning

Note, we will map the current folder to the container file system to persist data

mkdir -p runs &&
nvidia-docker run --gpus all --rm -d --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./yolov7-benchmark/train.sh

YOLOv8 Fine-Tuning

Note, Doc coming soon but we have the results (was done internally by another team member)

Note The model failed on these datasets: bone-fracture-7fylg, brain-tumor-m2pbp, cable-damage, circuit-elements and printed-circuit-board

Note The model failed on these datasets: apex-videogame, apples-fvpl5, hand-gestures-jps7z, road-traffic, wall-damage

mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
    -e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
    -v ${PWD}/runs:/workspace/runs \
    -v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
    rf100-benchmark ./GLIP-benchmark/train.sh

Note We create a notebook to analysis why and where GLIP did bad and good

COCO evaluation

If you are interesting in only the test annotations for rf100, e.g. you are using pycocotools to compute maps, we have create a zip file with root/<dataset_name>/<test_annotations>.json. We hope this will make it easier for you to evaluate your models.

Appendix

YOLOs Comparison

Note 5 datasets failed on v8, so the comparison is only on 95. Check v8 for more info.

Note We only train one seed for each model, take these results with a grain of salt

The following box plots shows [email protected] for each YOLOs model on RF100

The following bar plots shows [email protected] for each YOLOs model on each RF100 category

Samples

dataset category samples
hand-gestures-jps7z real world alt
smoke-uvylj real world alt
wall-damage real world alt
corrosion-bi3q3 real world alt
excavators-czvg9 real world alt
chess-pieces-mjzgj real world alt
road-signs-6ih4y real world alt
street-work real world alt
construction-safety-gsnvb real world alt
road-traffic real world alt
washroom-rf1fa real world alt
circuit-elements real world alt
mask-wearing-608pr real world alt
cables-nl42k real world alt
soda-bottles real world alt
truck-movement real world alt
wine-labels real world alt
digits-t2eg6 real world alt
vehicles-q0x2v real world alt
peanuts-sd4kf real world alt
printed-circuit-board real world alt
pests-2xlvx real world alt
cavity-rs0uf real world alt
leaf-disease-nsdsr real world alt
marbles real world alt
pills-sxdht real world alt
poker-cards-cxcvz real world alt
number-ops real world alt
insects-mytwu real world alt
cotton-20xz5 real world alt
furniture-ngpea real world alt
cable-damage real world alt
animals-ij5d2 real world alt
coins-1apki real world alt
apples-fvpl5 real world alt
people-in-paintings real world alt
circuit-voltages real world alt
uno-deck real world alt
grass-weeds real world alt
gauge-u2lwv real world alt
sign-language-sokdr real world alt
valentines-chocolate real world alt
fish-market-ggjso real world alt
lettuce-pallets real world alt
shark-teeth-5atku real world alt
bees-jt5in real world alt
sedimentary-features-9eosf real world alt
currency-v4f8j real world alt
trail-camera real world alt
cell-towers real world alt
apex-videogame videogames alt
farcry6-videogame videogames alt
csgo-videogame videogames alt
avatar-recognition-nuexe videogames alt
halo-infinite-angel-videogame videogames alt
team-fight-tactics videogames alt
robomasters-285km videogames alt
tweeter-posts documents alt
tweeter-profile documents alt
document-parts documents alt
activity-diagrams-qdobr documents alt
signatures-xc8up documents alt
paper-parts documents alt
tabular-data-wf9uh documents alt
paragraphs-co84b documents alt
underwater-pipes-4ng4t underwater alt
aquarium-qlnqy underwater alt
peixos-fish underwater alt
underwater-objects-5v7p8 underwater alt
coral-lwptl underwater alt
aerial-pool aerial alt
secondary-chains aerial alt
aerial-spheres aerial alt
soccer-players-5fuqs aerial alt
weed-crop-aerial aerial alt
aerial-cows aerial alt
cloud-types aerial alt
stomata-cells microscopic alt
bccd-ouzjz microscopic alt
parasites-1s07h microscopic alt
cells-uyemf microscopic alt
4-fold-defect microscopic alt
bacteria-ptywi microscopic alt
cotton-plant-disease microscopic alt
mitosis-gjs3g microscopic alt
phages microscopic alt
liver-disease microscopic alt
asbestos microscopic alt
thermal-dogs-and-people-x6ejw electromagnetic alt
solar-panels-taxvb electromagnetic alt
radio-signal electromagnetic alt
thermal-cheetah-my4dp electromagnetic alt
x-ray-rheumatology electromagnetic alt
acl-x-ray electromagnetic alt
abdomen-mri electromagnetic alt
axial-mri electromagnetic alt
gynecology-mri electromagnetic alt
brain-tumor-m2pbp electromagnetic alt
bone-fracture-7fylg electromagnetic alt
flir-camera-objects electromagnetic alt

Credits

We thank all the authors of the original datasets, below a table linking the Roboflow 100 dataset's name to its original counterpart

dataset original
hand-gestures-jps7z https://universe.roboflow.com/hand-gestures-recognition/hand-gestures-dataset
smoke-uvylj https://universe.roboflow.com/sigma-pub/smoke-detection-sigma
wall-damage https://universe.roboflow.com/sina-uyen0/damage_level_detection
corrosion-bi3q3 https://universe.roboflow.com/khaingwintz-gmail-com/dataset--2-pathein-train-plus-v-3-update-mm
excavators-czvg9 https://universe.roboflow.com/mohamed-sabek-6zmr6/excavators-cwlh0
chess-pieces-mjzgj https://universe.roboflow.com/joseph-nelson/chess-pieces-new
road-signs-6ih4y https://universe.roboflow.com/project-sign-detection/traffic-sign-cdfml
street-work https://universe.roboflow.com/cone/capacetes-e-cones
construction-safety-gsnvb https://universe.roboflow.com/computer-vision/worker-safety
road-traffic https://universe.roboflow.com/due/detection-dlzhy
washroom-rf1fa https://universe.roboflow.com/imagetaggingworkspace/washroom-image-tagging
circuit-elements https://universe.roboflow.com/new-workspace-rzrja/pcb-2.0
mask-wearing-608pr https://universe.roboflow.com/joseph-nelson/mask-wearing
cables-nl42k https://universe.roboflow.com/annotationericsson/annotation-2.0
soda-bottles https://universe.roboflow.com/food7/test1-iajnv
truck-movement https://universe.roboflow.com/psi-dhxqe/psi-rossville-pano
wine-labels https://universe.roboflow.com/wine-label/wine-label-detection
digits-t2eg6 https://universe.roboflow.com/dmrs/number-1gmaw
vehicles-q0x2v https://universe.roboflow.com/7-class/11-11-2021-09.41
peanuts-sd4kf https://universe.roboflow.com/molds-onbk3/peanuts-mckge/
printed-circuit-board https://universe.roboflow.com/new-workspace-rzrja/pcb-2.0
pests-2xlvx https://universe.roboflow.com/gugugu/pests-f8kkr
cavity-rs0uf https://universe.roboflow.com/duong-duc-cuong/cavity-n3ioq
leaf-disease-nsdsr https://universe.roboflow.com/puri/puri4-ygapu
marbles https://universe.roboflow.com/zhe-fan/marble-images
pills-sxdht https://universe.roboflow.com/mohamed-attia-e2mor/pill-detection-llp4r
poker-cards-cxcvz https://universe.roboflow.com/roboflow-100/poker-cards-cxcvz
number-ops https://universe.roboflow.com/mnist-bvalq/mnist-icrul
insects-mytwu https://universe.roboflow.com/nirmani/yolo-custome-925
cotton-20xz5 https://universe.roboflow.com/cotton-nqp2x/bt-cotton
furniture-ngpea https://universe.roboflow.com/minoj-selvaraj/furniture-sfocl
cable-damage https://universe.roboflow.com/st-hedgehog-yusupov-gmail-com/kanaaat
animals-ij5d2 https://universe.roboflow.com/dane-sprsiter/barnyard
coins-1apki https://universe.roboflow.com/labelimg/label_coin
apples-fvpl5 https://universe.roboflow.com/arfiani-nur-sayidah-9lizr/apple-sorting-2bfhk
people-in-paintings https://universe.roboflow.com/raya-al/french-paintings-dataset-d2vbe
circuit-voltages https://universe.roboflow.com/vanitchaporn/circuit-gexit
uno-deck https://universe.roboflow.com/joseph-nelson/uno-cards
grass-weeds https://universe.roboflow.com/jan-douwe/testbl
gauge-u2lwv https://universe.roboflow.com/evankim9903-gmail-com/gauge_detection
sign-language-sokdr https://universe.roboflow.com/david-lee-d0rhs/american-sign-language-letters
valentines-chocolate https://universe.roboflow.com/chocolates/valentines-chocolates
fish-market-ggjso https://universe.roboflow.com/commolybroken/dataset-z2vab
lettuce-pallets https://universe.roboflow.com/lettucedetector
shark-teeth-5atku https://universe.roboflow.com/sharks/shark-taxonomy
bees-jt5in https://universe.roboflow.com/jjb-object-detection-projects/bee-detection-pry0w
sedimentary-features-9eosf https://universe.roboflow.com/sedimentary-structures/sedimentary-features-rmadz
currency-v4f8j https://universe.roboflow.com/alex-hyams-cosqx/cash-counter/
trail-camera https://universe.roboflow.com/my-game-pics/my-game-pics
cell-towers https://universe.roboflow.com/yuyang-li/tower_jointv1
apex-videogame https://universe.roboflow.com/apex-esoic/apexyolov4
farcry6-videogame https://universe.roboflow.com/kais-al-hajjih/farcry6-hackathon
csgo-videogame https://universe.roboflow.com/new-workspace-rp0z0/csgo-train-yolo-v5
avatar-recognition-nuexe https://universe.roboflow.com/new-workspace-0pohs/avatar-recognition-rfw8d
halo-infinite-angel-videogame https://universe.roboflow.com/graham-doerksen/halo-infinite-angel-aim
team-fight-tactics https://universe.roboflow.com/lamaitw/lama-itw/
robomasters-285km https://universe.roboflow.com/etp5501-gmail-com/robomasters-colored
tweeter-posts https://universe.roboflow.com/tweeter/tweeter
tweeter-profile https://universe.roboflow.com/wojciech-blachowski/tweets
document-parts https://universe.roboflow.com/new-workspace-vf0ib/calpers
activity-diagrams-qdobr https://universe.roboflow.com/public1/activity-diagrams-s7sxv
signatures-xc8up https://universe.roboflow.com/signature-detection/signaturesdetectiob
paper-parts https://universe.roboflow.com/object-detection-alan-devera/object-detection-ycqjb
tabular-data-wf9uh https://universe.roboflow.com/rik-biswas/tabular-data-dh4ek
paragraphs-co84b https://universe.roboflow.com/new-workspace-4vus5/singlemcq
underwater-pipes-4ng4t https://universe.roboflow.com/underwaterpipes/underwater_pipes_orginal_pictures
aquarium-qlnqy https://universe.roboflow.com/brad-dwyer/aquarium-combined
peixos-fish https://universe.roboflow.com/nasca37/peixos3
underwater-objects-5v7p8 https://universe.roboflow.com/workspace-txxpz/underwater-detection
coral-lwptl https://universe.roboflow.com/nikita-manolis-je2ii/coral-growth-form
aerial-pool https://universe.roboflow.com/a-s/uwh
secondary-chains https://universe.roboflow.com/cc_moon/secondaries
aerial-spheres https://universe.roboflow.com/mevil-crasta/annotating-spheres---11-04
soccer-players-5fuqs https://universe.roboflow.com/ilyes-talbi-ptwsp/futbol-players
weed-crop-aerial https://universe.roboflow.com/new-workspace-csmgu/weedcrop-waifl
aerial-cows https://universe.roboflow.com/omarkapur-berkeley-edu/livestalk
cloud-types https://universe.roboflow.com/research-project/shallow-cloud
stomata-cells https://universe.roboflow.com/new-workspace-fditd/stomata02
bccd-ouzjz https://universe.roboflow.com/team-roboflow/blood-cell-detection-1ekwu
parasites-1s07h https://universe.roboflow.com/graduao/sistema-para-analise-de-ovos-de-parasitas-em-amostra-de-agua-e-sangue
cells-uyemf https://universe.roboflow.com/new-workspace-86q1t/t03-proyecto-celula-dataset-ampliado
4-fold-defect https://universe.roboflow.com/kat-laura/defect-detection-gil85
bacteria-ptywi https://universe.roboflow.com/terada-shoma/gram-positive-bacteria
cotton-plant-disease https://universe.roboflow.com/quandong-qian/desease-cotton-plant
mitosis-gjs3g https://universe.roboflow.com/20029-tkmce-ac-in/mitosis-dwute
phages https://universe.roboflow.com/danish2562022-gmail-com/microglia_rgb/
liver-disease https://universe.roboflow.com/liver-t5yvf/liver-diseases
asbestos https://universe.roboflow.com/ahmad-rabiee/asbest91
thermal-dogs-and-people-x6ejw https://universe.roboflow.com/joseph-nelson/thermal-dogs-and-people
solar-panels-taxvb https://universe.roboflow.com/new-workspace-rt1da/solarpaneldetectmodel
radio-signal https://universe.roboflow.com/danil/
thermal-cheetah-my4dp https://universe.roboflow.com/brad-dwyer/thermal-cheetah
x-ray-rheumatology https://universe.roboflow.com/publictestsite/xray-rheumatology-images-public
acl-x-ray https://universe.roboflow.com/objectdetection-9lu9z/detectron2-acl
abdomen-mri https://universe.roboflow.com/xinweihe/circle-3train
axial-mri https://universe.roboflow.com/tfg-2nmge/axial-dataset
gynecology-mri https://universe.roboflow.com/yuanyuanpei7/5-8w
brain-tumor-m2pbp https://universe.roboflow.com/yousef-ghanem-jzj4y/brain-tumor-detection-fpf1f
bone-fracture-7fylg https://universe.roboflow.com/science-research/science-research-2022:-bone-fracture-detection
flir-camera-objects https://universe.roboflow.com/thermal-imaging-0hwfw/flir-data-set