Webdatasets for Linear Probe evaluation

[shrebox]

Hi, I noticed only the following web datasets work for the linear probe evaluation. It would be nice to have the following things fixed for the same:

1. Support for the rest of the datasets to do linear probing evaluations. For example, the SUN397, the train shard file is missing from the HuggingFace datasets.

2. Similar to zero-shot evaluation, an automatic way to run the linear probe evaluations would be nice to benchmark the models quickly.

Web datasets that worked for linear probe evaluation:

wds/imagenet1k
wds/fer2013
wds/voc2007
wds/cars
wds/fgvc_aircraft
wds/mnist
wds/stl10
wds/gtsrb
wds/renderedsst2
wds/vtab/caltech101
wds/vtab/cifar10
wds/vtab/cifar100
wds/vtab/clevr_count_all
wds/vtab/clevr_closest_object_distance
wds/vtab/diabetic_retinopathy
wds/vtab/dmlab
wds/vtab/dsprites_label_orientation
wds/vtab/dsprites_label_x_position
wds/vtab/dsprites_label_y_position
wds/vtab/dtd
wds/vtab/eurosat
wds/vtab/kitti_closest_vehicle_distance
wds/vtab/flowers
wds/vtab/pets
wds/vtab/pcam
wds/vtab/resisc45
wds/vtab/smallnorb_label_azimuth
wds/vtab/smallnorb_label_elevation
wds/vtab/svhn

Thanks! :)

[CAOANJIA]

I encountered the same problem that the training split of SUN397 is missing. Have you solved it?

[shrebox]

I encountered the same problem that the training split of SUN397 is missing. Have you solved it?

Unfortunately not, as there's missing train shard file (as mentioned above in the issue) for using the pipeline with web datasets :/

[mehdidc]

@djghosh13 Could you please give me access to https://huggingface.co/clip-benchmark, maybe I can fix this (my username: https://huggingface.co/mehdidc).

[djghosh13]

@mehdidc I added you as admin to the org

However, the CLIP Benchmark SUN397 webdataset is equivalent to the CLIP Benchmark regular SUN397 dataset, which uses the version from torchvision.datasets. There is no train/test split defined in that version, which means the test split contains all 108k examples. The webdataset loader errors if you load the training split, the regular loader prints a warning and loads the full dataset instead.

If we want a proper train/test split for evaluation, we will need to create another version using one of the canonical splits from the SUN397 page. It's only a subset of 39.7k examples, and unclear which of the 10 canonical splits the original CLIP paper uses, but any of them should work.

[mehdidc]

Many thanks @djghosh13 !

[mehdidc]

I uploaded a new version of sun397 with Partition 1 of Train and Partition 1 of Test here (both have 19580 examples): https://huggingface.co/datasets/clip-benchmark/wds_sun397_official_splits. Thanks again @djghosh13 for info!

clip_benchmark eval --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_sun397_official_splits/tree/main" --dataset wds/sun397 --task linear_probe