Improvements to the baseline ViT

[Yash-10]

Notes

• We use class tokens currently (we perform regression, so these class tokens are "regression tokens" in reality). An alternative is to use Global Average Pooling (GAP), as mentioned in the ViT paper (https://arxiv.org/pdf/2010.11929v2.pdf). See discussion here: https://datascience.stackexchange.com/questions/90649/class-token-in-vit-and-bert
• https://wandb.ai/wandb_fc/tips/reports/Dos-and-Don-ts-of-Vision-Transformers-ViTs---VmlldzoyNDQ5MDQ3
• A summary of how to train your ViT: https://dailyai.github.io/2021-06-21/2106-10270

Papers

See the start of the ViT notebook for more links/references.

• https://arxiv.org/pdf/2105.09121.pdf