fasterrcnn_resnet50_fpn edits weights on-the-fly

[eflorico]

🐛 Describe the bug

fasterrcnn_resnet50_fpn() includes code to edit weights on-the-fly while loading them: https://github.com/pytorch/vision/blob/bf01bab6125c5f1152e4f336b470399e52a8559d/torchvision/models/detection/faster_rcnn.py#L578-578

Therefore, if you attempt to load weights after model instantiation, you will get wrong (but only slightly wrong!) weights:

# First:
model = fasterrcnn_resnet50_fpn()

# Later:
weights = FasterRCNN_ResNet50_FPN_Weights.COCO_V1
model.load_state_dict(weights.get_state_dict())

# Resulting weights will be wrong!

Instead, I would suggest applying the weight edits to the weights themselves, so that you can load e.g. FasterRCNN_ResNet50_FPN_Weights.COCO_V1_FIXED using load_state_dict, without having to dig into the torchvision source and manually editing weights yourself.

Versions

Torchvision 0.18.1