Deep Transfer Learning to Predict Modernity of Cars from Images

In this project, we finetune a pre-trained ResNet to classify cars according to their production year and subsequently compute a score for their modernity. We also investigate which pixels in an image are most informative for the resulting prediction.

Finetuning Pre-Trained Model

We use a pretrained ResNet-18 available from torchvision for which we freeze its trainable parameters and add an additional trainable nn.Linear layer used for finetuning:

deepCars/src/models.py

Lines 5 to 33 in edefb96

	def get_pretrained_model():
	return models.resnet18(weights="IMAGENET1K_V1")
	def get_fine_tuneable_model(
	num_classes: int,
	pretrained_model: torch.nn.Module = get_pretrained_model(),
	):
	in_features_final_layer = list(pretrained_model.parameters())[-2].size()[
	1
	] # in-features to final layer of resnet18
	finetuneable_model = torch.nn.Sequential(
	*(list(pretrained_model.children())[:-1]), torch.nn.Flatten()
	) # add all but final layer from pretrained model
	# freeze paramaters of pretrained model:
	for param in finetuneable_model.parameters():
	param.requires_grad = False
	# add custom final layer to resnet18
	finetuneable_model.add_module(
	"fc",
	torch.nn.Linear(in_features=in_features_final_layer, out_features=num_classes),
	)
	# # apply softmax to final layers' outputs
	# finetuneable_model.add_module("sm", torch.nn.Softmax(dim=0))
	return finetuneable_model

Training

We trained the model on the CPU of a MacBook Pro with 16GB of RAM. This places restrictions on the possible size of the training set and the number of epochs. Below we show the result of a training run on 10000 training datapoints run for 10 epochs. Both train and test loss reduce, while the accuracy increases, even if given the computational constraints predictive performance is still limited.

Modernity Score

A car's modernity is scored according to the probabilities for different production year categories.

Model Evaluation

Class Imbalance

One reason for the decrease in test cross-entropy loss, but limited accuracy after an hour of training could be the stark class imbalance. Many more cars from more recent years exist in the dataset:

Confusion Matrix

Taking a look at the confusion matrix shows that the decrease in cross-entropy loss on the test set corresponds to the model learning to predict rough year ranges for cars, that were produced more recently, even if the exact production year is not yet correctly classified.

Saliency Map

We can understand which pixels in input space are most informative by looking at a saliency map. It highlights pixels according to gradient magnitude for the predicted class score.