feat: Custom labels

[Jim-Encord]

Issue occurred around once we load the "dx" column, it was stored as a string representing each of the categories. Looked for a bit for a method that used either a nicer pytorch feature or instead some HF column transformer but couldn't find one.

This seems okay. A possible alternative approach is:

dataset.set_transform(lambda x: feature_map[model.get_transform(x)])

which may work better due to being it latter in the process. Would appreciate some eyes and discussion on best approach. Also very weird as to how to programmatically select out of 'NIH-Chest-X-ray-dataset'

[eloy-encord]

We shouldn't be adding dataset specific variables to all the HF datasets, can you check if the label can be used as param of load_dataset()? Otherwise would be wise to just add a line after adding the dataset that performs the subsetting if required.

[Jim-Encord]

I don't think that label can be used as an argument to load_dataset (I've looked into the source and checked whether their 'features' param can do it and believe not). The purpose of the label argument would be for datasets like the Skin_cancer dataset where they haven't explicitly declared a label column and encoded it. That we can select the column from inspecting the dataset. I can't really think of a workaround around this complexity. We could have in the HF loader method instead, that we apply a transformation that takes the "label" column, encodes it and makes that the "label" column. But I wanted to do it so it was applied 'lazily'

[eloy-encord]

We need to rethink this as the point of using **kwargs in the build method is to feed custom parameters to custom datasets, not a great plan to add the custom params to all the datasets. We are also thinking in those datasets that will require some data manipulation in the download/extraction phase, so probably an input manipulation function could be good.

Nevertheless we should avoid putting dataset logic in the embeddings when feasible and in this case we won't use the object detection layer.

[Jim-Encord]

Yeah fair point. I'll try to adapt this later, inspecting your encord PR as is

[eloy-encord]

Closing this one as we updated the code on other PRs and currently allow custom target columns and configurations for HF datasets and that NIH-Chest-X-ray-dataset requires much more customisation compared to the rest of the datasets.