documentation of nscores

[QikeLi]

Hi Edwin,

I really reading your NIPS papers and am trying to apply IBCC to my work. A problem I encountered while applying your ibcc.py was that I was not able to figure out what nscores is. Can the value of nscores be different from the value of nclasses? Is nscores the number of possible outputs from classifier k, and is nclasses the number of classes? Whenever I specified a different value of nscores from the value of nclasses, I would get an error.

Thanks,

[edwinrobots]

Hi Qike,

Apologies for slow reply -- I am just back from vacation.

Glad to hear you are interested in using IBCC! The value of nscores can be different to nclasses:

• nclasses is the number of true label values that you would like to infer
• nscores is the number of answers that the base classifiers/annotators/crowdworkers can give. For example, you may have a binary classification task, and the workers may choose class 0, class 1, or give an answer "I don't know". In this case you get nscores=3.

You can encode the relationship between the true class labels and the worker labels using the hyperparameter alpha0. This is a matrix: rows correspond to true class labels; columns to the worker labels. Typically, you would set it to have a value of 1 for the cases corresponding to workers making mistakes (i.e. where true class != worker). For entries corresponding to true class=worker label you typically set higher values (e.g. 5) to favour workers giving the correct answers.

I hope that helps... Let me know how it goes! What application are you testing at the moment?