Fixed nan returned for ndcg, rprec if no relevant docs retrieved #51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rieved Added tests for these cases
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I noticed that if no relevant documents are retrieved for a run then NDCG, Rprec and recip_rank all returned nan. For NDCG and Rprec I believe these should be 0 instead. I left recip_rank since it is not obvious which would be preferred here.
I wrote some tests for the fixes I made which ensure that the relevant metrics are 0 if no relevant documents were retrieved or if the documents retrieved were all unlabelled.
Note this is different from #25 since there are relevant documents they were just not retrieved during the run (or up to the depth evaluated). With regards to #25 it seems that the entries for NDCG per query do not exist if there are not relevant documents in the collection. This is not fixed here.