Found during the pre-release bug-hunt you asked for on #219.
_dcg in openagent_eval/metrics/retrieval/ndcg.py (lines ~25-29) computes the position discount as log2(i + 1) with 0-indexed i, special-casing i == 0 to 1.0. Position 2 (i == 1) therefore gets discount log2(2) = 1.0 — the same as position 1, i.e. no penalty at all. Standard DCG discounts by log2(rank + 1) with 1-indexed rank, i.e. log2(i + 2) here.
Repro (single relevant doc among 3 retrieved, k=3), metric output vs correctly-computed NDCG:
relevant at rank 1: metric NDCG=1.0000 correct NDCG=1.0000
relevant at rank 2: metric NDCG=1.0000 correct NDCG=0.6309
relevant at rank 3: metric NDCG=0.6309 correct NDCG=0.5000
NDCG should strictly decrease as the relevant doc moves down the ranking; instead ranks 1 and 2 are indistinguishable and every position is one slot too generous — the metric systematically overstates ranking quality for every consumer. Fix is the one-line discount change plus updating any tests that encode the shifted values.
Happy to PR this right away given the release timing — say the word (or I'll just open it).
Found during the pre-release bug-hunt you asked for on #219.
_dcginopenagent_eval/metrics/retrieval/ndcg.py(lines ~25-29) computes the position discount aslog2(i + 1)with 0-indexedi, special-casingi == 0to1.0. Position 2 (i == 1) therefore gets discountlog2(2) = 1.0— the same as position 1, i.e. no penalty at all. Standard DCG discounts bylog2(rank + 1)with 1-indexed rank, i.e.log2(i + 2)here.Repro (single relevant doc among 3 retrieved, k=3), metric output vs correctly-computed NDCG:
NDCG should strictly decrease as the relevant doc moves down the ranking; instead ranks 1 and 2 are indistinguishable and every position is one slot too generous — the metric systematically overstates ranking quality for every consumer. Fix is the one-line discount change plus updating any tests that encode the shifted values.
Happy to PR this right away given the release timing — say the word (or I'll just open it).