Skip to content

fix(asr): stop evaluate_eou returning negative false_negatives and missing - #16201

Open
Kayvan-Zahiri wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Kayvan-Zahiri:fix/eou-negative-counts
Open

fix(asr): stop evaluate_eou returning negative false_negatives and missing#16201
Kayvan-Zahiri wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
Kayvan-Zahiri:fix/eou-negative-counts

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown

evaluate_eou can return negative false_negatives and missing, which are counts.

When a prediction cuts a reference short, the reference index goes into earlycut_ids
and r_idx stays put. If a later prediction starts after that reference ends, the
while loop advances r_idx past it, so the trailing "unmatched references" block
never counts it. The unconditional -= len(earlycut_ids) still subtracts it, and both
counters go below zero.

Two references, one early cutoff followed by an exact hit:

reference  = [{"start_time": 0.0, "end_time": 2.0}, {"start_time": 5.0, "end_time": 7.0}]
prediction = [{"start_time": 0.0, "end_time": 1.0, "eou_prob": 0.9},
              {"start_time": 5.5, "end_time": 7.0, "eou_prob": 0.9}]
evaluate_eou(prediction=prediction, reference=reference, threshold=None, collar=0.1)
# false_negatives = -1, missing = -1

The fix discounts only the early cutoffs the trailing block actually counted, so a
trailing early cutoff is still discounted as before. Two regression tests added; the
first fails on main, the second guards the existing behavior.

Existing tests/collections/asr/test_asr_eou.py passes before and after.

…ssing

An early cutoff that r_idx later advanced past is recorded in earlycut_ids but is
never added to the trailing unmatched-reference count. Subtracting the full set
therefore takes both counters below zero.

Discount only the early cutoffs the trailing block actually counted.

Signed-off-by: Kayvan Zahiri <kayvanandre@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the ASR label Sep 2, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ASR community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants