fix(peagle): validate the packed document lengths instead of reusing them per row - #70
Conversation
…them per row seq_lengths is a flat list of document lengths for ONE packed sequence, which is what base_trainer builds for P-EAGLE. The forward loops over the batch but hands the whole tensor to every row, so a multi-row batch would silently give every row row zero's document layout. base_trainer always produces a single flat row today, so this is latent, but the loop is written for batch_size > 1 and the fallback branch does index per row. Nothing checked that the lengths cover the sequence either. The COD mask gives any tail past sum(seq_lengths) document id -1, and its is_not_padding term then makes those queries attend to nothing at all, so a short seq_lengths drafts on garbage rather than failing. Reject both, and rename the local to document_lengths since that is what the mask consumes. Signed-off-by: khazic <khazzz1c@gmail.com>
|
The
#68 and #70 are the clearest: neither touches the dspark or the dflash code path, yet those are the jobs that failed. Within a single run the three example jobs are serialized (on #63 they ran 11:25:05 to 11:42:21, 11:42:46 to 12:01:01, 12:01:27 to 12:19:55), so the contention comes from runs of different PRs overlapping. The failures cluster in the window where three PRs had example jobs in flight at once, and every job that started after that queue drained passed. The failed jobs also died in about 12 minutes against roughly 17 for a successful one, consistent with dying at engine init rather than during real work.
Could a maintainer re-run the failed job? I do not have the permission to ( |
Problem
seq_lengthsis a flat list of document lengths for one packed sequence.base_trainerbuilds it that way:The P-EAGLE forward loops over the batch but hands the whole tensor to every row:
Two gaps:
base_traineralways produces a single flat row today, so this is latent, but the loop is written forbatch_size > 1and the fallback branch does index per row.document_idsfromseq_lengthsand pads the remainder with-1:so any tail past
sum(seq_lengths)becomes a query that attends to nothing at all. That is not an error, it is a silently degenerate attention row that still flows into the loss.Fix
Reject both, and rename the local to
document_lengths, which is what the mask consumes.The length check costs one comparison per forward on a path that already synchronizes several times (
generate_cod_sample_indicesandcreate_block_maskboth do).Validation
Ran a before/after repro on both
mainand this branch, feeding the training model each malformed shape.Before/after repro output
Tests
Two new tests in
tests/integration/test_peagle_backend_contract.py:test_peagle_rejects_packed_lengths_for_a_multi_row_batchtest_peagle_rejects_packed_lengths_that_do_not_cover_the_sequenceFull CPU suite (
tests/integration tests/compat tests/config tests/examples) run on both sides:Test suite before/after
The same 9 tests fail on
mainand on this branch. They need optional dependencies (VeOmni, the NPU vLLM stack) that are absent in this environment, so they are pre-existing and unrelated. The+2on this branch are the new tests above.