neuralbench: sleep_onset crops make the crop start reveal the target on Sleep-EDF

[terezaokalova]

Description

In the eeg/sleep_onset task, AddSleepOnsetTargets with max_pre_n2_s: 1200 keeps the 1200 s before the first N2. Sleep-EDF (Kemp2000Analysis) recordings begin hours before bedtime, so every crop is exactly [n2_onset - 1200 s, n2_onset] and the target is a function of position in the crop: min(600, 1200 - seconds since crop start) reproduces it with no EEG. Anything that sees the crop start (an elapsed-time feature, a running per-recording normalisation, a reference taken from the first seconds of the crop) reads the label off the crop geometry.

How I noticed: a causal "reference to the first 60 s of the crop" feature improved test bMAE by about 6 s on Sleep-EDF; on recordings whose crop starts at lights-off (HMC, where the 1200 s margin is not available) the same feature is worth 1.3 s.

Suggestions: anchor the crop start to something independent of the label (lights-off, recording start, or a margin randomized per recording), or keep the full recording and mask windows more than 1200 s before onset in the loss and metric only.

Question for the competition: are the hidden Muse evaluation streams cut with a fixed margin before the annotated onset? If so, the same leak would become available on the leaderboard; if they run from the start of the night, it would be helpful to state that in the Track 3 guide.

Steps to reproduce

Build the sleep_onset task on kemp2000analysis with the default config (max_pre_n2_s: 1200), then for each recording compare the first segment's start time with the recording's N2 onset:

n2_onset - first_segment_start == 1200 for every Sleep-EDF recording, the first window of every crop carries target 600, and the predictor min(600, 1200 - t_since_crop_start) scores bMAE ~0 without reading any EEG.

Environment

• OS:
• Python version:
• neuralset version (pip show neuralset):
• Relevant package versions:

[hubertjb]

Thanks @terezaokalova for reporting the issue! Indeed, the fixed cropping means there is currently a direct mapping between position in the crop and the N2 onset. However, this is not an issue for the neuralbench warm up task: the model receives independent 5-s windows, and is not fed any timing/position information about the input window, and so cannot leverage a mapping position --> n2_onset.

As to whether the competition's evaluation will use a similar cropping strategy, good question - I will check with the track leaders and will include the info in the Track 3 guide.

[terezaokalova]

Sounds good, thank you! that makes sense for the warm-up task.

Also, if you end up hearing back from the track leaders, i think these would be quite helpful for the guide:

(1) whether the sealed Muse streams are cut with a fixed margin before the annotated onset or run from the start of the night
(2) some sort of confirmation that evaluation windows are presented independently (no recording order, no position information) (since that determines whether any model can use the preceding minutes of the recording at test time)