ENH: Spectrogram viewer

[alexrockhill]

Describe the problem

What would you think about implementing something like this:

https://www.persyst.com/technology/ieeg-imaging-analysis/

Describe your solution

You would probably want to pass frequencies and compute spectrograms on the fly because it would be too much to store in memory

Describe possible alternatives

You could precompute and decimate. Maybe it would make sense to have both as options.

Additional context

Any idea how hard this would be? I was thinking I might try and do this in a couple days but if it's going to take a long time maybe it would be better as a GSoC or something.

[larsoner]

It is a subgoal listed in https://github.com/mne-tools/mne-python/wiki/GSoC-Ideas#1a-improve-raw-data-browsing-pyqtgraph . @nmarkowitz is going to be doing GSoC to improve mne-qt-browser (yay!) but I don't think this was in the proposal so someone could probably work on it separately.

[alexrockhill]

Awesome! I'd be up for giving it a shot. Do you have an idea for the parameter updating interface and what it would look like? To my knowledge that would be a new menu.

[larsoner]

We currently have the ability to switch between individual traces and butterfly mode (traces overlaid basically). So I think a reasonable option would be a drop-down that is Traces / Butterfly / Spectrogram or similar.

[alexrockhill]

Sure but then would that spawn a new menu for changing nfft etc. or use something existing?

[larsoner]

It could un-hide or add some toolbar elements for example. Or maybe we want a separate menu. These things we'd need to discuss and converge on, probably easiest to do it with a video meeting at some point. Lots of options / considerations / possible opinions on UI design

[nmarkowitz]

My thinking/plan for this is roughly:

• Run a bunch of analyses/preprocessing steps you'd like visualized (like time-frequency or change of reference)
• pass all those datastructures into a call to databrowser
• switch between those with a dropdown selection

My thinking for this approach is that it allows for more complex analyses, like output of a neural network, to be visualized in conjunction with other data. So, hopefully by the end of the summer, this could be incorporated if additional processing is done before databrowser is called

[alexrockhill]

That sounds fantastic! But also maybe a different but related project to this. I think converting to a pretty standard spectral representation would be nice as a more standard alternate view. Maybe given that info, we ditch the extra menus and only allow changes using a spect_kwargs or such on initialization.

[drammock]

On first glance, I like the idea of channel viz choices being callables behind the scenes. It means you lose the ability to do things like adjust NFFT on the fly without leaving the GUI, but it also simplifies the GUI by not spawning new dropdowns or sliders for every new viz type that is parametrized.

@alexrockhill is adjusting NFFT on the fly necessary in an epilepsy diagnosis workflow? Or is is just "nice to have"?

[alexrockhill]

I think just nice to have and I'll have a gui that launches the browser gui so parameters could be set in a menu there. Closing and reopening isn't ideal but I think it's a good start.

[nmarkowitz]

It would be nice but also a problem is that there's essentially limitless types of preproc/analysis to view

• FFT
• ICA
• rereferencing scheme (a big one in epilepsy diagnosis workflow)
• ML/NN model output
• Filtering

Each having loads of different settings and many more possible outputs to visualize. To make this at all practical it would have to be a small subset of possible outputs. Also a question is to what extent would we want this? Maybe take a look at this databrowser (link below) for thoughts

https://engineering.jhu.edu/nsa/epviz/