Goals and Strategy

[jbteves]

I'm opening this issue so that we can discuss the goals and strategy for this analysis. I'd like for this issue to help us outline the goals and strategy. I would like to propose the below:

Goals:

• Quantify how robust 'mle' is to various random seeds
• Assemble a variety of data of various qualities and types (e.g., resting, a few headcoils, some tasks)
• Determine if 'mle' converges for data of "reasonable" quality (I expect there will be discussions on what that means)
• Determine whether afni_proc or fmri_prep affects resultant time series appreciably
• Determine to what degree the results depend on OS

Strategy:

• Assemble several datasets
• For each dataset:
  • Run afni_proc and fmriprep on all data sets
  • Run tedana with roughly 100 different random seeds
  • Subtract time series for each voxel and get the standard deviations across time series (open to ideas on summarizing differences between voxels and seeds)
  • Compare convergence to presumed data quality (based on motion parameters, number of outliers, etc.)
• Repeat on other operating systems

Apologies to @tsalo if this was discussed elsewhere or is in the code; I took a look and I see that you have some setup for fmriprep and running tedana already, but I would like to investigate the afni_proc preproc as well.

[tsalo]

I wasn't planning to look into preprocessing pipeline or data quality/acquisition parameters. Granted, that's mostly because I was limiting the scope of this analysis because I didn't have much time to dedicate to it. If you and @dowdlelt are planning to spend some time on this, then I think those can definitely fall within the scope of the project.

In terms of metrics, some additions to your own that I think would be useful are:

• Number of iterations required for convergence
  • We set maxrestart to 1 and maxit to a very large number (e.g., 100000)
  • How does this number correlate with relevant MRIQC IQMs?
• Number of restarts with consecutive seeds generally required for convergence with a reasonable maximum number of iterations.
• Rate of convergence failure (for a reasonable maximum number of iterations, like 500)
  • How does convergence failure correlate with relevant MRIQC IQMs?
• Run duration
  • How does run duration correlate with number of iterations (should be fairly linear)
• Carp-style maps of significance across random seeds
  1. Run a first-level model for the denoised data
  2. Threshold and binarize map from that model (if need be we can do this at the group level instead)
  3. Sum binarized, thresholded maps across random seeds
  4. Now we have essentially the proportion of tedana runs in which each voxel was significant!
• Identify comparable components from different runs (e.g., through correlations of component time series) and determine consistency of classification
  • Classification (accepted, ignored, or rejected)
  • Rationale for rejection or being ignored
• Variance explained

[jbteves]

Can't speak for @dowdlelt but I think it's worth doing the work to take that into account, and I suspect I have the time. These additional metrics sound good to me.

[dowdlelt]

I like those metrics, though there is an obvious difficulty with resting state data for which no model exists. Perhaps for resting state scans we could use a seed region, corresponding to a typical resting state node as the analyses method. Should get reasonable maps of a resting state network that way, and be able to generate similar voxel significance maps. Maybe a couple different seed regions...

[tsalo]

Yeah, a seed-to-voxel analysis with a common seed is a good option for doing this with resting-state data. What about one seed for each of the major canonical networks? E.g., default mode (PCC), executive control (dmPFC), and salience (anterior insula).

[tsalo]

One potential issue with group comparisons across seeds is that using the same seed for different data (e.g., different subjects) doesn't mean anything. If we compare group-level maps from one seed to the next, we have a combination of within-subject variability due to seed, between-subject variability due to the data, and between-subject variability due to seed. That's probably not the most accurate way of describing it, but I think the logic is sound.

I figure that we have three possible solutions:

1. Only look at individual subjects. This will be very difficult to interpret.
2. Ignore the issue and build our group-level maps by seed.
3. Build our group-level maps by randomly selecting across seeds.

[tsalo]

I would like to flesh out the analysis plan in a Google Doc, but before I start on that I want to ensure that we've figured out the cross-seed comparison issue. Would everyone please make sure to take a look at #3 and weigh in there?

Addendum: We also need to figure out the issue above (group comparisons when seed doesn't matter).

[dowdlelt]

Per the idea of predicting whether convergence issues can be predicted, two additional measures came to mind during the phone call:

TSNR of the data
Spatial Smoothness (whether FWHM, or something like AFNI acf values)

In addition, thiking out loud on others: sampling rate, voxel size (which would certainly relate the smoothness), type of head coil (num channels perhaps)?