Update Prairie View XML Framerate

[jmdelahanty]

It's been discovered through the use of an Oscilloscope as discussed here that the framerates Prairie View reports in its software and, therefore, API is incorrect. This means that any framerate used in Suite2p or SLEAPy faces will be using an incorrect framerate and things may not align correctly when plotting/analyzing data.

Bryan offered some awk code for parsing the XML from Prairie View for a given file:

!#/usr/bin/env bash

j=0; k=0; x=0; for i in $(grep absoluteTime *.xml | awk -FabsoluteTime=\" '{print $2}' | awk -F\" '{print $1}'); do echo -n "@${i}: "; j=$i; x=$(echo "scale=9; $j-$k" | bc); k=$i; echo $x; done  | awk -F:" " '{print $2}' | sort -n | uniq -c

I'm not particularly familiar with how exactly this bash script works as I'm very much an awk novice. The basic idea I think is:

• Set variables for timestamps that are gathered from the XML file, pipe that into awk
• Grab all absolute time flags from the XML through splitting absolute time on the second value found
• Pipe that output for the first value that's found
• Echo to a new line each value and store as a variable
• Echo that result, no idea what scale=9 does and subtract next value from current value
• Don't know what the pipe through bc does
• Sort by name
• Give unique values

This should be made into a script that can iterate over different directories either in Python or MATLAB probably since not too many people in lab are familiar with using awk other than Bryan I think...

Given that we can't trust that Bruker's software notes the correct framerate in their software/API and additionally since it's not known how long this incorrect value has been given from their software, we should go probably do the following for all our 2-photon recordings from the older Ultima Investigator:

Verify Framerates in Environment against Framerates from Timestamps

• Verify the framerate reported in the environment file matches the true framerate from the relative framerate timestamps
• If it's different, update the framerate in the environment file to the correct value

[weiglszonja]

Hey @jmdelahanty!
I found your issue while double-checking the frame rates that are reported in the Bruker XML files.

An example for reference:

<?xml version="1.0" encoding="utf-8"?>
<PVScan version="5.6.64.400" date="11/3/2022 11:20:34 AM" notes="">
  <PVStateShard>
    <PVStateValue key="framePeriod" value="0.032950206" />
  </PVStateShard>
</PVScan>

This would mean a rate of ~30.35, which is usually higher than what I get when calculating the "true" frame rate from the relative (or absolute) timestamps of the frames.

I was wondering whether you made the same conclusion here.
I'm looking at the other issues in the meantime, thank you for sharing these publicly, they are very useful!

[jmdelahanty]

Hello! I'm so glad you're finding some of this useful! I think a fair bit of it might be lab specific, but I hope there's at least a few parts that are helpful. It's one of my favorite things about OSS!

This is indeed what we're finding with versions past 5.5.64.500. I haven't systematically gone through all the recordings we have, but it's a problem. I've notified Bruker about this but thus far the response is that since the relative timestamps are accurate we still have the needed information. This is unfortunate since other processing/best practices relies upon having accurate framerates. I'm not sure if other labs/orgs are aware of it.

The frame period/frame rate is liable to change very slightly from recording to recording but appears to be precise within recordings. It's apparently due to "weirdness in how scanning is done over the 8kHz line".

There's another repo in this lab's org that goes over things related to Bruker's data/information called bruker_control but I don't know if any of that would be useful to you.

If there's other ways I can help let me know! Could also be helpful to check out the nwb_utils.py script that @chrisroat and @RyanLy helped me make. That's meant to grab any metadata fields that were relevant to NWB files and populate them with information.