Fix ek80 calibration

[iambaim]

We have some EK80 files that triggered errors when calling get_calibration(). I'm quoting Tomasz's email below:

File "/Users/tf/PycharmProjects/pythonProject/echolab2/instruments/EK80.py", line 4202, in from_raw_data
super(ek80_calibration, self).from_raw_data(raw_data,
File "/Users/tf/PycharmProjects/pythonProject/echolab2/instruments/util/simrad_calibration.py", line 259, in from_raw_data
self.set_attribute_from_raw(raw_data, param_name,
File "/Users/tf/PycharmProjects/pythonProject/echolab2/instruments/util/simrad_calibration.py", line 371, in set_attribute_from_raw
param_data = self.get_attribute_from_raw(raw_data, param_name,
File "/Users/tf/PycharmProjects/pythonProject/echolab2/instruments/EK80.py", line 4270, in get_attribute_from_raw
new_data[idx] = param_data[match_idx][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

It seems that param_data[match_idx] can be empty at times. This fix can avoid the error, however I'm unsure if this is the correct way to do it.

Thanks!

[rhtowler]

Hi Ibrahim,

Can you confirm that this issue is from the 2021 data you are working with. Specifically the data recorded using EK80 v2.1.x?

I believe that this issue is a result of a file format change made by Simrad in a recent EK80 version. When converting FM data recorded on a system without an FM calibration applied one must look up gain and sa correction values from the gain and sa correction tables present in the configuration header. These tables are indexed by pulse duration, so one first finds the index of the pulse duration in the pulse duration table, then uses that index to extract the proper gain and sa correction.

Within the configuration header there are two pulse duration tables: pulse_duration and pulse_duration_fm. The index for gain and sa correction for CW data is obtained from the pulse_duration. You would think it would be safe to assume that the index for FM data would be obtained from pulse_duration_fm, but up until this latest version that was not correct and pyEcholab always looked up the index from pulse_duration regardless of FM or CW operation. It seems that this has changed with files recorded using EK80 version > 2.1.x. The error you are running into is a result of the lookup failing in the pulse_duration.

The latest commit has a fix that looks up the index from pulse_duration_fm if it fails to find a match in pulse_duration. I need to do additional testing, and I would like to confirm my theory re: change with v2.1.x, but I think this resolves the issue and allows for backwards compatibility with older files.

I will look into this further and comment here.

[iambaim]

Sorry for the late reply, I was on a leave earlier. This issue is not only from processing the 2021 data, but from the 2020 data from our Sandeel survey.

Where can I check the version of the EK80 data?

I'll try to run with the latest update to check if the problem has been resolved. Thank you!

[iambaim]

I can confirm that the latest commit still produces error during calibration. However, this doesn't happen anymore with the 2021 data, only with the 2020 EK80 data:

  File "CRIMAC_preprocess.py", line 217, in process_data_to_xr
    cal_obj = raw_data.get_calibration()
  File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 2422, in get_calibration
    cal_obj.from_raw_data(self)
  File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 4243, in from_raw_data
    super(ek80_calibration, self).from_raw_data(raw_data,
  File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/util/simrad_calibration.py", line 259, in from_raw_data
    self.set_attribute_from_raw(raw_data, param_name,
  File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/util/simrad_calibration.py", line 371, in set_attribute_from_raw
    param_data = self.get_attribute_from_raw(raw_data, param_name,
  File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 4359, in get_attribute_from_raw
    gain = param_data[match_idx][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

You can find the sample data in the Azure share path:
cruise_data/2020/S2020821_PEROS_3317/ACOUSTIC/EK80/EK80_RAWDATA/2020821-D20200423-T112014.raw

Thank you.

[rhtowler]

I looked into this further and realize that I need additional information from Simrad to determine the correct fix.

pulse_duration and pulse_duration_fm contain up to 5 values representing the possible system transmit pulse durations and are stored in the raw file header. One is supposed to find the index of the system's commanded pulse duration in one of these tables, then use that index to extract gain and/or sa correction from the gain and sa_correction tables. pulse_duration_fm was added in EK80 and one would assume that when running in FM mode without a broadband calibration, the index for the gain lookup would come from the pulse_duration_fm and indeed this was how I initially implemented this. But this Echoview bug fix notice states that the index for the gain lookup should come from the pulse_duration table for wideband files without a wideband calibration and not from pulse_duration_fm.

The issue here is that the files in question do not have a matching pulse duration for the commanded 2048 us in the pulse_duration table so the lookup returns empty, hence the error. The value does exist in the pulse_duration_fm table. So, are we supposed to look this up in pulse_duration_fm, going against what Echoview is doing? Or did the EK80 change how it is storing lookup table data in these later versions of EK80 and the lookup depends on the application_version?

I will attempt to get this info from Simrad, but they have been extremely reluctant to provide any information regarding their file format beyond what is published in the EK80 interface spec doc which does not include details such as this. I will also contact Echoview to see if they have any additional information on this.

My guess is that the lookup for FM data without a BB cal should come from pulse_duration_fm as I originally assumed, but I would like confirmation before implementing.

[iambaim]

Thank you @rhtowler ! We have asked Simrad to look into it. I'll include @nilsolav and @tomaszfurmanek here for future follow up on this issue.

[rhtowler]

I have not received any further guidance from Simrad but have been told that they will be releasing a reference implementation of FM to Sv conversion "soon" where this should be answered.

I am anticipating that my original implementation where gain for FM data without a wideband calibration is extracted from the pulse_duration_fm was correct and I have reimplemented that in commit 0ddb295. I will revisit this after Simrad releases the reference implementation.