v0.9.0

Overview

This release includes a number of new features and performance enhancements, as well as fixes for bugs and warnings. We have fully switched to using the trunk-based development workflow in this release.

New features

• Add noise and attenuation functions from Ryan et. al. 2015 and echopy by @ctuguinay in #1316
• Add .bot and .idx files to EK60/EK80 parsing by @ctuguinay in #1318
• Add index aggregation option for cleaning mask functions by @ctuguinay in #1326
• Added alpha-version code for new AZFP6 format by @dash-uvic in #1323
• Use latest MRU parser version and enable ep.consolidate.add_location can be based off datagram type by @ctuguinay in #1359

Enhancements

• Support lazy-loaded EK80 broadband-complex data by @leewujung in #1311
• Allow MVBS into ep.mask.apply_mask, and add alignment check for target variable and mask by @ctuguinay in #1345
• Enhance ep.calibrate.compute_Sv scalability by enabling EchoData chunking and removing eager computation by @ctuguinay in #1331
• Use Echodata Platform and Beam_group1 variables to compute depth for EK60/EK80 by @ctuguinay in #1319
• Allow waveform mode "FM" in calibrate by @ctuguinay in #1363
• Allow external arrays in add_depth and add an universal ping time alignment function by @ctuguinay in #1369

Bug and warning fixes

• Fix duck array RuntimeWarning for EK60 ep.open_raw by @ctuguinay in #1337
• Coding: channel_mode dtype from float16 to float32 by @ctuguinay in #1350 and #1351
• Fix problems in applying compute_Sv on combined EchoData object by @leewujung in #1328
• Fix data type check for padding short complex pings by @ctuguinay in #1353
• Update xr.drop to xr.drop_vars by @ctuguinay in #1364

Documentation

• Update docs for visualization by @leewujung in #1347
• Merge data processing level changes from stable by @leewujung in #1355
• Add the Ping acoustic library to other resources by @gavinmacaulay in #1362
• Update contributing guide and README/index by @leewujung in #1356 and #1373

Others

• Remove the deprecated visualize subpackage by @leewujung in #1336
• Temporarily skip datagrams generated by EC150 by @leewujung in #1357
• Modify dask image in requirement.txt by @leewujung in #1361

New Contributors

• @dash-uvic made their first contribution in #1323

Full Changelog: v0.8.4...v0.9.0

v0.8.4

Overview

This release includes a few performance enhancements, bug fixes, and improving warnings/error messages. We are also preparing to switch to using the trunk-based development workflow in the next release to streamline new contributions.

Enhancements

• Upgrade compress_pulse to use parallelized convolution by @anantmittal in #1208
• Optimize harmonize_env_param_time by @anantmittal in #1235
• Optimize frequency differencing with dask by @anantmittal in #1198
• Optimize get_vend_cal params_power by @anujsinha3 in #1285
• apply_mask changes: Allow multi-channel masks, allow the depth dimension, and simplify fill value by @ctuguinay in #1230
• Update _calc_Sv_offset for AZFP parsing by @leewujung in #1304
• Move compute_Sv_offset to inside of AZFP parser by @ctuguinay in #1307
• Support calibration of EK80 broadband complex data from sequential pinging by @ctuguinay in #1302
• Add skipna argument to compute_MVBS/NASC to control mean behavior over NaN elements by @ctuguinay in #1269

Bug fixes and warnings

• Fix error parsing integer in frequency-differencing equation by @leewujung in #1278
• Fix time encoding to use int64 and "nanoseconds since 1970-01-01 00:00:00Z" by @ctuguinay in #1299
• Add warnings and raised errors for add_location by @ctuguinay in #1296

Others

• One time dev merge to main to facilitate switching to trunk-based workflow by @leewujung in #1297
• Engaging in Google Summer of Code 2024 program by @leewujung #1266, #1267, #1274
• Update Development roadmap by @leewujung in #1262
• Deprecating echopype.visualize by @leewujung in #1310

New Contributors

• @anantmittal made their first contribution in #1208
• @anujsinha3 made their first contribution in #1285
• @ctuguinay made their first contribution in #1230

Full Changelog: v0.8.3...v0.8.4

v0.8.3

Overview

This release includes a bug fix for changes from the previous release and a few functionality enhancements.

Enhancements

• Add parser support for EK80 MRU1 datagram (#1242)
• Add support for consolidate subpackage functions to accept both in-memory or stored datasets (#1216)
• Add test for ES60 spare field decoding issue (#1233)
• Add test for EK80 missing receiver_sampling_freq error (#1234)

Bug fixes

• Fixed reshape bug in pad_shorter_ping that was remnant from use_swap full refactoring (#1234)

v0.8.2

Overview

This release includes a number of important performance enhancements, bug fixes, and under-the-hood refactoring to improve code readability.

Enhancements

• Full refactor of the functionality to directly storing parsed data to zarr before set_groups_* (#1185, #1218, #1070)
  • This avoids memory expansion during NaN padding across channel
• Improve compute_MVBS using flox (#1124)
• Revive and improve compute_NASC using flox (#1167)
• Refactor AZFP XML parser
  • Improved XML parsing by using xml.etree.ElementTree to be consistent with other XML parsers in echopype (#1135)
  • Parse more parameters and store in the resulting EchoData objects (#1135)
  • Add partial support for AZFP multiple phase settings (#1182)
• Overhaul AZFP Environment group
  • Set mandatory variables not in data to NaN (sounds_speed_indicative, absorption_indicative) (#1226)
  • Parse and store AZFP pressure data when exist (#1189, #1226)
• Add utils.misc.depth_from_pressure function for converting pressure to depth (#1207)
• Remove unused mechanism to selectively parse some EK datagrams (#1214)
• Enhancements of EchoData.update_platform method
  • Factor out utility functions to a submodule (#1209)
  • Auto-assign timestamp for fixed-location external data using first ping_time(#1196)
• Add support for consolidate subpackage functions to accept both in-memory or stored datasets (#1216) [NOT MERGED]

Bug fixes

• Fix EchoData.to_zarr encoding (#1128)
  • Eliminate preferred chunks
  • Ensuring chunk alignment and encoding only when handling dask arrays
• Fix scaling bugs in compute_NASC (#1167)
• Fix UnicodeDecodeError for ES60 files (#1215)
• Handle missing receiver_sampling_frequency for EK80 data (#1219)

Infrastructure and refactoring

• Add support for running individual test files (#1166)
• Add module based testing (#1180)
• Update CI to barebone python, without conda (#1192)

v0.8.1

Overview

This release includes a bug fix and a few infrastructure upgrades and enhancements.

Enhancements and bug fix

• More intuitive input syntax for frequency_differencing (#1106)
• Set EK60/80 Platform and NMEA nan timestamp to first ping_time value (#1154)
  • This avoids warnings due to xarray's new behavior with nan timestamp in coordinate
• Remove automatic version conversion in open_converted (#1143)

Infrastructure

• Bump testing minimum Python version to 3.9 (#1146)
• Experiment with Python 3.11 in testing (#1148, #1156)
  • Some tests related to parsed2zarr fail due to the pinned more_itertools version in python 3.11. parsed2zarr is being overhauled in #1070 so these are deferred to the next release.
• Remove logging during testing (#1141)
• Remove NASC import causing failure (#1145)

v0.8.0

Overview

This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple EchoData objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.

Enhancement

• Overhaul the combine_echodata function (#1042, #1116)
  • No longer require spinning up dask client under the hood during combine
  • Use xr.concat directly compatible with delayed computation
  • No longer require destination path as input argument
• Overhaul EchoData.update_platform (#1060)
  • Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., time3, time4, etc)
  • Only allow updating standardized, sonar model-dependent Platform group variables

Raw-converted data format changes

• Drop the beam and ping_time dimensions for some parameters (#1056, #1083)
  • The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
• Standardize backscatter_r/i long_name in the Sonar/Beam_groupX group, and correct units (#1047)
• Standardize use of transmit_frequency_start/stop for all echosounder models (#1091)
• Move filter coefficients and decimation factor to variables in EK80 Vendor_specific group (#1044, #1046, #1105)
• Add new Provenance group combination_* attributes to combined EchoData object, mirroring the conversion_* attributes (#1113)
• Bring more consistency in the Platform group across sensors on conversion (#1058, #1061)
• Add missing mandatory variables and other data format changes (#1094, #1099, #1101, #1102, #1103, #1104, #1107, #1114)
• Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (#1117)

Packaging and infrastructure

• Remove Pandas<2 pinning (#1080)
• Pin netcdf to >1.6 and add explicit encoding (#1112)
• Update CI to prepare for python 3.11 (#1108)
• Switch github action for get-changed-files (#1118)
• Fix discrepancy on test log output (#1119)

Others

• Set and fix encoding for variable encoding (#1072)
  • Specify handling for expected string type
  • Sanitize and set encoding as part of EchoData
• Handling problematic NMEA messages when setting lat/lon in Platform group (#1067)
• Update permission check filename to be UUID to make it thread safe (#1110)
• use pulse_form for EK80 transmit type checks (#1091)
• Factor out a normalization factor from pulse compression function for flexible use (#1105)
• Add existence checking for angle parameters in consolidate.add_splitbeam_angle (#1105)
• Revise qc function to clean up reversed time (#1065)
• Update matplotlib call to register EK500 colormap (#1068)
• Temporarily remove compute_NASC for debugging and overhaul (#1136)
• Remove the deprecated preprocess subpackage (#1077, #1082)
• Remove the ZarrCombine module since it is no longer used (#1117)
• Remove outdated ui_param (#1071)

Documentation

• Add rendering for example EchoData objects for all supported echosounders to show the updated format (#1089)
• Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (#1093, #1132, #1134)
• Add discussion on z-axis vertical coordinate system and variables (#1133)
• Add bibliography page linked from inline citations (#1137)

v0.7.1

Overview

This release includes important updates to the requirements, small bug fixes, and refactoring of AZFP parser code to handle files from glider deployment and improve code coherence.

Enhancement and bug fixes

• AZFP conversion enhancements
  • Add error catch to handle AZFP files with no temperature or tilt, such as those from glider deployments (#1020)
  • Refactor AZFP parser code to improve code coherence (#1024)
  • Correct and add missing SONAR-netCDF4 variable attributes for AZFP (#1023)
• Improve impedance parameter handling for EK80 files (#1019)
  • allow impedance data type to be float
  • unify naming of impedance parameters
• Allow only one chirp generation option for EK80 broadband processing (#1019)
  • previously there was an option to use the implementation from Matlab echolab
  • starting from this version only the implementation identical to those from Lars Andersen (repo)
• Remove unneeded return self in processing level decorator wrapper-inner function for class method (#1037)

Tests and infrastructure

• Unpin xarray version and pin pandas < 2 (#981)
• allow v0.7.x in map_ep_version checks

v0.7.0

Overview

This release includes new features to interface with Echoview ECS files for computing Sv, reorganization of computing functions into new subpackages, addition of data processing level attributes to data products, and other improvements and bug fixes.

New features and major changes

• Allow using ECS for calibrating Simrad echosounders (#996, #1004)
  • This functionality is in a beta testing stage
  • Details of implementation may change and bugs are possible
  • Expand ECS parser to accept frequency-dependent values in EK80 ECS files
• Overhaul env_params to ensure correct intake for calibration (#985)
  • Now allows using env_params entries that are xr.DataArrays
• Move functions previously in preprocess subpackage to new subpackages (#993). Calling these functions from preprocess is deprecated and will be removed in v0.7.1.
  • clean: remove_noise, estimate_noise
  • commongrid: compute_MVBS, compute_MVBS_index_binning
• Add commongrid.compute_NASC (#1005)
  • The current implementation uses brute force looping for mean Sv computation, this will be refactored and optimized together with other functions requiring the same pattern in an upcoming release
• Add global attributes for data processing levels (#1001).
  • This functionality is in a beta testing stage
  • See data processing level specifications for functions and conditions under which such attributes are added
• Expand mask.apply_mask to handle multi-channel Sv datasets (#1010)
• Standardize sonar metadata for EK80 data (#992)
  • sonar_serial_number is now an empty global attribute, no longer a variable, as in the EK60 case
  • transducer_name, transducer_serial_number, transceiver_serial_number based on parser parameter transducer_name, transducer_serial_number, and serial_number, respectively

Bug fixes

• Fix scaling bug for beamwidth_alongship and beamwidth_athwartship from CW-based parameters to values corresponding to center frequency of broadband transmit signals (#998)

Tests and infrastructure

• Add more comprehensive tests for add_location (#1000)
• Add test for splitbeam angle ek80_CW_power case (#994)
• Add unit and integration tests for env_params intake for calibration (#985)
• Exclude test_data folder in codespell pre-commit hook (#1016)

v0.6.4.1

Overview

This is a minor release that includes a small bug fix and an enhancement, which removes excessive warnings due to changes in v0.6.4.

Bug fix

• Fix a bug that prevented passing in env_params for EK60 calibration (#987)

Enhancement

• Handling non-positive values that causes log10 warnings on EK calibration (#986)

v0.6.4

Overview

This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.

New features

• Allow passing in cal_params as a dictionary for narrowband and broadband calibration (#955)
• Add default chunk encoding for zarr output (#939)
• Add add_splitbeam_angle function to the consolidate subpackage (#916, #971)
• Add a new mask subpackage
  • Add apply_mask function to the mask subpackage (#912)
  • Add frequency-differencing function to the mask subpackage (#901)
• Allow selection of a subset of channels when combining multiple echodata objects (#892)
  • This is done via the added channel_selection input argument to combine_echodata
• Add default consolidated flag for echodata.to_zarr (#855)
• Generalize and improve efficiency of compute_MVBS (#878)
  • Allow echo_range that vary with ping_time
  • Allow Sv data that are dask arrays
• Add .nbytes to obtain that data size of an echodata object (#874)
• Add new data variables from the raw data files to facilitate EK80 calibration (#944)
  • transceiver type, transmit impedance, receive impedance, receiver sampling frequency

Enhancements and other changes

• Unify the order of dimensions of echo_range across all sonar models (#968)
• Create and use the default echopype home directory (#896, #954)
  • This default directory is at ~/.echopype
• Refactor the calibrate subpackage (#904)
  • routines for env_params and cal_params intake and range computation are now in different modules rather than methods in each classes
  • Tidy up cal_params related routines (#953)
  • Revise env_params related routines (#952)
• Improving pulse compressed broadband Sv computation and echo range computation (#944)
  • The resulting values are tested against pyEcholab and Echoview outputs
  • Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
• Change input argument names for open_raw (#962)
  • Change offload_to_zarr to use_swap
  • Change max_zarr_mb to max_mb

Bug fixes

• Handling of provenance attributes in apply-mask and add-depth, especially for testing (#930)
• Fix bugs in combining multiple echodata objects (the combine_echodata function)
  • Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (#913)
• Fix bugs in env_params intake for calibration (#952)

Infrastructure

• Update docker build for arm64 mac silicon chips (#964)
• Fix windows github actions workflow (#947)
• Remove mamba dependency (#946)
• Fix pre-commit ci and update RTD config for jupyter-book (#934)