v0.5.0 (Major API update)

Overview

This release includes:

• major API updates to provide a more coherent data access pattern
• restructuring of subpackages and classes to allow better maintenance and future expansion
• reorganization of documentation, which also documents the API changes
• overhaul and improvements of CI, including removing the use of Git LFS to store test data
• new features
• bug fixes

API updates

The existing API for converting files from raw instrument formats to a standardized format, and for calibrating data and performing operations such as binned averages and noise removal has been updated.

The new API uses a new EchoData object to encapsulate all data and metadata related to/parsed from a raw instrument data file. Beyond the calibration of backscatter quantities, other processing functions follow a consistent form to take an xarray Dataset as input argument and returns another xarray Dataset as output.

The major changes include:

• change from an object-oriented method calls to functional calls for file conversion (using the new convert subpackage), and deprecate the previous Convert class for handling file parsing and conversion
• deprecate the previous Process class, which use object-oriented method calls for performing both calibration and data processing
• separate out calibration functions to a new calibrate subpackage
• separate out noise removal and data reduction functions to a new preprocess subpackage
• create a new EchoData object class that encapsulates all raw data and metadata from instrument data files, regardless of whether the data is being parsed directly from the raw binary instrument files (returned by the new function open_raw) or being read from an already converted file (returned by the new function open_converted)

See documentation for detailed usage of the new API.

Subpackage and class restructuring

The subpackages and classes were restructured to improve modularity that will help will future expansion and maintenance. The major restructuring includes:
("SONAR" below is used to indicate the sonar model, such as EK60, EK80 or AZFP)

• consolidate overlapping EK60/EK80 components, deprecate the previous Convert classes that handled file parsing and serialization, and revise new ParseSONAR and SetGroupsSONAR classes for file parsing and serialization
• consolidate all calibration-related components to a new calibrate submodule, which uses CalibrateSONAR classes under the hood
• consolidate all preprocessing functions into a a new preprocess submodule, which will be later expanded to include other functions with similar use in a workflow

CI overhaul and improvements

• Added github workflows for testing, building test docker images, and publishing directly to PyPI
• Deprecated usage of Travis CI
• Test run is now selective on Github, to run tests only on changed/added files. Or run all locally with run-test.py script. (#280, #302)

Documentation reorganization and updates

• Re-organization of pages with better grouping
• Added What's New page
• Added Contributing to echopype page
• Overhaul API reference page

New features

• Add interfacing capability to read from and write to cloud object storage directly. (#216, #240)
• Add support to convert .ad2cp files generated by Nortek's Signature series ADCP (#326)
• Allow environmental and calibration parameters to be optionally used in calibration in place of the values stored in data file
• Mean volume backscattering strength (MVBS) can now be computed based on actual time interval (specified in seconds) and range (specified in meters) (#54)
• Add NMEA message type as a data variable in the Platform group (#232), which allows users to freely select the suitable ones depending on use

Bug fixes

• Fix EK80 config XML parsing problem for files containing either PulseDuration or PulseLength (#305)
• Fix time encoding discrepancy in AZFP conversion (#328)
• Fix problematic automatic encoding of AZFP frequency (previously as int) to float64 (#309)
• Overhaul EK80 pulse compressed calibration (current implementation remaining in beta, see #308)

Patches and enhancements to file conversion

This minor release includes the following changes:

Bug fixes

• Fix bug in top level .nc output when combining multiple AZPF .01A files
• Correct time stamp for .raw MRU data to be from the MRU datagram, instead of those from the RAW3 datagrams (although they are identical from the test files we have).
• Remove unused parameter sa_correction from broadband .raw files
• Make sure import statement works on Google colab

Enhancements

• Parse Simrad EK80 config XML correctly for data generated by WBAT and WBT Mini, and those involving the 2-in-1 "combi" transducer
• Parse Simrad .raw files with NME1 datagram, such as files generated by the Simrad EA640 echosounder
• Handle missing or partially valid GPS data in .raw files by padding with NaN
• Handle missing MRU data in .raw files by padding with NaN
• Parse .raw filename with postfix beyond HHMMSS
• Allow export EK80 XML configuration datagram as a separate XML file

Notes

To increase maintenance efficiency and code readability we are refactoring the convert and process modules. Some usage of these modules will change in the next major release.

Add EK80 conversion, rename subpackage model to process

New features

• Add EK80 support:
  • File conversion from EK80 .raw files to netCDF and zarr formats
  • "Simple" calibration to frequency-average Sv based on pulse compression output is implemented but needs to be thoroughly tested.
• Rename subpackage echopype.model to echopype.process
  • The new name better describes the subpackage's function to process data for further analysis
  • Also rename class EchoData to Process to mirror the structure in Convert better.
  • Importing using the old names will be deprecated in the next release.
• Overhaul converting multiple files with combine_opt=True
  • If target format is netCDF, temporary files will be created and finally combined to a single netCDF. This is due to current restriction that xarray does not allow simply appending new data to an existing file.
  • If target format is zarr, data in each file are unpacked and appended to the same output file.
• Allow reading Zarr into Process in addition to netCDF: thanks @lsetiawan!
• Add a logo!

Bug fixes

• Fix bugs in slicing NMEA group data based on the same time base when range_bin is changed

Bug fix for remove noise and testing env update

This minor release includes the following:

• fix the bug in remove_noise function for AZFP data (issue #110)
• update testing to use both python 3.7 and 3.8
• include how to pull test files using git LFS in documentation
• remove a stalled command line file converter that won't be further developed

Bug fixes and calibration parameter updates

This minor release includes a few bug fixes:

• fix zarr saving error in the Provenance group in the previous release
• fix bugs in updating environmental parameters (used for calibration) for EK60 and AZFP
• allow users to update calibration parameters in EK60
• update notebook EK60_demo_OOI.ipynb to use new raw data archive folder structure

Add more convert options, accommodate range differences

This release contains updates to the convert and model module:

• Correct file parsing and storage for split-beam angle.
• Allow user-defined environmental variables to be used in calibration.
• Accommodate EK60 setting variabilities in the range_bin dimension.
  • The raw file will be split into multiple .nc files if the length of range_bin changes in the middle.
  • For cases when different frequency has different length of range_bin, the shorter channels are padded with NaN.
• Allow combining multiple files during unpacking.
• Options to:
  • overwrite previously converted files.
  • specify output directory and filename postfix for .calibration() and .get_MVBS() operations.
• Add support to convert to .zarr file. Note processing functionalities have not been fully supported.

Add support for AZFP with uniform interface

This release includes major changes to the package structure and functionality improvements:

• Added support for handling .01A data files from the AZFP echosounder.
• Changed the module structure to enable a uniform interface to work with EK60 and AZFP data.
• Improvements to EK60 components:
  • Included NMEA datagram unpacking in file conversion routine.
  • Added handling for pings with different range setting within the same data file. This feature is experimental and the default behavior will change in future release.

Updated versioning setup

Minor update:

• use versioneer for auto-versioning

Added EK60 data model and methods

Merge pull request #31 from leewujung/master

Added EK60 model class and methods