Description

gliderdac is a code set that takes data from a Teledyne Webb Research (TWR) Slocum Glider and prepares it for submission to the IOOS National Glider Data Assembly Center (NGDAC or DAC). Submission to the DAC requires processing the glider data into NetCDF profiles (a single dive or climb) from the glider's segment data (multiple dives and climbs between surface moments). So the gliderdac code takes in segment data files (ASCII processed from Slocum binary data), does some processing, and splits a segment into profiles and writes each profile to its own NetCDF file and includes all of the relevant CF compliant metadata (controlled by configuration files input to the code)

The main goal

This description is a higher level description of the code's operation. See the How to use gliderdac page for setting a run up and starting a run.

The main code call is from the ooidba_to_ngdac_profile_nc.py inputs are configuration JSON files that describe the translation of Slocum variables/sensors to DAC NetCDF variables and the raw data Slocum ASCII files.

1. The code first takes a list of the ASCII files (we'll call them dba files for short from here on) and tries to sort them in chronological order using file name.
2. using the extra arguments or defaults and configuration JSON files it creates a NetCDF writing object with variable definitions ready to go. If this deployment has been run once before (in the case of an ongoing deployment where more data accumulates), it will check a saved status.json file in the configuration directory to keep track of profile_id and created profiles.
3. It provides minimal checks that the variable definitions fall in line with DAC specifications
4. The variable definitions are checked to see if any processing corrections are to be made by recalculating any variables from their more raw form (e.g. chlorophyll from raw counts and calibration coefficients). If there are any, it stores the recalculation inputs. At this time, re-calculation can only be done for Chlorophyll, PAR, and Oxygen Concentration
5. For each dba file input, the file is checked for goodness, which is to see that there is any underwater time, any science data at all to process (determined by science variables listed in configuration.py), that the file isn't one of the Slocum glider missions that are standby missions or status checking missions (status.mi, lastgasp.mi, or initial.mi).
6. Processing begins by removing any initialization lines from the science data where it is all zeros and if any science variables are intended to be in the data, but are missing, will add in a variable of that name full of NaNs
7. The CTD pressure is converted it from bars to decibars and creating a depth from the GSW pressure to depth conversion
8. latitude and longitude are processed to decimal degrees and interpolated to a linear track between surface GPS position fixes.
9. Pitch and Roll are processed from radians into degrees
10. CTD conductivity, temperature, and pressure are processed into Salinity and Density data using the GSW toolbox
11. If any corrections are needed for oxygen, they are mode, and then Oxygen is processed to umol/kg and adjusted for pressure and salinity.
12. if any corrections are needed for Chlorophyll, they are made
13. if any corrections are needed for PAR, they are made.
14. Backscatter is processed.
15. The Depth Averaged Velocity (DAV) is attempted to be retrieved from up to 2 segment files ahead from the Slocum data variables m_final_water_vx/vy, if they are not found, the variable is gathered from m_initial_water_vx/vy if it exists or m_water_vx/vy if it exists.
    • during this stage, the coordinates associated with the DAV are gathered, the profile's mid-segment latitude/longitude is used for lat/lon_uv and the mid-segment time for time_uv.
16. Next is the major stage where the data from a segment is split into profiles based on inflection points in the depth record. Then profiles are refined to exclude false turn arounds and wiggles at a constant depth before turning. This does not however currently work to determine if a section is a hover or not, this only works for a saw-tooth diving and climbing glider.
17. Profiles are then filtered out based on some criteria (criteria are set in profile_filters.py and configuration.py) such as if no science data exists in the profile (determined as before in configuration.py), the profile is not long enough in time or data measured, and several different ways to determine if a profile was not meant to be a profile but has data wrapped around from another profile (like if sampling on the dive only and a few data points are wrapped into the beginning and end of a climb, but the majority is blank. The latter are determined based on data to dive ratios that can be changed.
18. Once good profiles remain, for each one, data is reduced to timestamps that only science data exists for (according to the list of science variables in configuration.py)
19. For each profile, the netCDF writer determines a lot for the actual writing of the NetCDF file and then creates and writes them. This is first done to a temporary file in a temporary directory created in the directory where the NetCDF profiles are to be written so that if there is a crash of the program, a corrupted file does not exist. Once completely written, though the file is moved to the correct directory and filename.
20. For each successful NetCDF written, the processed segment file and created NetCDF profiles are recorded in a newly created status.json. Profile_id if it is meant to be a sequential number is kept in status.json too.
21. loop until done for profiles and for segment dba files.