diff --git a/.gitignore b/.gitignore index c3eeec8..11b5c4b 100644 --- a/.gitignore +++ b/.gitignore @@ -6,9 +6,7 @@ agate/draft/* # personal settings files kept locally only agate/settings/secret/* !agate/settings/secret/README-secret.md -# personal workflow files kept locally only -agate/workflows/* -!agate/workflows/README-workflow.md + # CONFIG FILES # diff --git a/README.md b/README.md index aa2fc8d..86d7304 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,24 @@ # Acoustic Glider Analysis Tools and Environment *Last Update: 25 May 2023* +src="README_files/figure-gfm/fa-icon-bcbd573b0c4bf742a1242819bebaac28.svg" +style="width:0.88em;height:1em" /> *Last Update: 05 Aug 2024* -A collection of MATLAB-based tools to be used for piloting acoustic +A collection of MATLAB-based tools for piloting passive acoustic gliders, processing glider positional and environmental data, and -analyzing glider-collected acoustic data for surveys of marine mammals. +analyzing glider-collected passive acoustic data, particularly for +surveys of marine mammals. -This package is very much under development, but my hope is that I will -maintain a [stable +This package is very much under development and is hosted and version +controlled on [GitHub](https://github.com/sfregosi/agate-public). My +hope is that I will maintain a [stable release](https://github.com/sfregosi/agate-public/releases) and then for -those interested in the latest functionality, the repository can be -cloned. For those interested in contributing to the package, I suggest -creating a fork and using pull requests to contribute. +those interested in the latest functionality, the GitHub repository can +be cloned. For those interested in to the package, I suggest creating a +fork and using pull requests to contribute. See the [How to +contribute](https://sfregosi.github.io/agate-public/contribute.html) +page for more detail. Please contact me if you have any questions, +feedback, or suggestions! This code was either developed or most recently updated and tested with MATLAB version 2022b, but has undergone some testing with 2020b. @@ -26,27 +31,6 @@ MATLAB version 2022b, but has undergone some testing with 2020b. *(the documentation is also under construction and I welcome any and all feedback!)* -## Background - -These tools were initially developed for use with passive acoustic -glider surveys conducted by the OSU/NOAA CIMERS Bioacoustics Lab -bioacoustics.us and as part of my PhD. Initial -development was for surveys using Seaglider platforms and either the -Wideband Intelligent Signal Processor and Recorder (WISPR) or PMAR-XL -recording systems. - -Through NOAA’s Uncrewed Systems Initiative (UxS) we received funding to -develop and improve these tools into a more broadly applicable and -user-friendly tool box that could be used by all interested in -conducting glider-based passive acoustic surveys for marine mammals. - -Please contact me if you have any questions, feedback, or suggestions! - -selene \[dot\] fregosi \[at\] noaa.gov or -Report an issue on GitHub - ## Disclaimer The scientific results and conclusions, as well as any views or diff --git a/README.rmd b/README.rmd index 9cc0e3a..ba6400f 100644 --- a/README.rmd +++ b/README.rmd @@ -19,7 +19,7 @@ library(fontawesome) A collection of MATLAB-based tools for piloting passive acoustic gliders, processing glider positional and environmental data, and analyzing glider-collected passive acoustic data, particularly for surveys of marine mammals. -This package is very much under development, but my hope is that I will maintain a [stable release](https://github.com/sfregosi/agate-public/releases) and then for those interested in the latest functionality, the repository can be cloned. For those interested in contributing to the package, I suggest creating a fork and using pull requests to contribute. +This package is very much under development and is hosted and version controlled on [GitHub](https://github.com/sfregosi/agate-public). My hope is that I will maintain a [stable release](https://github.com/sfregosi/agate-public/releases) and then for those interested in the latest functionality, the GitHub repository can be cloned. For those interested in to the package, I suggest creating a fork and using pull requests to contribute. See the [How to contribute](https://sfregosi.github.io/agate-public/contribute.html) page for more detail. Please contact me if you have any questions, feedback, or suggestions! This code was either developed or most recently updated and tested with MATLAB version 2022b, but has undergone some testing with 2020b. @@ -27,17 +27,6 @@ This code was either developed or most recently updated and tested with MATLAB v *(the documentation is also under construction and I welcome any and all feedback!)* -## Background - -These tools were initially developed for use with passive acoustic glider surveys conducted by the OSU/NOAA CIMERS Bioacoustics Lab [bioacoustics.us](https://bioacoustics.us){index='_blank'} and as part of my PhD. Initial development was for surveys using Seaglider platforms and either the Wideband Intelligent Signal Processor and Recorder (WISPR) or PMAR-XL recording systems. - -Through NOAA's Uncrewed Systems Initiative (UxS) we received funding to develop and improve these tools into a more broadly applicable and user-friendly tool box that could be used by all interested in conducting glider-based passive acoustic surveys for marine mammals. - -Please contact me if you have any questions, feedback, or suggestions! - -selene [dot] fregosi [at] noaa.gov or [Report an issue on GitHub](https://github.com/sfregosi/agate-public/issues/new){target='_blank'} - - ## Disclaimer The scientific results and conclusions, as well as any views or opinions expressed herein, are those of the author(s) and do not necessarily reflect the views of NOAA or the Department of Commerce. diff --git a/README_files/figure-gfm/fa-icon-bcbd573b0c4bf742a1242819bebaac28.svg b/README_files/figure-gfm/fa-icon-bcbd573b0c4bf742a1242819bebaac28.svg new file mode 100644 index 0000000..8430a8b --- /dev/null +++ b/README_files/figure-gfm/fa-icon-bcbd573b0c4bf742a1242819bebaac28.svg @@ -0,0 +1,4 @@ + + + + diff --git a/agate/agate.m b/agate/agate.m index 49b6c96..70e770a 100644 --- a/agate/agate.m +++ b/agate/agate.m @@ -1,4 +1,4 @@ -function agate(missionCnf) +function CONFIG = agate(missionCnf) %AGATE Initialize a new session of agate % % Syntax: @@ -14,7 +14,8 @@ function agate(missionCnf) % particular mission e.g., 'sg639_MHI_Apr2023.cnf' % % Outputs: -% No workspace outputs. Generates a global CONFIG variable +% CONFIG [struct] containing all the user-set configurations such as +% paths, basestation login info, etc % % Examples: % agate agate_sgXXX_Location_MonYear_config.cnf @@ -25,19 +26,18 @@ function agate(missionCnf) % S. Fregosi % % FirstVersion: 06 April 2023 -% Updated: 09 March 2024 +% Updated: 06 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -clear global CONFIG; % clear out old globals +clear CONFIG; warning off % this is turned off for plotting messages -global CONFIG -% CONFIG = struct; +CONFIG = struct; -CONFIG.ver = '0.1.20240309 https://github.com/sfregosi/agate-public'; -fprintf(' agate version %s\n\n', CONFIG.ver) +CONFIG.ver = '0.1.20240806 https://github.com/sfregosi/agate-public'; +fprintf(' agate version %s\n', CONFIG.ver) if nargin < 1 CONFIG.missionCnf = []; @@ -48,18 +48,20 @@ function agate(missionCnf) % get matlab version for differences and backwards compatibility CONFIG.mver = version; -checkPath; +CONFIG = checkPath(CONFIG); -setCONFIG(CONFIG.missionCnf); +CONFIG = setCONFIG(CONFIG); + +fprintf(' loaded config file %s\n\n', CONFIG.missionCnf) end -function checkPath +function CONFIG = checkPath(CONFIG) %CHECKPATH Check the necessasry folders are there and on the path % % Syntax: -% CHECKPATH +% CONFIG = CHECKPATH(CONFIG) % % Description: % Called from the agate initialization. Sets up the necessary paths @@ -67,13 +69,15 @@ function agate(missionCnf) % are present and on the path. If not, it makes them. % % Inputs: -% none +% CONFIG [struct] containing all the user-set configurations such as +% paths, basestation login info, etc % % Outputs: -% none +% CONFIG [struct] containing all the user-set configurations such as +% paths, basestation login info, etc % % Examples: -% checkPath +% CONFIG = checkPath(CONFIG); % % See also % agate @@ -86,13 +90,11 @@ function agate(missionCnf) % https://github.com/MarineBioAcousticsRC/Triton/ % % FirstVersion: 06 April 2023 -% Updated: 04 March 2024 +% Updated: 06 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -global CONFIG - % root directory CONFIG.path.agate = fileparts(which('agate')); addpath(CONFIG.path.agate); % no subdirs diff --git a/agate/archive/workflow_createTargetsFileFromKML.m b/agate/archive/workflow_createTargetsFileFromKML.m new file mode 100644 index 0000000..115b519 --- /dev/null +++ b/agate/archive/workflow_createTargetsFileFromKML.m @@ -0,0 +1,113 @@ +% workflow to get a Google Earth created proposed trackline into a targets +% file and then into a 5-km spacing schedule to share with the Navy + +% 1 - create track manually in Google Earth, using input from collaborators +% and sponsor. Save as .kml when finalized. + +% kmlFile = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\UxS_PIFSC_twoDeployments.kml'; +% kmlFile = 'C:\Users\Selene.Fregosi\Desktop\sg680_HI_Apr2022\Leeward Glider_revised_20220503.kml'; +% kmlFile = 'C:\Users\Selene.Fregosi\Desktop\sg639_HI_Apr2022\Windward Glider_revised_20220509.kml'; +% kmlFile = 'C:\Users\Selene.Fregosi\Desktop\sg680_HI_Apr2022\Leeward Glider_revised_20220524.kml'; +% kmlFile = 'C:\Users\Selene.Fregosi\Desktop\sg639_HI_Apr2022\Windward Glider_revised_20220524.kml'; +[kmlFileName, kmlPath] = uigetfile('*.kml', 'Select .kml track'); +kmlFile = fullfile(kmlPath, kmlFileName); +[~, kmlName, kmlExt] = fileparts(kmlFileName); + +% kml saves the vertices of lines as decimal degrees in a code snippet like +% % this: +% +% Leeward Glider +% #inline13 +% +% 1 +% +% -159.3180977061045,21.93379120241162,0 -159.3915294840902,21.50741673680377,0 -158.8809373915385,21.73528994893612,0 -158.9763412128043,21.27910694076397,0 -158.4324636016379,21.47030430788609,0 -158.5300178504395,20.97351624005566,0 -158.0118868380484,21.04165660408255,0 -157.9015894020643,20.52832280966129,0 -157.3568488495222,20.82859768139042,0 -157.5622783640389,20.30084306318159,0 -157.0340176577208,20.63583528150905,0 -157.2485263069477,20.08415381579547,0 -156.5958158941286,20.38616727824648,0 -156.9019851472443,19.83461456110239,0 -156.1842310583716,20.20053668786948,0 -156.557056073062,19.59525648291685,0 -156.1124900576321,19.74835140750147,0 +% +% +% +% FOR NOW +% search for the name of the path you want and find the coordinates line. +% then just copy and paste the coordinates into a text file alone. +% txtCoordFile = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\kml_leeward_coords.txt'; +% txtCoordFile = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\kml_windward_coords.txt'; +% txtCoordFile = 'C:\Users\Selene.Fregosi\Desktop\sg680_HI_Apr2022\kml_leeward_revised_20200503.txt'; +% txtCoordFile = 'C:\Users\Selene.Fregosi\Desktop\sg639_HI_Apr2022\kml_windward_revised_20200509.txt'; +% txtCoordFile = 'C:\Users\Selene.Fregosi\Desktop\sg680_HI_Apr2022\kml_leeward_revised_20200524.txt'; +% txtCoordFile = 'C:\Users\Selene.Fregosi\Desktop\sg639_HI_Apr2022\kml_windward_revised_20200524.txt'; +[txtCoordFileName, txtCoordPath] = uigetfile([kmlPath '\*.txt'], 'Select .txt coordinate file'); +txtCoordFile = fullfile(txtCoordPath, txtCoordFileName); + +% read in the coords and rearrange in a readable way +fid = fopen(txtCoordFile, 'r'); +decDegCoords = textscan(fid, '%f%f%f', 'delimiter', ',', 'CollectOutput',1); +fclose(fid); +decDegCoords = decDegCoords{1,1}; % convert to array +decDegCoords = decDegCoords(:,1:2); % get rid of the "z" coord which should be 0s +% leeward glider has to be flipped because track order goes from N to S +% decDegCoords = flip(decDegCoords); + +% convert to deg decmins +degMinLons = decdeg2degmin(decDegCoords(:,1)); +degMinLats = decdeg2degmin(decDegCoords(:,2)); +% define waypoint names +% wpNames = {'LW01', 'LW02', 'LW03', 'LW04', 'LW05', 'LW06', 'LWmA', 'LW07', ... +% 'LWmB', 'LW08', 'LWmC', 'LW09', 'LWmD', 'LW10', 'LWmE', 'LW11', 'LWmF', ... +% 'LW12', 'LWmG', 'LW13', 'LWmH', 'LW14', 'LW15', 'LWmI', 'LWmJ', 'RECV'}'; +% wpNames = {'LW01', 'LW02', 'LW03', 'LW04', 'LW05', 'LW06', 'LW07', 'LW08', ... +% 'LW09', 'LW10', 'LW11', 'LW12', 'LW13', 'LW14', 'LW15', 'LW16', 'RECV'}'; + +% wpNames = {'WW01', 'WW02', 'WW03', 'WW04', 'WW05', 'WW06', 'WW07', 'WW08', ... +% 'WW09', 'WW10', 'WW11', 'WW12', 'WW13', 'WW14', 'WW15', 'WW16', 'RECV'}'; +% wpNames = {'WW01', 'WW02', 'WWaa', 'WWab', 'WW03', 'WWm4', 'WW04', 'WWmE', ... +% 'WW05', 'WWmF', 'WW06', 'WWmG', 'WW07', 'WWmH', 'WW08', 'WWmI', 'WW09', ... +% 'WWmJ', 'WW10', 'WWmK', 'WW11', 'WWmL', 'WW12', 'WWmM', 'WW13', 'WWmN', ... +% 'WW14', 'WWmO', 'WW15', 'WWmP', 'WW16', 'WWmQ', 'WW17', 'WWmR', 'WWmS', ... +% 'WWmT', 'RECV'}'; +wpNames = {'WW01', 'WW02', 'WW03', 'WW04', 'WW05', 'WW06', 'WW07', 'WW08', ... + 'WW09', 'WW10', 'WW11', 'WW12', 'WW13', 'WW14', 'WW15', 'WW16', ... + 'WW17', 'RECV'}'; + +% now write it into a targets file +% example header text +% / Glider survey plan for UxS HI March 2022 +% / Deployment will take place at WPo1 +% / template WPo lat= lon= radius=2000 goto=WPo +% / radius set to 2000 m + +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_leeward_20220121_auto'; +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_windward_20220121_auto'; +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_leeward_revised_20220503_auto'; +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_windward_revised_20220509_auto'; +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_leeward_revised_20220524_auto'; +% targetsOut = 'C:\Users\Selene.Fregosi\Documents\UxS_HI_Glider\piloting\targets_windward_revised_20220524_auto'; + +targetsOut = fullfile(kmlPath, ['targets_' kmlName]); +fid = fopen(targetsOut, 'w'); +% write the header text +% fprintf(fid, '%s\n', '/ Glider survey plan for UxS HI April 2022 - Leeward Glider'); +% fprintf(fid, '%s\n', '/ Deployment will take place at LW01, recovery at RECV'); +fprintf(fid, '%s\n', '/ Glider survey plan for UxS MHI April 2023 - SG639 - Windward Glider'); +fprintf(fid, '%s\n', '/ Deployment will take place at WW01, recovery at RECV'); + +fprintf(fid, '%s\n', '/ template WPxx lat=DDMM.MMMM lon=DDDMM.MMMM radius=2000 goto=WPo'); +fprintf(fid, '%s\n', '/ radius set to 2000 m'); + +for f = 1:length(wpNames)-1 +fprintf(fid, '%s lat=%d%07.4f lon=%d%07.4f radius=2000 goto=%s\n', ... + wpNames{f}, degMinLats(f,1), degMinLats(f,2), degMinLons(f,1), degMinLons(f,2), wpNames{f+1}); +end +f = length(wpNames); +fprintf(fid, '%s lat=%d%07.4f lon=%d%07.4f radius=2000 goto=%s', ... + wpNames{f}, degMinLats(f,1), degMinLats(f,2), degMinLons(f,1), degMinLons(f,2), wpNames{f}); +fclose(fid); + +%% plot +glider = ['sg639']; % leeward +% glider = 'sg680'; % windward +latLim = [17 22]; +lonLim = [-157 -152]; +path_shp = 'C:\Users\Selene.Fregosi\Documents\GIS\'; + +plotGliderPath_etopo(glider, latLim, lonLim, pp, path_out, path_shp, figNum) + + diff --git a/agate/convertAcoustics/convertPmar/convertPmarFun.m b/agate/convertAcoustics/convertPmar/convertPmarFun.m index 5371048..aa2c766 100644 --- a/agate/convertAcoustics/convertPmar/convertPmarFun.m +++ b/agate/convertAcoustics/convertPmar/convertPmarFun.m @@ -13,11 +13,11 @@ function convertPmarFun(CONFIG) % convertPMARFun.m is a functionized version of the convertPmar.m script. % it allows for a CONFIG input argument that is created from the % pmarConvertConfig_template.m, which is meant to keep configuration for -% each survey organized in its own file +% each mission organized in its own file % Dave Mellinger % Oregon State Univ. -% last modified 2024 02 28 S. Fregosi selene.fregosi@gmail.com +% last modified 2024 08 07 S. Fregosi selene.fregosi@gmail.com %% %%%%%%%%%%%%%%%%%%%%%%%%%%%% Configuration %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/agate/convertAcoustics/convertWispr/convertWispr.m b/agate/convertAcoustics/convertWispr/convertWispr.m index c38148f..f9499fb 100644 --- a/agate/convertAcoustics/convertWispr/convertWispr.m +++ b/agate/convertAcoustics/convertWispr/convertWispr.m @@ -19,7 +19,7 @@ function convertWispr(CONFIG) % % % convertPmarFun.m is a functionized version of the convertPmar.m script. % % % it allows for a CONFIG input argument that is created from the % % % pmarConvertConfig_template.m, which is meant to keep configuration for -% % % each survey organized in its own file +% % % each mission organized in its own file % % Inputs: % CONFIG [struct] optional agate global mission configuration @@ -40,7 +40,7 @@ function convertWispr(CONFIG) % S. Fregosi % % FirstVersion: 30 June 2023 -% Updated: +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/agate/example_workflows/README_example_workflows.md b/agate/example_workflows/README_example_workflows.md new file mode 100644 index 0000000..06dc45b --- /dev/null +++ b/agate/example_workflows/README_example_workflows.md @@ -0,0 +1,3 @@ +README example_workflows + +Large files/folders that cannot be hosted on GitHub but are useful for testing can be found on Google Drive at [agate_test_data](https://drive.google.com/drive/u/0/folders/1GnItHEbR1tEG1HANGU7hliI625q2OORB) \ No newline at end of file diff --git a/agate/example_workflows/workflow_convertPMAR.m b/agate/example_workflows/workflow_convertPMAR.m index aeb07b0..3642174 100644 --- a/agate/example_workflows/workflow_convertPMAR.m +++ b/agate/example_workflows/workflow_convertPMAR.m @@ -31,8 +31,7 @@ % - specify path to PMAR convert configuration file with CONFIG.pm.cnfFile % initialize agate - either specify the mission .cnf or leave blank to browse/select -agate agate_mission_config.cnf -global CONFIG +CONFIG = agate('agate_mission_config.cnf'); % convert! convertPmarFun(CONFIG) diff --git a/agate/example_workflows/workflow_downloadScript.m b/agate/example_workflows/workflow_downloadScript.m index d911dbc..d4a4430 100644 --- a/agate/example_workflows/workflow_downloadScript.m +++ b/agate/example_workflows/workflow_downloadScript.m @@ -33,8 +33,7 @@ % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % initialize agate -agate agate_mission_config.cnf -global CONFIG +CONFIG = agate('agate_mission_config.cnf'); % specify the local piloting folder for this trip in CONFIG.path.mission % set up nested folders for basestation files and piloting outputs @@ -46,7 +45,7 @@ mkdir(path_bsLocal); %% (1) download files from the basestation -downloadBasestationFiles(CONFIG639, path_bsLocal) +downloadBasestationFiles(CONFIG, path_bsLocal) % To plot Seaglider Piloting Tools plots at this point, run DiveData below % DiveData diff --git a/agate/example_workflows/workflow_surveyTrackPlanning.m b/agate/example_workflows/workflow_missionTrackPlanning.m similarity index 72% rename from agate/example_workflows/workflow_surveyTrackPlanning.m rename to agate/example_workflows/workflow_missionTrackPlanning.m index fe20e2b..e90c648 100644 --- a/agate/example_workflows/workflow_surveyTrackPlanning.m +++ b/agate/example_workflows/workflow_missionTrackPlanning.m @@ -1,44 +1,37 @@ -% WORKFLOW_SURVEYTRACKPLANNING +% WORKFLOW_MISSIONTRACKPLANNING % Planned mission path kmls to targets file and pretty map % % Description: -% This script takes a survey track created in Google Earth and saved -% as a .kml file and prepares it for the survey +% This script takes a planned track created in Google Earth and saved +% as a .kml file and prepares it for the mission % (1) creates a properly formatted 'targets' file to be loaded onto % the glider -% (2) creates a high quality planned survey map figure +% (2) creates a high quality planned mission map % (3) creates a plot of the bathymetry profile along the targets % track -% (4) calculates full planned track distance and distance to end from +% (4) exports a .csv of approx 5-km spaced trackpoints for estimating +% arrival dates/times +% (5) calculates full planned track distance and distance to end from % each waypoint for mission duration estimation % % This requires access to bathymetric basemaps for plotting and % requires manual creation of the track in Google Earth. Track must -% be saved as a kml containing just a single track/path. To properly -% save: within Google Earth, right click on track name in left panel, -% select save place as, change file type from .kmz to .kml, save +% be saved as a kml containing just a single track/path. More +% information on creating a path in Google Earth can be found at +% https://sfregosi.github.io/agate-public/mission-planning.html#create-planned-track-using-google-earth-pro % -% See also -% -% TO DO: -% [ ] link to documentation on kml track creation -% [ ] simplify how header information is input/updated (have it all -% be changed in one place in the script instead of several lines % % Authors: % S. Fregosi % % FirstVersion: 05 April 2023 -% Updated: 10 May 2023 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % initialize agate - either specify a .cnf or leave blank to browse/select -agate agate_mission_config.cnf - -global CONFIG - +CONFIG = agate('agate_mission_config.cnf'); %% (1) Generate targets file from Google Earth path saved as .kmml @@ -106,14 +99,30 @@ CONFIG.mission, '_targetsBathymetryProfile_' targetsName, '.png']), ... 'Resolution', 300) -%% (4) Calculate track distance and mission duration +%% (4) Export interpolated track points + +% create interpolated trackpoints every approx 5 km +interpTrack = interpolatePlannedTrack(CONFIG, targetsFile, 5); +% the spacing will not be perfectly at 5 km, but will break each track +% segment up into the number of points to be near 5 km betwee each +% If spacing is required to be more exact, it needs to be done in a +% GIS program (QGIS or ArcGIS) using the line splitting tools + +% write to csv +writetable(interpTrack, fullfile(CONFIG.path.mission, ... + ['trackPoints_', targetsName, '.csv'])); +% timing information can now be manually added in Exxcel based on planned +% deployment date/time and estimated speed + + +%% (5) Calculate track distance and mission duration % if no targetsFile specified, will prompt to select [targets, targetsFile] = readTargetsFile(CONFIG); % OR specify targetsFile variable from above [targets, targetsFile] = readTargetsFile(CONFIG, targetsFile); -% loop through all targets (expect RECV), calc distance between waypoints +% loop through all targets (except RECV), calc distance between waypoints for f = 1:height(targets) - 1 [targets.distToNext_km(f), ~] = lldistkm([targets.lat(f+1) targets.lon(f+1)], ... [targets.lat(f) targets.lon(f)]); diff --git a/agate/example_workflows/workflow_plotMultipleGliders.m b/agate/example_workflows/workflow_plotMultipleGliders.m new file mode 100644 index 0000000..cf72ef1 --- /dev/null +++ b/agate/example_workflows/workflow_plotMultipleGliders.m @@ -0,0 +1,86 @@ +% WORKFLOW_PLOTMULTIPLEGLIDERS.M +% Plot multiple glider paths and/or cetacean encounters on a single map +% +% Description: +% Example script for creating a single map with multiple glider +% tracks. Glider labels and plotting colors are defined at the top +% and then a basemap is made where glider tracks or cetacean events +% can be added on top. A title and legend are added last. +% +% Notes +% +% See also CREATEBASEMAP +% +% +% Authors: +% S. Fregosi +% +% FirstVersion: 14 March 2024 +% Updated: 08 August 2024 +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%% (1) Define glider labels and colors + +% set some colors +col_sg = [1 1 0; % yellow + 1 0.4 0]; % orange + +% what should each glider be called? +gliders = {'sg001', 'sg002'}; + + +%% (2) Create the basemap + +% configure agate with one of the gliders to get map settings +CONFIG = agate('agate_mission_config_sg001.cnf'); + +% set bathymetry or contours on or off +bathyOn = 1; contourOn = 0; +% north arrow, scale bar or map limits can be defined in the CONFIG file +% specified above or manually set here +% e.g., to set north arrow location +CONFIG.map.naLat = 19.4; +CONFIG.map.naLon = -159.6; + +% create basemap figure +[baseFig] = createBasemap(CONFIG, bathyOn, contourOn); +baseFig.Position = [20 80 1200 700]; % set position on screen + + +%% (3) Plot first glider's surface positions + +% load gpsSurf table created with extractPositionalData +load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.gmStr '_gpsSurfaceTable.mat'])); +% plot +h(1) = plotm(gpsSurfT.startLatitude, gpsSurfT.startLongitude, ... + 'Color', col_sg(1,:), 'LineWidth', 1.5, 'DisplayName', gliders{1}); + + +%% (4) Plot second glider's surface positions + +% reinitilize agate with correct config file to get correct paths/strings +CONFIG = agate('agate_mission_config_sg002.cnf'); + +% load gpsSurf table created with extractPositionalData +load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.gmStr '_gpsSurfaceTable.mat'])); +% plot +h(2) = plotm(gpsSurfT.startLatitude, gpsSurfT.startLongitude, ... + 'Color', col_sg(2,:), 'LineWidth', 1.5, 'DisplayName', gliders{2}); + +% this could be repeated for as many gliders as exist + +%% (5) Add legend, title and save + +% add legend +legend(h, 'Location', 'northeast', 'FontSize', 14) + +% add title +title('Glider mission', 'Interpreter', 'none') + +% save it +exportgraphics(gca, 'combined_map.png', 'Resolution', 600); + diff --git a/agate/example_workflows/workflow_processPositionalData.m b/agate/example_workflows/workflow_processPositionalData.m index ca2cd0a..04a481f 100644 --- a/agate/example_workflows/workflow_processPositionalData.m +++ b/agate/example_workflows/workflow_processPositionalData.m @@ -12,6 +12,19 @@ % % It requires an agate configuration file during agate initialization % +% Sections +% (0) Initialization - initializes agate with proper config file +% (1) Extract positional data - create `locCalcT` and `gpsSurfT` +% tables with glider timing, positions, speed, etc +% (2) Simplify positional data into smaller .csvs for to include with +% as metadata when sending sound files to NCEI +% (3) Plot sound speed profiles +% (4) PAM status get more accurate info on recording times and +% durations from the files themselves, and update positional data +% tables with a flag for PAM on or off at each sample or dive +% (5) Extract location data for each individual PAM file +% (6) Summarize acoustic effort by minutes, hours, and days +% % Notes % % See also @@ -21,16 +34,15 @@ % S. Fregosi % % FirstVersion: 21 April 2023 -% Updated: 11 April 2024 +% Updated: 08 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % initialize agate -agate agate_mission_config.cnf % or just agate and select file -global CONFIG +CONFIG = agate('agate_mission_config.cnf'); % or just agate and select file -%% extract positional data +%% (1) Extract positional data % This step can take some time to process through all .nc files [gpsSurfT, locCalcT] = extractPositionalData(CONFIG, 1); @@ -48,49 +60,65 @@ writetable(locCalcT, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.csv'])); -%% save positional data for packaging for NCEI +%% (2) Simplify positional data for packaging for NCEI -% gps surface table +% surface location table +% load gpsSurfT if not already loaded if ~exist('gpsSurfT', 'var') load(fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.mat'])); end + +% clean up columns/names keepCols = {'dive', 'startDateTime', 'startLatitude', 'startLongitude', ... 'endDateTime', 'endLatitude', 'endLongitude'}; gpsSurfSimp = gpsSurfT(:,keepCols); newNames = {'DiveNumber', 'StartDateTime_UTC', 'StartLatitude', 'StartLongitude', ... 'EndDateTime_UTC', 'EndLatitude', 'EndLongitude'}; gpsSurfSimp.Properties.VariableNames = newNames; + +% write to csv writetable(gpsSurfSimp, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_GPSSurfaceTableSimple.csv'])) -% location table +% claculated location table +% load locCalcT if not already loaded if ~exist('locCalcT', 'var') load(fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat'])) end + +% clean up columns/names keepCols = {'dateTime', 'latitude', 'longitude', 'depth', 'dive'}; locCalcSimp = locCalcT(:,keepCols); newNames = {'DateTime_UTC', 'Latitude', 'Longitude', 'Depth_m', 'DiveNumber'}; locCalcSimp.Properties.VariableNames = newNames; + +% write to csv writetable(locCalcSimp, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_CalculatedLocationTableSimple.csv'])) % environmental data +% load locCalcT if not already loaded if ~exist('locCalcT', 'var') load(fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat'])) end + +% clean up columns/names keepCols = {'dive', 'dateTime', 'latitude', 'longitude', 'depth', ... 'temperature', 'salinity', 'soundVelocity', 'density'}; locCalcEnv = locCalcT(:,keepCols); newNames = {'DiveNumber', 'DateTime_UTC', 'Latitude', 'Longitude', 'Depth_m', ... 'Temperature_C', 'Salinity_PSU', 'SoundSpeed_m_s', 'Density_kg_m3', }; locCalcEnv.Properties.VariableNames = newNames; + +% write to csv writetable(locCalcEnv, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_CTD.csv'])) -%% plot sound speed profile +%% (3) Plot sound speed profile + % load locCalcT if not already loaded if ~exist('locCalcT', 'var') load(fullfile(CONFIG.path.mission, 'profiles', ... @@ -98,50 +126,95 @@ end plotSoundSpeedProfile(CONFIG, locCalcT); + +% save as .png and .pdf exportgraphics(gcf, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_SSP.png'])) exportgraphics(gcf, fullfile(CONFIG.path.mission, 'profiles', ... [CONFIG.glider, '_', CONFIG.mission, '_SSP.pdf'])) -%% BELOW SECTIONS ARE NOT YET OPERATIONAL -% Below called functions are in the 'drafts' folder and need to be adapted -% for WISPR2 and also updated to new CONFIG system of agate -%% extract acoustic system status for each dive and sample time +%% (4) Extract acoustic system status for each dive and sample time -% *** NEEDS WORK! *** to be updated to deal with WISPR2 and speed up with -% PMARXL using .eng files rather than having to read in list of sound files +% load locCalcT and gpsSurfT if not already loaded +if ~exist('locCalcT', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat'])) +end +if ~exist('gpsSurfT', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.mat'])); +end -% % this looks at each recorded file timestamp to populate a 'pam' column -% % that is added to locCalcT and gpsSurfT that specifies the status of the -% % pam system for each entry. -% -% fileLength = 600; % in seconds -% dateFormat = 'yyMMdd-HHmmss'; -% dateStart = 1; % what part of file name starts the date format -% -% [gpsSurfT, locCalcT, pam] = extractPAMStatusByFile(gldr, lctn, dplymnt, ... -% fileLength, dateFormat, dateStart, gpsSurfT, locCalcT); -% % saved automatically gpsSurfTable_pam.mat and locCalcT_pam.mat and -% % _pamByFile.mat +% loop through sound files to gets 'status' for existing positional tables +[gpsSurfT, locCalcT, pamFiles, pamByDive] = extractPAMStatus(CONFIG, ... + gpsSurfT, locCalcT); +fprintf('Total PAM duration: %.2f hours\n', hours(sum(pamFiles.dur, 'omitnan'))); -%% extract positional data for each sound file -% -% secs = 180; -% -% filePosits = extractPositsPerPAMFile(gldr, lctn, dplymnt, ... -% pam, locCalcT,secs, path_profiles); -% -% % this saves _pamFilePosits.mat and .csv +% save updated positional tables and pam tables +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider, '_', ... + CONFIG.mission, '_pamFiles.mat']), 'pamFiles'); +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider, '_', ... + CONFIG.mission, '_pamByDive.mat']), 'pamByDive'); + +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider '_' ... + CONFIG.mission '_locCalcT_pam.mat']), 'locCalcT'); +writetable(locCalcT, fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider '_' CONFIG.mission '_locCalcT_pam.csv'])); -%% extract positional data and acoustic effort by minute +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider '_' ... + CONFIG.mission '_gpsSurfaceTable_pam.mat']), 'gpsSurfT'); +writetable(gpsSurfT, fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider '_' CONFIG.mission '_gpsSurfaceTable_pam.csv'])); -% % create byMin, minPerHour, minPerDay matrices for full experiment extent -% % will need this for comparison down the line -% [pamByMin, pamMinPerHour, pamMinPerDay] = ... -% calcPAMEffort(gldr, lctn, dplymnt, expLimits, gpsSurfT, path_profiles); -% % this saves _pamByMin.mat -% -% + +%% (5) Extract positional data for each sound file + +% load locCalcT and pamFiles if not already loaded +if ~exist('locCalcT', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat'])) +end +if ~exist('pamFiles', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_pamFiles.mat'])) +end + +% set a time buffer around which locations are acceptable +timeBuffer = 180; +% get position at start of each sound file +pamFilePosits = extractPAMFilePosits(pamFiles, locCalcT, timeBuffer); + +% save as .mat and .cs +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider '_' ... + CONFIG.mission '_pamFilePosits.mat']), 'pamFilePosits'); +writetable(pamFilePosits, fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider '_' CONFIG.mission '_pamFilePosits.csv'])); + + +%% (6) Summarize acoustic effort + +% load gpsSurfT, pamFiles, pamByDive if not already loaded +if ~exist('gpsSurfT', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.mat'])) +end +if ~exist('pamFiles', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_pamFiles.mat'])) +end +if ~exist('pamByDive', 'var') + load(fullfile(CONFIG.path.mission, 'profiles', ... + [CONFIG.glider, '_', CONFIG.mission, '_pamByDive.mat'])) +end + +% create byMin, minPerHour, minPerDay matrices +[pamByMin, pamMinPerHour, pamMinPerDay, pamHrPerDay] = calcPAMEffort(... + CONFIG, gpsSurfT, pamFiles, pamByDive); + +% save as .mat +save(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.glider '_' ... + CONFIG.mission '_pamEffort.mat']), ... + 'pamByMin', 'pamMinPerHour', 'pamMinPerDay', 'pamHrPerDay'); diff --git a/agate/scratch/extractPAMStatusByFile.m b/agate/scratch/extractPAMStatusByFile.m deleted file mode 100644 index a8b5bb6..0000000 --- a/agate/scratch/extractPAMStatusByFile.m +++ /dev/null @@ -1,234 +0,0 @@ -function [gpsSurfT, locCalcT, pam] = extractPAMStatusByFile(CONFIG, gpsSurfT, locCalcT) -%EXTRACTPAMSTATUSBYFILE Extracts PAM system on/off information from sound files -% -% Syntax: -% [GPSSURFT, LOCCALCT, PAM] = EXTRACTPAMSTATUSBYFILE(CONFIG, GPSSURFT, LOCCALCT) -% -% Description: -% -% -% Inputs: -% CONFIG agate mission configuration file with relevant mission and -% glider information. Minimum CONFIG fields are 'glider', -% 'mission', 'path.mission' -% fileLength [double] duration of files expected, in seconds -% dateFormat [string] format of timestamp in file name, in datetime -% input format syntax e.g., 'yyMMdd-HHmmss' -% https://www.mathworks.com/help/matlab/ref/datetime.html#d122e273617 -% gpsSurfT [table] glider surface locations exported from -% extractPositionalData -% locCalcT [table] glider fine scale locations exported from -% extractPositionalData -% -% Outputs: -% gpsSurfT [table] glider surface locations, from GPS, one per -% dive. Input gpsSurfT is updated to now include a pam -% column with the minutes of PAM recording for that dive. -% Origingal columns include dive start and end -% time/lat/lon, dive duration, depth average current, -% average speed over ground as northing and easting, -% calculated by the hydrodynamic model or the glide slope -% model -% locCalcT [table] Glider calculated locations underwater every -% science file sampling interval. Input locCalcT is updated -% to include a pam column that has a 1 for pam system on or -% 0 for pam system off for each location entry. Original -% instantaneous flight details and includes columns -% for time, lat, lon from hydrodynamic and glide slope -% models, displacement from both models, temperature, -% salinity, density, sound speed, glider vertical and -% horizontal speed (from both models), pitch, glide -% angle, and heading -% -% Examples: -% -% See also EXTRACTPOSITIONALDATA -% -% TO DO: -% - build in option for FLAC -% -% Authors: -% S. Fregosi -% -% FirstVersion: ?? -% Updated: 03 April 2024 -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -% Example inputs: -% gldr = 'sg607'; -% lctn = 'CatBasin'; -% dplymnt = 'Jul16'; -% fileLength = 120; % in seconds; either 120 or 80 for wispr -% sampleRate = 1000; % in Hz, runs fastest on downsampled 1 kHz data -% dateFormat = 'yyMMdd-HHmmss'; - % this needs to be in datetime input format sytnax - % https://www.mathworks.com/help/matlab/ref/datetime.html#d122e273617 -% dateStart = 1; % what part of file name starts the date format - - - -%% initialization - -% check acoustic system -if isfield(CONFIG, 'pm') && CONFIG.pm.loggers == 1 - loggerType = 'PMARXL'; -elseif isfield(CONFIG, 'ws') && CONFIG.ws.loggers == 1 - loggerType = 'WISPR'; -else - fprintf(1, 'Unknown acoustic logger type. Exiting\n'); - return -end - -% if specified in CONFIG, get file length (in seconds) and sample rate -switch loggerType - case 'PMARXL' - if isfield(CONFIG.pm, 'fileLength') - fileLength = CONFIG.pm.fileLength; -% sampleRate = CONFIG.pm.sampleRate; - else - fprintf(1, ['No file length specified in .cnf, using PMARXL ', ... - 'default fileLength = 600 s and sampleRate = 180260\n']); - fileLength = 600; -% sampleRate = 180260; - end - case 'WISPR' - fprintf(1, 'File length not set up yet for WISPR. Exiting\n'); - return -end - -% OLD - but saving in case needed - from 1 kHz data!!! -% get num of samples per file (if max file length) -% if fileLength == 120 % wispr -% numSamples = 119931; % for 2 min file -% elseif fileLength == 80 % wispr -% numSamples = 79954; % for 1 min 20 sec files downsampled to 1000 hz -% elseif fileLength == 600 % PMARXL -% numSamples = 600000; -% else -% fprintf('Unknown fileLength...aborting\n') -% return -% end - -% specify deployment date and time to ignore all files before that -% (sometimes test files recorded in lab are in dataset) -deplDate = gpsSurfT.startDateTime(1); - -% select folder with sound files and where to save -% path_in = uigetdir('G:\','Select base folder'); -path_acous = uigetdir('C:\', 'Select folder with 1 kHz acoustic data'); -% pick 1 kHz data because it will run faster...but could change in future. -path_out = uigetdir(path_acous, 'Select profiles folder to save outputs'); - -%% Read in files and extract duration information -files = dir(fullfile(path_acous, '*.wav')); -if isempty(files) - fprintf('No .wav files found...aborting\n'); - return - % *****later build in switch to flac?? - % files=dir([path_flac '*.flac']); -end - -pam = table; -shortfiles = []; -fprintf(1,'%i files:\n', length(files)); - -% make matrix with start and end times for all PAM files -for f = 1:length(files) - % calc file duration in sec using sampling rate..slow but works - more accurate - try - wavInfo = audioinfo(fullfile(path_acous, files(f,1).name)); - files(f,1).dur = wavInfo.TotalSamples./wavInfo.SampleRate; - if files(f,1).dur < fileLength - shortfiles = [shortfiles; f wavInfo.TotalSamples]; %#ok - fprintf(1,'%s is short: %i samples, %.2f seconds\n', files(f,1).name, ... - wavInfo.TotalSamples, files(f,1).dur); - end - % get start timing information from file name - dtIdx = CONFIG.pm.dateStart:length(CONFIG.pm.dateFormat)+CONFIG.pm.dateStart-1; - pam.fileStart(f,1) = datetime(files(f).name(dtIdx), ... - 'InputFormat', CONFIG.pm.dateFormat); - pam.fileEnd(f,1) = pam.fileStart(f,1) + seconds(files(f,1).dur); - - catch % if there is some issue reading a file - fprintf(1, '%s is corrupt\n', files(f,1).name); - end - if rem(f,1000)==0; fprintf(1,'%i DONE\n',f);end % counter -end - -% remove all before deployment date -[r, ~] = find(pam.fileStart < deplDate,1,'last'); -if ~isempty(r) - pam = pam(r+1:end,:); -end - -save([path_out '\' glider '_' deploymentStr '_pamByFile.mat'],'pam'); - -%% Specify 1's and 0s per locCalcT row -% 1 if pam on, 0 if off - -locCalcT.pam = zeros(height(locCalcT),1); -fprintf(1,'%s - %d samples:\n', [glider '_' deploymentStr], height(locCalcT)); -for f = 1:height(locCalcT) - idx = find(isbetween(locCalcT.dateTime(f),pam.fileStart,pam.fileEnd),1); - if ~isempty(idx) - locCalcT.pam(f) = 1; - end - clear idx - - fprintf(1,'.'); - if (rem(f,80) == 0), fprintf(1,'\n%3d',floor((height(locCalcT)-f)/80)); end -end - -save([path_out '\' glider '_' deploymentStr '_locCalcT_pam.mat'],'locCalcT'); -writetable(locCalcT, [path_out '\' glider '_' deploymentStr '_locCalcT_pam.csv']); - -% % plotting test -% plotDiveProfile(locCalcT) - -%% Calculate duration of files and Total Duration - -pam.dur = pam.fileEnd - pam.fileStart; -totDur = nansum(pam.dur); % as datetime duration format -totDurHrs = hours(totDur); - -fprintf('Total PAM duration: %.2f hours\n', totDurHrs); - -save([path_out '\' glider '_' deploymentStr '_pamByFile.mat'],'pam','totDur','totDurHrs'); - -%% duration per dive - -pamByDive = table; -pamByDive.dive = gpsSurfT.dive; -pamByDive.diveStart = gpsSurfT.startDateTime; -pamByDive.diveEnd = gpsSurfT.endDateTime; -pamByDive.numFiles = nan(height(pamByDive),1); - -for f = 1:height(pamByDive) - [r, ~] = find(isbetween(pam.fileStart, pamByDive.diveStart(f), ... - pamByDive.diveEnd(f))); - if ~isempty(r) - pamByDive.numFiles(f,1) = length(r); - pamByDive.pamDur(f,1) = sum(pam.dur(r)); - pamByDive.pamStart(f,1) = pam.fileStart(r(1)); - pamByDive.pamEnd(f,1) = pam.fileEnd(r(end)); - end - -end - -pamByDive.lagStart = pamByDive.pamStart - pamByDive.diveStart; -pamByDive.lagEnd = pamByDive.diveEnd - pamByDive.pamEnd; - -save([path_out '\' glider '_' deploymentStr '_pamByFile.mat'],'pam','totDur','totDurHrs','pamByDive'); - -% append to gpsSurfT and save -gpsSurfT.pamDur = pamByDive.pamDur; -gpsSurfT.pamNumFiles = pamByDive.numFiles; -gpsSurfT.pamStart = pamByDive.pamStart; -gpsSurfT.pamEnd = pamByDive.pamEnd; - -save([path_out '\' glider '_' deploymentStr '_gpsSurfaceTable_pam.mat'],'gpsSurfT'); -writetable(gpsSurfT, [path_out '\' glider '_' deploymentStr '_gpsSurfaceTable_pam.csv']); - - diff --git a/agate/scratch/tempSalinDataToDensity.m b/agate/scratch/tempSalinDataToDensity.m new file mode 100644 index 0000000..24ea5fb --- /dev/null +++ b/agate/scratch/tempSalinDataToDensity.m @@ -0,0 +1,54 @@ +% read in temp/salinity stuff from Andy +% calculate density +% sigma has no units...add 1000 to get it in kg/m3 +% get min and max for the whole deployment + +warning off +path_in = 'C:\Users\selene\Box\HDR-SOCAL-2018\piloting\'; + +fNames = dir([path_in '*.info']); +headerSpec = '%*s %f %*s %f %*[^\n]'; +colNamesSpec = '%s %s %s %s'; +dataSpec = '%f %f %f %f'; +out = table; +outIdx = 0; + +for f = 1:length(fNames) + fid = fopen([path_in fNames(f).name],'r'); + for ll = 1:5 % 5 lat/lons to check + outIdx = outIdx + 1; + c = textscan(fid, headerSpec, 1); % reads to blank line + [lat lon] = c{:}; + colNames = textscan(fid, colNamesSpec, 1); + colNames = string(colNames); + c = textscan(fid, dataSpec, 'CollectOutput',true); + data = c{:}; + data = array2table(data, 'VariableNames', colNames); + % this reads to the blank line, then repeat? + data.density = density(data.Sal, data.Temp); + data.densityP = densatp(data.Sal, data.Temp, data.Depth(1)); + + [mi mx] = calcDensityRange(data, lat, lon); +% fprintf(1,'%s Lat: %f Lon: %f min: %f and max: %f\n', ... +% fNames(f).name, lat, lon, mi, mx) + out.month{outIdx,1} = fNames(f).name(1:3); + out.lat(outIdx,1) = lat; + out.lon(outIdx,1) = lon; + out.minSigma(outIdx,1) = mi; + out.maxSigma(outIdx,1) = mx; + out.minDens(outIdx,1) = min(data.density(data.Depth <= 1000,1)); + out.maxDens(outIdx,1) = max(data.density(data.Depth <= 1000,1)); + + end + fclose(fid); +end + +min(out.minSigma) +max(out.maxSigma) + +densRange = out; +save([path_in 'densityRanges.mat'],'densRange'); + + +min(out.minDens) +max(out.maxDens) diff --git a/agate/scratch/workflow_cleanLogs.m b/agate/scratch/workflow_cleanLogs.m deleted file mode 100644 index 7024b74..0000000 --- a/agate/scratch/workflow_cleanLogs.m +++ /dev/null @@ -1,23 +0,0 @@ -% WORKFLOW_CLEANLOGS.M -% Example workflow for cleaning up/combining Triton or ERMA logs -% -% Description: -% Detailed description here, please -% -% Notes -% -% See also -% -% -% Authors: -% S. Fregosi -% -% FirstVersion: 13 March 2024 -% Updated: -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -% created this placeholder but have done no work on it. - -% see glider-mhi repo for example from MHI project - workflow_cleanUpLogs.m \ No newline at end of file diff --git a/agate/scratch/workflow_cleanUpLogs.m b/agate/scratch/workflow_cleanUpLogs.m new file mode 100644 index 0000000..c879f47 --- /dev/null +++ b/agate/scratch/workflow_cleanUpLogs.m @@ -0,0 +1,75 @@ +% WORKFLOW_CLEANLOGS.M +% Example workflow for cleaning up/combining Triton or ERMA logs +% +% Description: +% Detailed description here, please +% +% Notes +% +% See also +% +% +% Authors: +% S. Fregosi +% +% FirstVersion: 13 March 2024 +% Updated: +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +% created this placeholder but have done no work on it. + +% see glider-mhi repo for example from MHI project - workflow_cleanUpLogs.m + + +% workflow for cleaning up Triton logs for fkw analysis +% +% This requires the agate-public repository, access at: +% (github.com/sfregosi/agate-public) +% +% S. Fregosi 15 February 2023 + +addpath(genpath('C:\Users\Selene.Fregosi\Documents\GitHub\agate-public')); + +path_analysis = 'C:\Users\Selene.Fregosi\Documents\GitHub\glider-MHI\analysis\'; + +glider = 'sg639'; mission = 'MHI_Apr2023'; +% glider = 'sg680'; mission = 'MHI_Apr2022'; +% glider = 'sg679'; mission = 'MHI_May2023'; + +% logFileName = [glider '_' mission '_MW.xls']; +% logFile = fullfile(path_analysis, 'wood', [glider '_' mission '_log_mw'], ... +% logFileName); +% logFile = ['C:\Users\Selene.Fregosi\Documents\GitHub\glider-MHI\' ... +% 'analysis\wood\' glider '_MHI_log_mw\', glider '_MHI_log_mw_sf.xls']; + +logFileName = [glider '_' mission '_MW_recheck.xls']; +logFile = fullfile(path_analysis, 'wood', [glider '_' mission '_recheck_mw'], ... + logFileName); +eventGap = 15; + +[tl, tlm] = collapseTritonLog(logFile, eventGap); +save(fullfile(path_analysis, 'fkw', [glider '_' mission '_log_merged.mat']), ... + 'tl', 'tlm'); + +% simplify for banter and add glider to eventID string +eventStr = arrayfun(@(x) num2str(x, '%02.f'), tl.eventNum, 'UniformOutput', 0); +gc = cell(height(tl), 1); +gc(:) = {glider}; +tl.eventID = cellfun(@(x,y) [x '_' y], gc, eventStr, 'UniformOutput', 0); + +eventStr = arrayfun(@(x) num2str(x, '%02.f'), tlm.eventNum, 'UniformOutput', 0); +gc = cell(height(tlm), 1); +gc(:) = {glider}; +tlm.eventID = cellfun(@(x,y) [x '_' y], gc, eventStr, 'UniformOutput', 0); + +tls = tlm(:, [2 3 4 7]); +tls.Properties.VariableNames = {'start', 'end', 'sp', 'id'}; +tls.start.Format = 'MM/dd/uuuu HH:mm:ss'; +tls.end.Format = 'MM/dd/uuuu HH:mm:ss'; +writetable(tls, fullfile(path_analysis, 'fkw', [glider '_' mission '_log_merged.csv'])); + + + + diff --git a/agate/scratch/workflow_plotMultipleGliders.m b/agate/scratch/workflow_plotMultipleGliders.m deleted file mode 100644 index e159101..0000000 --- a/agate/scratch/workflow_plotMultipleGliders.m +++ /dev/null @@ -1,87 +0,0 @@ -% WORKFLOW_PLOTMULTIPLEGLIDERS.M -% One-line description here, please -% -% Description: -% Detailed description here, please -% -% Notes -% -% See also -% -% -% Authors: -% S. Fregosi -% -% FirstVersion: 14 March 2024 -% Updated: -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%% (3) Create encounter map - -% set some colors -col_pt = [1 1 1]; % planned track -col_rt = [0 0 0]; % realized track -% col_ce = [1 0.4 0]; % cetacean events - orange -col_ce = [1 1 0.2]; % cetacean events - yellow - -% generate the basemap with bathymetry as figure 82, don't save .fig file -[baseFig] = createBasemap(CONFIG, 1, 82); -baseFig.Position = [20 80 1200 700]; - -% add original targets -targetsFile = fullfile(CONFIG.path.mission, 'basestationFiles', 'targets'); -[targets, ~] = readTargetsFile(CONFIG, targetsFile); - -h(1) = plotm(targets.lat, targets.lon, 'Marker', 's', 'MarkerSize', 4, ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_pt, 'Color', col_pt, ... - 'DisplayName', 'planned track'); -% textm(targets.lat, targets.lon, targets.name, 'FontSize', 10) - -% plot realized track -% load surface positions -load(fullfile(CONFIG.path.mission, 'profiles', 'sg679_MHI_May2023_gpsSurfaceTable.mat')); -h(2) = plotm(gpsSurfT.startLatitude, gpsSurfT.startLongitude, ... - 'Color', col_rt, 'LineWidth', 1.5, 'DisplayName', 'realized track'); - -% plot acoustic events -h(3) = scatterm(tlm.lat, tlm.lon, 30, 'Marker', 'o', ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_ce, ... - 'DisplayName', 'cetacean event'); - -% add legend -legend(h, 'Location', 'eastoutside', 'FontSize', 14) - -% add title -title('SG679 MHI May 2023', 'Interpreter', 'none') - -%% (4) Add second glider to plot - -col_ce_sg639 = [1 0.4 0]; % cetacean events - orange - -% targets -targetsFile_sg639 = 'D:\sg639_MHI_Apr2023\piloting\basestationFiles\targets'; -[targets_sg639 ,~] = readTargetsFile(CONFIG, targetsFile_sg639); - -h(4) = plotm(targets_sg639.lat, targets_sg639.lon, 'Marker', 's', 'MarkerSize', 4, ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_pt, 'Color', col_pt, ... - 'HandleVisibility', 'off'); - -% realized track -loadTmp = load(fullfile('D:\sg639_MHI_Apr2023\piloting\profiles\sg639_MHI_Apr2023_gpsSurfaceTable.mat')); -gpsSurfT_sg639 = loadTmp.gpsSurfT; -h(5) = plotm(gpsSurfT_sg639.startLatitude, gpsSurfT_sg639.startLongitude, ... - 'Color', col_rt, 'LineWidth', 1.5, 'DisplayName', 'realized track'); - -% plot acoustic events -h(6) = scatterm(tlm.lat, tlm.lon, 30, 'Marker', 'o', ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_ce, ... - 'DisplayName', 'cetacean event'); - - -% add legend -legend(h, 'Location', 'eastoutside', 'FontSize', 14) - -% add title -title('SG679 MHI May 2023', 'Interpreter', 'none') - diff --git a/agate/scratch/workflow_processCetaceanEncounters_sg639_MHI_Apr2023.m b/agate/scratch/workflow_processCetaceanEncounters_sg639_MHI_Apr2023.m deleted file mode 100644 index 0060c75..0000000 --- a/agate/scratch/workflow_processCetaceanEncounters_sg639_MHI_Apr2023.m +++ /dev/null @@ -1,51 +0,0 @@ -% WORKFLOW_PROCESSCETACEANENCOUNTERS_SG639_MHI_APR2023 -% Workflow for creating summary plots and tables for cetacean encounters -% -% Description: -% This script takes inputs of glider positional data and cetacean -% acoustic encounter data (identified by automated or manual methods) -% and pairs them up to summarize cetacean events/encounters and -% create maps of glider locations at the time of cetacean acoustic -% encounters -% -% Sections: -% (1) Cleanup event logs (either collapse Triton logs, or combine -% ERMA detection logs -% (2) Create encounter map -% -% Notes -% -% See also -% -% -% Authors: -% S. Fregosi -% -% FirstVersion: 09 March 2024 -% Updated: 14 March 2024 -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 - -% initialize agate - either specify a .cnf or leave blank to browse/select -agate secret/agate_config_sg639_MHI_Apr2023.cnf -global CONFIG - -%% (1) Cleanup encounter logs -% Triton manual pick logs -logFileName = [CONFIG.gmStr '_MW_recheck.xls']; -logFile = fullfile(CONFIG.path.analysis, 'triton', 'wood', ... - [CONFIG.gmStr '_recheck_mw'], logFileName); -eventGap = 15; -[tl, tlm] = collapseTritonLog(logFile, eventGap); -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'triton', 'wood', ... - [CONFIG.gmStr '_log_merged.mat']), 'tl', 'tlm'); -writetable(tlm, fullfile(CONFIG.path.analysis, 'triton', 'wood', ... - [CONFIG.gmStr '_log_merged.csv'])); - - - -%% (1) Plot encounter on map - -% generate the basemap with bathymetry as figure 82, don't save .fig file -[baseFig] = createBasemap(CONFIG, 1, 82); diff --git a/agate/scratch/workflow_processCetaceanEncounters_sg679_MHI_May2023.m b/agate/scratch/workflow_processCetaceanEncounters_sg679_MHI_May2023.m deleted file mode 100644 index 2bb8ec3..0000000 --- a/agate/scratch/workflow_processCetaceanEncounters_sg679_MHI_May2023.m +++ /dev/null @@ -1,212 +0,0 @@ -% WORKFLOW_PROCESSCETACEANENCOUNTERS_SG679_MHI_MAY2023 -% Workflow for creating summary plots and tables for cetacean encounters -% -% Description: -% This script takes inputs of glider positional data and cetacean -% acoustic encounter data (identified by automated or manual methods) -% and pairs them up to summarize cetacean events/encounters and -% create maps of glider locations at the time of cetacean acoustic -% encounters -% -% Sections: -% (1) Cleanup event/encounter logs (either collapse Triton logs, or -% combine ERMA detection logs -% (2) Get locations of each event -% -% Notes -% -% See also -% -% -% Authors: -% S. Fregosi -% -% FirstVersion: 09 March 2024 -% Updated: 14 March 2024 -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 - -% initialize agate - either specify a .cnf or leave blank to browse/select -agate secret/agate_config_sg679_MHI_May2023.cnf -global CONFIG - -%% (1) Cleanup event/encounter logs - -% %%% TRITON %%% -% Triton manual pick logs -logFileName = [CONFIG.gmStr '_MW.xls']; -logFile = fullfile(CONFIG.path.analysis, 'triton', 'wood', ... - [CONFIG.gmStr '_log_mw'], logFileName); -eventGap = 15; -[tl, tlm] = collapseTritonLog(logFile, eventGap); -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - [CONFIG.gmStr '_log_merged.mat']), 'tl', 'tlm'); -writetable(tlm, fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - [CONFIG.gmStr '_log_merged.csv'])); - -% %%% ERMA %%% -% ERMA detection logs -ermaDets = combineErmaLogs(CONFIG.path.bsLocal, []); -% ignoring last event (Dive 173) because shore dive right before recovery -ermaDets(665,:) = []; -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined.mat']), 'ermaDets'); -writetable(ermaDets, fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined.csv'])); - -% ERMA log was manually checked for false positives -ermaDetsChecked = readtable(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined_trueChecked.csv'])); -% ignoring last event (Dive 173) because shore dive right before recovery -ermaDetsChecked(665,:) = []; -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined_trueChecked.mat']), 'ermaDetsChecked'); -writetable(ermaDetsChecked, fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined_trueChecked.csv'])); - - -% pull only true detections -trueIdx = find(strcmp(ermaDetsChecked.truePm, 'Y')); -ermaDetsTrue = ermaDetsChecked(trueIdx,:); -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_truePmOnly.mat']), 'ermaDetsTrue'); -writetable(ermaDetsTrue, fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_truePmOnly.csv'])); - -% collapse true sperm whale detections into 'encounters' following 15 min -% gap rule like Triton logs -ermaDetsTrueEnc = ermaDetsTrue(1,:); -counter = 2; -for f = 2:height(ermaDetsTrue) - if ermaDetsTrue.start(f) - minutes(15) < ermaDetsTrueEnc.stop(counter-1) - within = true; - ermaDetsTrueEnc.stop(counter-1,:) = ermaDetsTrue.stop(f); - else % no collapsing - ermaDetsTrueEnc(counter,:) = ermaDetsTrue(f,:); - counter = counter + 1; - end -end - -% remove the cols that no longer apply -ermaDetsTrueEnc.nClicks = []; -ermaDetsTrueEnc.startDatenum = []; -ermaDetsTrueEnc.stopDatenum = []; -ermaDetsTrueEnc.truePm = []; -ermaDetsTrueEnc.diveEncounter = []; - -% calculate duration -ermaDetsTrueEnc.duration_min = minutes(ermaDetsTrueEnc.stop - ermaDetsTrueEnc.start); - -% save as .mat and .csv -save(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_truePmOnly_merged.mat']), 'ermaDetsTrueEnc'); -writetable(ermaDetsTrueEnc, fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_truePmOnly_merged.csv'])); - -%% (1b) OR load previously processed logs - -% unchecked Triton events -load(fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - [CONFIG.gmStr '_log_merged.mat'])); - -% unchecked ERMA events -load(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined.mat'])); -% checked ERMA events -ermaDetsChecked = readtable(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_events_combined_trueChecked.csv'])); - -% true ERMA sperm whale events merged into encounters -load(fullfile(CONFIG.path.analysis, 'erma', ... - [CONFIG.gmStr '_ERMA_truePmOnly_merged.mat'])); - - -%% (2) Summarize some counts - -% %%% ERMA Detections -fprintf(1, ['%i total ERMA detections:\n %i true Pm detections\n', ... - ' %i false positives\n %i duplicates\n %i unknowns\n'], height(ermaDets), ... -length(find(strcmp(ermaDetsChecked.truePm, 'Y'))), ... -length(find(strcmp(ermaDetsChecked.truePm, 'N'))), ... -length(find(strcmp(ermaDetsChecked.truePm, 'dup'))), ... -length(find(strcmp(ermaDetsChecked.truePm, '?')))); - -fprintf(1, ['%i true Pm detections as encounters. ', ... - 'Median (IQR) duration: %.2f (%.2f) minutes\n'], ... - height(ermaDetsTrueEnc), median(ermaDetsTrueEnc.duration_min), ... - iqr(ermaDetsTrueEnc.duration_min)); - - - -%% (2) Get locations of each event - -% load previously created locCalcT -load(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.gmStr '_locCalcT.mat'])); -% find calculated location for midpoint of event - tlm.midTime = tlm.start + (tlm.stop - tlm.start)/2; -for f = 1:height(tlm) - [~, mIdx] = min(abs(locCalcT.dateTime - tlm.midTime(f))); - tlm.lat(f) = locCalcT.latitude(mIdx); - tlm.lon(f) = locCalcT.longitude(mIdx); -end -% save this -save(fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - [CONFIG.gmStr '_log_merged_wLocations.mat']), 'tlm'); - -% OR load if previously processed -load(fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - [CONFIG.gmStr '_log_merged_wLocations.mat'])); - -%% (3) Create encounter map - -% set some colors -col_pt = [1 1 1]; % planned track -col_rt = [0 0 0]; % realized track -% col_ce = [1 0.4 0]; % cetacean events - orange -col_ce = [1 1 0.2]; % cetacean events - yellow - -% generate the basemap with bathymetry as figure 82, don't save .fig file -[baseFig] = createBasemap(CONFIG, 1, 82); % with bathymetry -[baseFig] = createBasemap(CONFIG, false, 82); - -baseFig.Position = [20 80 1200 700]; - -% add original targets -targetsFile = fullfile(CONFIG.path.mission, 'basestationFiles', 'targets'); -[targets, ~] = readTargetsFile(CONFIG, targetsFile); - -h(1) = plotm(targets.lat, targets.lon, 'Marker', 's', 'MarkerSize', 4, ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_pt, 'Color', col_pt, ... - 'DisplayName', 'planned track'); -% textm(targets.lat, targets.lon, targets.name, 'FontSize', 10) - -% plot realized track -% load surface positions -load(fullfile(CONFIG.path.mission, 'profiles', [CONFIG.gmStr '_gpsSurfaceTable.mat'])); -h(2) = plotm(gpsSurfT.startLatitude, gpsSurfT.startLongitude, ... - 'Color', col_rt, 'LineWidth', 1.5, 'DisplayName', 'realized track'); - -% plot acoustic events -h(3) = scatterm(tlm.lat, tlm.lon, 30, 'Marker', 'o', ... - 'MarkerEdgeColor', [0 0 0], 'MarkerFaceColor', col_ce, ... - 'DisplayName', 'cetacean event'); - -% add legend -legend(h, 'Location', 'eastoutside', 'FontSize', 14) - -% add title -title(sprintf('%s %s', upper(CONFIG.glider), CONFIG.mission), ... - 'Interpreter', 'none'); - -% save -saveName = fullfile(CONFIG.path.analysis, 'cetaceanEvents', ... - ['map_tritonEvents_' CONFIG.gmStr]); -set(baseFig,'renderer','Painters'); -exportgraphics(baseFig, [saveName,'.jpg'], 'Resolution', 300); -exportgraphics(baseFig, [saveName, '.pdf']) -% with export_fig from FEX, can export as vector (with lower res bathy) -% export_fig([saveName '_ef_painters.pdf'], '-pdf', '-painters'); diff --git a/agate/scratch/workflow_processPositionalData_sg639_MHI_Apr2023.m b/agate/scratch/workflow_processPositionalData_sg639_MHI_Apr2023.m deleted file mode 100644 index 5bdb3da..0000000 --- a/agate/scratch/workflow_processPositionalData_sg639_MHI_Apr2023.m +++ /dev/null @@ -1,113 +0,0 @@ -% WORKFLOW_PROCESSPOSITIONALDATA.M -% Process glider positional data at the end of a mission -% -% Description: -% This script provides a workflow for processing Seaglider positional -% data after the end of a mission. It reads in basestation-generated -% .nc files and reorganizes the data into two output tables: -% -% gpsSurfT - gps surface table -% locCalcT - calculated location table (dead reckoned track) -% Both tables are saved as .mat and .csv -% -% It requires an agate configuration file during agate initialization -% -% Notes -% -% See also -% -% -% Authors: -% S. Fregosi -% -% FirstVersion: 21 April 2023 -% Updated: 09 March 2024 -% -% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -% initialize agate -agate secret/agate_config_sg639_MHI_Apr2023.cnf -global CONFIG - -%% extract positional data -[gpsSurfT, locCalcT] = extractPositionalData(CONFIG, 1); -% 0 in plotOn argument will not plot 'check' figures, but change to 1 to -% plot basic figures for output checking - -% save as .mat and .csv -save(fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.mat']), 'gpsSurfT'); -writetable(gpsSurfT,fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.csv'])) - -save(fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat']),'locCalcT'); -writetable(locCalcT, fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.csv'])); - -%% load previously extracted positional data - -% load locCalcT if not already loaded -if ~exist('locCalcT', 'var') - load(fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_locCalcT.mat'])) -end - -% load gpsSurfT if not already loaded -if ~exist('gpsSurfT', 'var') - load(fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_gpsSurfaceTable.mat'])) -end -%% plot sound speed profile - -plotSoundSpeedProfile(CONFIG, locCalcT); -% save as .png and .pdf -exportgraphics(gcf, fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_SSP.png'])) -exportgraphics(gcf, fullfile(CONFIG.path.mission, 'profiles', ... - [CONFIG.glider, '_', CONFIG.mission, '_SSP.pdf'])) - -%% BELOW SECTIONS ARE NOT YET OPERATIONAL -% Below called functions are in the 'drafts' folder and need to be adapted -% for WISPR2 and also updated to new CONFIG system of agate - -%% extract acoustic system status for each dive and sample time - -% *** NEEDS WORK! *** to be updated to deal with WISPR2 and speed up with -% PMARXL using .eng files rather than having to read in list of sound files - -% % this looks at each recorded file timestamp to populate a 'pam' column -% % that is added to locCalcT and gpsSurfT that specifies the status of the -% % pam system for each entry. -% -% fileLength = 600; % in seconds -% dateFormat = 'yyMMdd-HHmmss'; -% dateStart = 1; % what part of file name starts the date format -% -% [gpsSurfT, locCalcT, pam] = extractPAMStatusByFile(gldr, lctn, dplymnt, ... -% fileLength, dateFormat, dateStart, gpsSurfT, locCalcT); -% % saved automatically gpsSurfTable_pam.mat and locCalcT_pam.mat and -% % _pamByFile.mat - -[gpsSurfT, locCalcT, pam] = extractPAMStatusByFile(CONFIG, gpsSurfT, locCalcT); - -%% extract positional data for each sound file -% -% secs = 180; -% -% filePosits = extractPositsPerPAMFile(gldr, lctn, dplymnt, ... -% pam, locCalcT,secs, path_profiles); -% -% % this saves _pamFilePosits.mat and .csv - -%% extract positional data and acoustic effort by minute - -% % create byMin, minPerHour, minPerDay matrices for full experiment extent -% % will need this for comparison down the line -% [pamByMin, pamMinPerHour, pamMinPerDay] = ... -% calcPAMEffort(gldr, lctn, dplymnt, expLimits, gpsSurfT, path_profiles); -% % this saves _pamByMin.mat -% -% - diff --git a/agate/settings/agate_config_example.cnf b/agate/settings/agate_config_example.cnf index 0775cd5..4d1b38b 100644 --- a/agate/settings/agate_config_example.cnf +++ b/agate/settings/agate_config_example.cnf @@ -3,15 +3,15 @@ % Guide for creating configuration files: % https://sfregosi.github.io/agate-public/configuration.html -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% %%% TEMPLATE - PUT SURVEY METADATA HERE %%% -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%% TEMPLATE - PUT MISSION METADATA HERE %%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % %%%% REQUIRED CONFIG PARAMETERS %%%%%%%%%%% % define glider and mission information -CONFIG.glider = 'sgXXX'; -CONFIG.mission = 'Location_Mon20XX'; % defined in sg_calib_constants.m -CONFIG.sgVer = XX.XX; % 66.12 or 67.0 +CONFIG.glider = 'sgXXX'; % glider serial number +CONFIG.mission = 'Location_Mon20XX'; % mission identifier defined in sg_calib_constants.m +CONFIG.sgVer = XX.XX; % 66.12 or 67.0, firmware version displayed on glider start up CONFIG.tmd = XX; % target mission duration in days % define paths @@ -40,11 +40,23 @@ CONFIG.pm.activeCard = 0; % current active card; optional for sgVer 66.12 % necessary to update throughout mission for sgVer 67.0 CONFIG.pm.numCards = 4; % total number loaded SD cards % after mission is complete, set below for raw .dat file conversion -CONFIG.pm.convert = 0; % 0 during mission, set 1 to run conversion +CONFIG.pm.convert = 0; % 0 during mission or analysis, set 1 to run conversion CONFIG.pm.convCnfFile = fullfile(CONFIG.path.mission, 'PMARConvert_sgXXX_Location_MonYYYY.cnf'); +% after mission is complete, set below for data processing/analysis +CONFIG.pm.fileLength = 600; % in seconds +CONFIG.pm.sampleRate = 180260; % sample rate +CONFIG.pm.dateFormat = 'yyMMdd-HHmmss.SSS'; +CONFIG.pm.dateStart = 19; % what character of file name starts the date format +% example file name: sg639_MHI_Apr2023_230411-185050.484.wav % WISPR settings +% only needed after mission is complete (no piloting functionality for WISPR) CONFIG.ws.loggers = 0; % 1 for active, 0 for inactive +CONFIG.ws.fileLength = 60; % in seconds +CONFIG.ws.sampleRate = 180000; % sample rate +CONFIG.ws.dateFormat = 'yyMMdd_HHmmss'; +CONFIG.ws.dateStart = 7; % what character of file name starts the date format +% example file name: WISPR_230504_202206.wav % %%%% OPTIONAL - plotting %%%%%%%%%%%%%%%%%% % maps @@ -64,7 +76,7 @@ CONFIG.map.scalePos = [0 0]; % optional arguments for scale bar location CONFIG.map.scaleMajor = [0:50:100]; CONFIG.map.scaleMinor = [0:12.5:25]; -% general plotting settings +% piloting plot settings % pre allocate fig numbers so figures are updated rather than created new CONFIG.plots.figSeed = 111; % suggest using the glider serial e.g., 639 CONFIG.plots.figNumList = CONFIG.plots.figSeed + [0 1 2 3 4 5 6 7 8 9]; diff --git a/agate/settings/pmarConvert_example.cnf b/agate/settings/pmarConvert_example.cnf index c2a12e6..ba0d285 100644 --- a/agate/settings/pmarConvert_example.cnf +++ b/agate/settings/pmarConvert_example.cnf @@ -3,9 +3,9 @@ % Guide for creating configuration files: % https://sfregosi.github.io/agate-public/configuration.html -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% %%% TEMPLATE - PUT SURVEY METADATA HERE %%% -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%% TEMPLATE - PUT MISSION METADATA HERE %%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % configuration parameters here, descriptions are below. diff --git a/agate/settings/secret/README-secret.md b/agate/settings/secret/README-secret.md index e589121..fafcdfe 100644 --- a/agate/settings/secret/README-secret.md +++ b/agate/settings/secret/README-secret.md @@ -1 +1,3 @@ -This folder is ignored by GitHub. Any files stored here will ONLY be kept locally and will not be uploaded to GitHub.com, but they will not be under version control. \ No newline at end of file +This folder is ignored by GitHub. +Any files stored here will ONLY be kept locally and will not be uploaded to +GitHub.com. They will not be under version control. \ No newline at end of file diff --git a/agate/settings/wisprConvert_example.cnf b/agate/settings/wisprConvert_example.cnf index 5b6d162..1cd074c 100644 --- a/agate/settings/wisprConvert_example.cnf +++ b/agate/settings/wisprConvert_example.cnf @@ -1,8 +1,8 @@ % configuration file for convertWispr.m -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -% %%% TEMPLATE - PUT SURVEY METADATA HERE %%% -% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% %%% TEMPLATE - PUT MISSION METADATA HERE %%% +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % configuration settings here, descriptions are below. diff --git a/agate/utils/calcPAMEffort.m b/agate/utils/calcPAMEffort.m new file mode 100644 index 0000000..b8c392e --- /dev/null +++ b/agate/utils/calcPAMEffort.m @@ -0,0 +1,160 @@ +function [pamByMin, pamMinPerHour, pamMinPerDay, pamHrPerDay] = ... + calcPAMEffort(CONFIG, gpsSurfT, pamFiles, pamByDive, expLimits) +%CALCPAMEFFORT Calculates acoustic recording effort by minute, hour, day +% +% Syntax: +% [GPSSURFT, LOCCALCT, pamFiles] = CALCPAMEFFORT(CONFIG, GPSSURFT, PAMFILES, PAMBYDIVE, EXPLIMITS) +% +% Description: +% Summarizes recording effort in several ways by creating tables of +% all possible recording minutes, hours, and days, and quantifying +% how many of each of those bins contain recordings. The assessment +% of minutes is someone imperfect because some minutes only contain +% partial recordings (if a file ends within that minute) and some +% minutes are missed (if a recording starts partway through a minute) +% +% Experiment limits can be defined if multiple instruments were +% deployed and you want to compare across the maximum deployment time +% for all of them. Optionally can be left out and the bins will just +% populate from the first sound file to the end of the last file +% +% Inputs: +% CONFIG agate mission configuration file with relevant mission and +% glider information. Minimum CONFIG fields are 'glider', +% 'mission', 'path.mission', logger field (either 'pm' or +% 'ws') and logger sub fields 'fileLength', 'dateStart', +% 'dateFormat' +% See exaxmple config file and config file help for more +% detail on each field: +% https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf +% https://sfregosi.github.io/agate-public/configuration.html#mission-configuration-file +% gpsSurfT [table] glider surface locations exported from +% extractPositionalData +% pamFiles [table] name, start and stop time and duration of all +% recorded sound files, created with extractPAMStatus +% pamByDive [table] summary of recording start and stop, number of +% files for each dive. Includes dive start and stop times +% and offset of start and stop of pam relative to dive +% times, created with extractPAMStatus +% expLimits [vector] two datetimes defining the start and end of an +% 'experiment' to set limits of the maximum possible +% recording times +% +% Outputs: +% pamByMin [table] one minute bins with 1 for recordings during +% this minute, 0 for no recordings this minute, and +% NaNs if the glider was at the surface or not deployed +% pamMinPerHour [table] one hour bins with the total number of +% minutes of that hour with recordings +% pamMinPerDay [table] daily bins with total number of minutes with +% recordings per day +% pamHrPerDay [table] daily bins with total hours with recordings +% each day. This is the total minutes/60 so is total +% complete hours, not the number of hour bins with any +% partial amount of recording +% +% Examples: +% +% See also EXTRACTPAMSTATUS +% +% Authors: +% S. Fregosi +% +% FirstVersion: ?? +% Updated: 12 July 2024 +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +if nargin < 5 % default 'experiment time' is deploy start/end + expLimits(1) = dateshift(gpsSurfT.startDateTime(1), 'start', 'minute'); + expLimits(2) = dateshift(gpsSurfT.endDateTime(end), 'end', 'minute'); +end + +% build empty table for whole deployment +dm = (expLimits(1):minutes(1):expLimits(2))'; + +pamByMin = table; +pamByMin.min = dm; + +% build minutes per hour empty table +dh = (dateshift(expLimits(1), 'start', 'hour'):hours(1): ... + dateshift(expLimits(2), 'start', 'hour'))'; +pamMinPerHour = table; +pamMinPerHour.hour = dh; + +% build minutes per day empty table +ddm = (dateshift(expLimits(1),'start','day'):days(1):dateshift(expLimits(2), ... + 'start', 'day'))'; +pamMinPerDay = table; +pamMinPerDay.day = ddm; + +% build hours per day empty table +ddh = (dateshift(expLimits(1),'start','day'):days(1):dateshift(expLimits(2), ... + 'start', 'day'))'; +pamHrPerDay = table; +pamHrPerDay.day = ddh; + +fprintf(1,'Calculating PAM status by min: %s\n', CONFIG.glider) + +% pamCheck = [pamByDive.pamStart pamByDive.pamEnd]; % this works when not duty cycling +pamCheck = [pamFiles.start pamFiles.stop]; +diveCheck = [pamByDive.diveStart pamByDive.diveStop]; + +% by Minute - NaN if not deployed or at surface, 0 if PAM OFF, 1 if ON +for f = 1:length(dm) + dc = dm(f); + % is this minute within a dive? + [rD, ~] = find(isbetween(dc, diveCheck(:,1), diveCheck(:,2))); + if isempty(rD) % if not, put NaN + pamByMin.pam(f,1) = nan; + end + % is PAM on in this minute? + [rP, ~] = find(isbetween(dc, pamCheck(:,1), pamCheck(:,2))); + if ~isempty(rP) + pamByMin.pam(f,1) = 1; + end +end +fprintf(1, '%s: %i minutes with PAM on\n', CONFIG.glider, ... + sum(pamByMin.pam, 'omitnan')); +% this is not perfect...not always full minutes (at end of a recording +% and misses some partial minutes (At the start of a recording) + +% by Hour +for f = 1:length(dh) + dc = dh(f); + hourTmp = pamByMin.pam(isbetween(pamByMin.min, dc, ... + dc + minutes(59) + seconds(59))); + pamMinPerHour.pam(f,1) = sum(hourTmp, 'omitnan'); +end +pamMinPerHour.pam(pamMinPerHour.pam == 0) = nan; % if all zeros, make nan +fprintf(1, '%s: %i partial hours with PAM on, total %.2f hours\n', ... + CONFIG.glider, sum(~isnan(pamMinPerHour.pam)), ... + sum(pamMinPerHour.pam, 'omitnan')/60); + +% by Day +% Minutes per day +for f = 1:length(ddm) + dc = ddm(f); + dayTmp = pamByMin.pam(isbetween(pamByMin.min, dc, ... + dc + minutes(1439) + seconds(59))); + pamMinPerDay.pam(f,1) = sum(dayTmp, 'omitnan'); +end +pamMinPerDay.pam(pamMinPerDay.pam == 0) = nan; +% Hours per day +for f = 1:length(ddh) + dc = ddh(f); + dayTmp = pamMinPerHour.pam(isbetween(pamMinPerHour.hour, dc, ... + dc + minutes(1439) + seconds(59))); + pamHrPerDay.pam(f,1) = round(sum(dayTmp, 'omitnan')/60, 2); +end +pamHrPerDay.pam(pamHrPerDay.pam == 0) = nan; + +fprintf(1, '%s: %i partial days with PAM on, total %.2f days\n', ... + CONFIG.glider, sum(~isnan(pamMinPerDay.pam)), ... + sum(pamMinPerDay.pam, 'omitnan')/(60*24)); + +end + + + diff --git a/agate/utils/createBasemap.m b/agate/utils/createBasemap.m index c13e5a1..1cba71e 100644 --- a/agate/utils/createBasemap.m +++ b/agate/utils/createBasemap.m @@ -1,8 +1,8 @@ -function [baseFig] = createBasemap(CONFIG, bathyOn, figNum, outFig) +function [baseFig] = createBasemap(CONFIG, bathyOn, contourOn, figNum) % CREATEBASEMAP Create a basemap of the bathymetry for the mission area % % Syntax: -% OUTPUT = CREATEBASEMAP(CONFIG, OUTFIG) +% BASEFIG = CREATEBASEMAP(CONFIG, BATHYON, CONTOURON, FIGNUM, OUTFIG) % % Description: % Function to create a basemap for a glider mission, using the lat @@ -10,23 +10,24 @@ % include a north arrow and scale bar. Optional to plot bathymetry % data (if available) otherwise will just plot land (specific to US % states at this time). It can be saved as a .fig file that can be -% added to (add glider path, labels, and acoustic encounters). +% added to (add glider path, labels, and acoustic encounters). % % Inputs: -% CONFIG [struct] mission/agate global configuration variable. +% CONFIG [struct] mission/agate configuration variable. % Required fields: CONFIG.map entries % bathyOn [double] set to 1 to plot bathymetry or 0 to only plot % land +% contourOn [double] set to 1 to plot bathymetry contours or 0 for +% no contour lines % figNum [double] optional to specify figure number so won't % create repeated versions when updated -% outFig [string] optional argument to save the .fig % % Outputs: % baseFig [handle] figure handle % % Examples: -% % Create a basemap that includes bathymetry, do not save output -% createBasemap(CONFIG, 1); +% % Create a basemap that includes bathymetry but no contour lines +% baseFig = createBasemap(CONFIG, 1, 0); % % See also % @@ -34,7 +35,7 @@ % S. Fregosi % % FirstVersion: 09 March 2024 -% Updated: +% Updated: 06 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -42,16 +43,10 @@ %%%%%%%%%%%%%% % for testing % figNum = 82; -% outFig = []; %%%%%%%%%%%%%% -% save output fig or not -if nargin < 4 - outFig = []; -end - % specify a figure number -if nargin < 3 +if nargin < 4 figNum = []; end @@ -63,8 +58,8 @@ baseFig = figure(figNum); end -mapFigPosition = [100 100 800 600]; -baseFig.Position = mapFigPosition; +% mapFigPosition = [100 100 800 600]; +% baseFig.Position = mapFigPosition; baseFig.Name = 'Base map'; % clear figure @@ -83,7 +78,7 @@ % add north arrow - if location specified if isfield(CONFIG.map, 'naLat') && isfield(CONFIG.map, 'naLon') CONFIG.map.na = northarrow('latitude', CONFIG.map.naLat, 'longitude', ... - CONFIG.map.naLon, 'FaceColor', [1 1 1], 'EdgeColor', [1 1 1]); + CONFIG.map.naLon, 'FaceColor', [1 1 1], 'EdgeColor', [0 0 0]); end if isfield(CONFIG.map, 'scalePos') scaleruler on @@ -97,37 +92,50 @@ end -%% plot bathymetry - slow step +%% plot bathymetry and/or contours - slow step + +if contourOn == 1 || bathyOn == 1 % if either, load raster data -if bathyOn == 1 if isfield(CONFIG.map, 'bathyFile') bathyFile = CONFIG.map.bathyFile; else % prompt to choose file [fn, path] = uigetfile(fullfile(CONFIG.path.shp, '*.tif;*.tiff'), ... 'Select etopo raster file'); bathyFile = fullfile(path, fn); - end [Z, refvec] = readgeoraster(bathyFile, 'OutputType', 'double', ... 'CoordinateSystemType', 'geographic'); [Z, refvec] = geocrop(Z, refvec, CONFIG.map.latLim, CONFIG.map.lonLim); Z(Z >= 10) = 100; - geoshow(Z, refvec, 'DisplayType', 'surface', ... - 'ZData', zeros(size(Z)), 'CData', Z); - - cmap = cmocean('ice'); - cmap = cmap(150:256,:); - colormap(cmap) - % matlab renamed caxis to clim in R2022a...so try both - try clim([-6000 0]); catch caxis([-6000 0]); end %#ok - brighten(.4); - - [~,~] = contourm(Z, refvec, [-5000:1000:1000], 'LineColor', [0.6 0.6 0.6]); %#ok - [~,~] = contourm(Z, refvec, [-1000 -1000], 'LineColor', [0.3 0.3 0.3], ... - 'LineWidth', 0.8); - [~,~] = contourm(Z, refvec, [-900:100:0], 'LineColor', [0.8 0.8 0.8]); %#ok - [~,~] = contourm(Z, refvec, [-500 -500], 'LineColor', [0.6 0.6 0.6]); + + if bathyOn == 1 % plot it + + Z(Z >= 10) = 100; + geoshow(Z, refvec, 'DisplayType', 'surface', ... + 'ZData', zeros(size(Z)), 'CData', Z); + + cmap = cmocean('ice'); + cmap = cmap(150:256,:); + colormap(cmap) + % matlab renamed caxis to clim in R2022a...so try both + try clim([-6000 0]); catch caxis([-6000 0]); end %#ok + brighten(.4); + + % colorbar for bathymetry is too unreliable +% cb = colorbar; +% cb.Location = 'eastoutside'; +% cb.Label.String = 'depth [m]'; +% cbPosit = cb.Position; + end + + if contourOn == 1 % plot it + [~,~] = contourm(Z, refvec, [-5000:1000:1000], 'LineColor', [0.6 0.6 0.6]); %#ok + [~,~] = contourm(Z, refvec, [-1000 -1000], 'LineColor', [0.3 0.3 0.3], ... + 'LineWidth', 0.8); + [~,~] = contourm(Z, refvec, [-900:100:0], 'LineColor', [0.8 0.8 0.8]); %#ok + [~,~] = contourm(Z, refvec, [-500 -500], 'LineColor', [0.6 0.6 0.6]); + end end %% plot land @@ -135,11 +143,5 @@ 'BoundingBox', [CONFIG.map.lonLim' CONFIG.map.latLim']); geoshow(states, 'FaceColor', [0 0 0], 'EdgeColor', 'k') - -%% save as .fig -if ~isempty(outFig) - savefig(outFig); -end - end diff --git a/agate/utils/downloadBasestationFiles.m b/agate/utils/downloadBasestationFiles.m index c78638c..b7c9f99 100644 --- a/agate/utils/downloadBasestationFiles.m +++ b/agate/utils/downloadBasestationFiles.m @@ -17,7 +17,7 @@ function downloadBasestationFiles(CONFIG, path_bsLocal) % download_files_cache.txt % % Inputs: -% CONFIG Deployment parameters - glider serial num, survey +% CONFIG Deployment parameters - glider serial num, mission % ID, pmcard % path_bsLocal Path to directory to save the downloaded files % locally, e.g., path_bsLocal = @@ -39,7 +39,7 @@ function downloadBasestationFiles(CONFIG, path_bsLocal) % % FirstVersion: 7/22/2016. % Originally for AFFOGATO project/CatBasin deployment -% Updated: 02 June 2023 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.9.0.1524771 (R2020b) Update 2 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/agate/utils/extractPAMFilePosits.m b/agate/utils/extractPAMFilePosits.m new file mode 100644 index 0000000..7d5b5a2 --- /dev/null +++ b/agate/utils/extractPAMFilePosits.m @@ -0,0 +1,78 @@ +function pamFilePosits = extractPAMFilePosits(pamFiles, locCalcT, timeBuffer) +%EXTRACTPAMFILEPOSITS Extracts glider location for each acoustic file +% +% Syntax: +% PAMFILEPOSITS = EXTRACTPAMFILEPOSITS(CONFIG, PAMFILES, LOCCALCT) +% +% Description: +% Extract glider positional data for each acoustic file. Includes +% depth, lat, lon, vertical velocity, horizontal velocity, speed, and +% sound speed. Uses glider data sample closest to start of sound +% file, up to an optional buffer time specified as 'timeBuffer'. If +% not positional data is available within that buffer, no position is +% provided for that file. +% +% Inputs: +% pamFiles [table] name, start and stop time and duration of all +% recorded sound files +% locCalcT [table] glider fine scale locations exported from +% extractPositionalData +% timeBuffer [double] optional argument to specify time (sec) around +% agiven file you are willing to accept a position. +% Default is 180 sec +% +% Outputs: +% pamFilePosits [table] glider positional info at the start of each +% acoustic file +% +% Examples: +% +% See also +% +% +% Authors: +% S. Fregosi +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% +% FirstVersion: 26 July 2018 +% Updated: 11 July 2024 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +if nargin < 3 + timeBuffer = 180; % in seconds +end + +% set up empty output table +pamFilePosits = table; +pamFilePosits.fileName = pamFiles.name; +pamFilePosits.fileStart = pamFiles.start; + +% set up a row of nan's for situations with no sample match +noMatch = locCalcT(1,:); +noMatch(1,[1:2 4:end]) = array2table(NaN(1,width(locCalcT)-1)); +noMatch.dateTime = NaT; + +% fileDate = datetime(fileName(1:end-4),'InputFormat','yyMMdd-HHmmss'); +for f = 1:height(pamFilePosits) + % find the closest positional data +% [m,i] = min(abs(pamFilePosits.fileStart(f)-locCalcT.dateTime)); + i = find(pamFilePosits.fileStart(f) <= locCalcT.dateTime, 1, 'first'); + m = pamFilePosits.fileStart(f) - locCalcT.dateTime(i); + if m < seconds(timeBuffer) % within buffer specified by timeBuffer + pamFilePosits(f,3:width(locCalcT)+2) = locCalcT(i,:); + elseif m > seconds(timeBuffer) % outside buffer + fprintf(1, 'file %s closest time is > %i seconds\n', ... + pamFilePosits.fileStart(f), timeBuffer) + pamFilePosits(f, 3:width(locCalcT) + 2) = noMatch; + else % error?? + fprintf(1, 'file %s error\n', pamFilePosits.fileStart(f)) + pamFilePosits(f,3:width(locCalcT)+2) = noMatch; + end +end + +% add locCalc column names and clean up a bit +pamFilePosits.Properties.VariableNames(3:end) = locCalcT.Properties.VariableNames; +pamFilePosits.time = []; +pamFilePosits.Properties.VariableNames(4) = {'sampleDateTime'}; + +end diff --git a/agate/utils/extractPAMStatus.m b/agate/utils/extractPAMStatus.m new file mode 100644 index 0000000..045228c --- /dev/null +++ b/agate/utils/extractPAMStatus.m @@ -0,0 +1,271 @@ +function [gpsSurfT, locCalcT, pamFiles, pamByDive] = extractPAMStatus(... + CONFIG, gpsSurfT, locCalcT) +%EXTRACTPAMSTATUSBYFILE Extracts PAM system on/off information from sound files +% +% Syntax: +% [GPSSURFT, LOCCALCT, pamFiles] = EXTRACTPAMSTATUS(CONFIG, GPSSURFT, LOCCALCT) +% +% Description: +% Function to read info (audioinfo) and compile timing information +% from all recorded acoustic files. User is prompted to select a +% folder of .wav (or .flac?) files which will be opened one at a +% time. Because this is slow, best to use the lowest frequency +% dataset available (if it was downsampled) or from a local hard +% drive rather than data on a server. The file start time is +% extracted from the filename and the duration is pulled from the +% file itself so stop time can be calculated. A list of all files and +% timing info is output to the 'pamFiles' table. +% +% 'pamFiles' is then used to populate an additional 'pam' column in +% locCalcT with a 0 (off) or 1 (on) for the pam system status at each +% glider positional sample. Several columns are also added to +% gpsSurfT with the pam duration and number of files for each dive. A +% a separate 'pamByDive' summary table is also created with dive and +% recording system timing info. +% +% Inputs: +% CONFIG agate mission configuration file with relevant mission and +% glider information. Minimum CONFIG fields are 'glider', +% 'mission', 'path.mission', logger field (either 'pm' or +% 'ws') and logger sub fields 'fileLength', 'dateStart', +% 'dateFormat' +% See exaxmple config file and config file help for more +% detail on each field: +% https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf +% https://sfregosi.github.io/agate-public/configuration.html#mission-configuration-file +% gpsSurfT [table] glider surface locations exported from +% extractPositionalData +% locCalcT [table] glider fine scale locations exported from +% extractPositionalData +% +% Outputs: +% gpsSurfT [table] glider surface locations, from GPS, one per +% dive. Input gpsSurfT is updated to now include a pam +% column with the minutes of PAM recording for that dive. +% Origingal columns include dive start and end +% time/lat/lon, dive duration, depth average current, +% average speed over ground as northing and easting, +% calculated by the hydrodynamic model or the glide slope +% model +% locCalcT [table] Glider calculated locations underwater every +% science file sampling interval. Input locCalcT is updated +% to include a pam column that has a 1 for pam system on or +% 0 for pam system off for each location entry. Original +% instantaneous flight details and includes columns +% for time, lat, lon from hydrodynamic and glide slope +% models, displacement from both models, temperature, +% salinity, density, sound speed, glider vertical and +% horizontal speed (from both models), pitch, glide +% angle, and heading +% pamFiles [table] name, start and stop time and duration of all +% recorded sound files +% pamByDive [table] summary of recording start and stop, number of +% files for each dive. Includes dive start and stop times +% and offset of start and stop of pam relative to dive +% times +% +% Examples: +% +% See also EXTRACTPOSITIONALDATA +% +% TO DO: +% - build in option for FLAC +% +% Authors: +% S. Fregosi +% +% FirstVersion: ?? +% Updated: 11 July 2024 +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%% initialization + +% check acoustic system +if isfield(CONFIG, 'pm') && CONFIG.pm.loggers == 1 + loggerType = 'PMARXL'; +elseif isfield(CONFIG, 'ws') && CONFIG.ws.loggers == 1 + loggerType = 'WISPR'; +else + fprintf(1, 'Unknown acoustic logger type. Exiting\n'); + return +end + +% if specified in CONFIG, get file length (in seconds) and sample rate +switch loggerType + case 'PMARXL' + if isfield(CONFIG.pm, 'fileLength') + fileLength = CONFIG.pm.fileLength; + % sampleRate = CONFIG.pm.sampleRate; + else + fprintf(1, ['No file length specified in .cnf, using PMARXL ', ... + 'default fileLength = 600 s and sampleRate = 180260\n']); + fileLength = 600; + % sampleRate = 180260; + end + if isfield(CONFIG.pm, 'dateStart') + dateStart = CONFIG.pm.dateStart; + else + fprintf(1, ['No dateStart specified in .cnf. Must specify ', ... + 'character where date string starts in file name.', ... + 'Exiting...']) + return + end + if isfield(CONFIG.pm, 'dateFormat') + dateFormat = CONFIG.pm.dateFormat; + else + fprintf(1, ['No dateFormat specified in .cnf. Must specify ', ... + 'format of date string in file name.', ... + 'Exiting...']) + return + end + + case 'WISPR' + if isfield(CONFIG.ws, 'fileLength') + fileLength = CONFIG.ws.fileLength; + else + fprintf(1, ['No file length specified in .cnf, using WISPR ', ... + 'default fileLength = 60 s and sampleRate = 180 kHz\n']); + fileLength = 60; + end + if isfield(CONFIG.ws, 'dateStart') + dateStart = CONFIG.ws.dateStart; + else + fprintf(1, ['No dateStart specified in .cnf. Must specify ', ... + 'character where date string starts in file name.', ... + 'Exiting...']) + return + end + if isfield(CONFIG.ws, 'dateFormat') + dateFormat = CONFIG.ws.dateFormat; + else + fprintf(1, ['No dateFormat specified in .cnf. Must specify ', ... + 'format of date string in file name.', ... + 'Exiting...']) + return + end +end + +% specify deployment date and time to ignore all files before that +% (sometimes test files recorded in lab are in dataset) +deplDate = gpsSurfT.startDateTime(1); + +% select folder with sound files and where to save +% path_in = uigetdir('G:\','Select base folder'); +% pick 1 kHz data if available because it will run faster. +path_acous = uigetdir('C:\', ['Select folder with acoustic data. ' ... + 'Lower sample rate and local hard drive will run fastest.']); + +%% Read in files and extract duration information +files = dir(fullfile(path_acous, '*.wav')); +if isempty(files) + fprintf('No .wav files found...aborting\n'); + return + % *****later build in switch to flac?? + % files=dir([path_flac '*.flac']); +end + +pamFiles = table; +pamFiles.name = cell(length(files), 1); +pamFiles.start = NaT(length(files), 1, 'Format', 'yyyy-MM-dd HH:mm:ss.SSS'); +pamFiles.stop = NaT(length(files), 1, 'Format', 'yyyy-MM-dd HH:mm:ss.SSS'); + +shortfiles = []; +fprintf(1,'%i files:\n', length(files)); + +% make matrix with start and end times for all PAM files +for f = 1:length(files) + % calc file duration in sec using sampling rate..slow but works - more accurate + try + wavInfo = audioinfo(fullfile(path_acous, files(f,1).name)); + files(f,1).dur = wavInfo.TotalSamples./wavInfo.SampleRate; + if files(f,1).dur < fileLength + shortfiles = [shortfiles; f wavInfo.TotalSamples]; %#ok + fprintf(1,'%s is short: %i samples, %.2f seconds\n', ... + files(f,1).name, wavInfo.TotalSamples, files(f,1).dur); + end + % get start timing information from file name + pamFiles.name{f} = files(f).name; + dtIdx = dateStart:length(dateFormat) + dateStart - 1; + pamFiles.start(f) = datetime(files(f).name(dtIdx), 'InputFormat', ... + dateFormat); + pamFiles.stop(f) = datetime(pamFiles.start(f,1) + ... + seconds(files(f,1).dur), 'Format', 'yyyy-MM-dd HH:mm:ss.SSS'); + pamFiles.dur(f) = seconds(files(f,1).dur); + + catch % if there is some issue reading a file + fprintf(1, '%s is corrupt\n', files(f,1).name); + end + if rem(f,1000)==0; fprintf(1,'%i DONE\n',f);end % counter +end + +% remove all before deployment date +[r, ~] = find(pamFiles.start < deplDate, 1, 'last'); +if ~isempty(r) + pamFiles = pamFiles(r+1:end,:); +end + +%% Specify 1's and 0s per locCalcT row +% 1 if pam on, 0 if off + +locCalcT.pam = zeros(height(locCalcT),1); +fprintf(1,'%s - %d science samples:\n', [CONFIG.glider '_' CONFIG.mission], ... + height(locCalcT)); +fprintf(1, '\n%3d', floor((height(locCalcT))/8000)); +for f = 1:height(locCalcT) + idx = find(isbetween(locCalcT.dateTime(f), pamFiles.start, ... + pamFiles.stop), 1); + if ~isempty(idx) + locCalcT.pam(f) = 1; +% else % for debugging +% fprintf(1, 'no idx match %s\n', locCalcT.dateTime(f)); + end + clear idx + + % fprintf(1, '.'); + if rem(f, 100) == 0 + fprintf(1, '.'); + end + if rem(f, 8000) == 0 + fprintf(1, '\n%3d', floor((height(locCalcT) - f)/8000)); + end +end +fprintf(1, '\n'); + +% % plotting test +% plotDiveProfile(locCalcT) + + +%% duration per dive + +pamByDive = table; +pamByDive.dive = gpsSurfT.dive; +pamByDive.diveStart = gpsSurfT.startDateTime; +pamByDive.diveStop = gpsSurfT.endDateTime; +pamByDive.numFiles = nan(height(pamByDive),1); + +for f = 1:height(pamByDive) + [r, ~] = find(isbetween(pamFiles.start, pamByDive.diveStart(f), ... + pamByDive.diveStop(f))); + if ~isempty(r) + pamByDive.numFiles(f,1) = length(r); + pamByDive.pamDur(f,1) = sum(pamFiles.dur(r)); + pamByDive.pamStart(f,1) = pamFiles.start(r(1)); + pamByDive.pamStop(f,1) = pamFiles.stop(r(end)); + end +end + +% calc time between the dive start and the pam start, pam stop/dive stop +pamByDive.lagStart = pamByDive.pamStart - pamByDive.diveStart; +pamByDive.lagStop = pamByDive.diveStop - pamByDive.pamStop; + +% append to gpsSurfT and save +gpsSurfT.pamDur = pamByDive.pamDur; +gpsSurfT.pamNumFiles = pamByDive.numFiles; +gpsSurfT.pamStart = pamByDive.pamStart; +gpsSurfT.pamStop = pamByDive.pamStop; + + + + diff --git a/agate/utils/interpolatePlannedTrack.m b/agate/utils/interpolatePlannedTrack.m new file mode 100644 index 0000000..adc6913 --- /dev/null +++ b/agate/utils/interpolatePlannedTrack.m @@ -0,0 +1,115 @@ +function interpTrack = interpolatePlannedTrack(CONFIG, targetsFile, spacing) +% INTERPOLATEPLANNEDTRACK Generate lat/lon points at specified spacing between waypoints +% +% Syntax: +% INTERPTRACK = INTERPOLATEPLANNEDTRACK(CONFIG, TARGETSFILE, SPACING) +% +% Description: +% Detailed description here, please +% Inputs: +% CONFIG [struct] agate configuration settings from .cnf +% targetsFile [string] optional argument to targets file. If no file +% specified, will prompt to select one, and if no path +% specified, will prompt to select path +% [table] alternatively can just reference a targets +% table that has already been read in to the workspace +% spacing [double] optional argument to set spacing between +% interpolated points, in km. Default is 5 km. +% +% Outputs: +% interpTrack [table] lat/lon points interpolated between the +% targets at approx the spacing setting +% +% Examples: +% +% See also TRACK2, PLOTTRACKBATHYPROFILE +% +% Authors: +% S. Fregosi +% +% FirstVersion: 06 August 2024 +% Updated: 07 August 2024 +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +if nargin < 3 + spacing = 5; +end + +% select targetsFile if none specified +if nargin < 2 + [fileName, filePath] = uigetfile([CONFIG.path.mission, '*.*'], ... + 'Select targets file'); + targetsFile = fullfile(filePath, fileName); + fprintf('targets file selected: %s\n', fileName); +end + +% check that targetsFile exists if specified, otherwise prompt to select +if ischar(targetsFile) + if ~exist(targetsFile, 'file') + fprintf(1, 'Specified targetsFile does not exist. Select targets file to continue.\n'); + [fileName, filePath] = uigetfile([CONFIG.path.mission, '*.*'], ... + 'Select targets file'); + targetsFile = fullfile(filePath, fileName); + fprintf('targets file selected: %s\n', fileName); + end + % read in targets file + [targets, ~] = readTargetsFile(CONFIG, targetsFile); +elseif istable(targetsFile) + targets = targetsFile; +end + + +% setup output matrix +it = []; + +% matlab track function can create track points between a series of +% waypoints but no way to define size, can only specify number of points +% between each pair of waypoints +% +% track2 operates between just one start/end point and you can specify the +% number of points so can divide total length by spacing to get as close as +% possible to the spacing value, then concatenated + +% loop through each waypoint to create interpolated points based on length +% of each segment between waypoints +for f = 1:height(targets)-1 + pairDist = distance(targets.lat(f), targets.lon(f), ... + targets.lat(f+1), targets.lon(f+1), referenceEllipsoid('wgs 84'))/1000; + npts = round(pairDist/spacing); + tt = track2(targets.lat(f), targets.lon(f), ... + targets.lat(f+1), targets.lon(f+1), [1 0], 'degrees', npts); + it = [it; tt(1:end-1,:)]; % remove the last one so no dupes +end +% add the recv point to the end +it = [it; targets.lat(end) targets.lon(end)]; + +% turn into a table +interpTrack = array2table(it, 'VariableNames', {'latitude', 'longitude'}); +% add in the distance between +interpTrack.dist_km(2:height(interpTrack),1) = distance( ... + interpTrack.latitude(1:end-1), interpTrack.longitude(1:end-1),... + interpTrack.latitude(2:end), interpTrack.longitude(2:end), ... + referenceEllipsoid('wgs 84'))/1000; + +% label actual waypoints (have to round or sometimes misses them) +interpTrack.waypoint(1,1) = targets.name(1); +for f = 1:height(targets) + wpIdx = find(round(interpTrack.latitude, 8) == round(targets.lat(f), 8) & ... + round(interpTrack.longitude, 8) == round(targets.lon(f), 8)); + interpTrack.waypoint(wpIdx) = targets.name(f); +end + +% % testing +% figure(1) +% scatter(it(:,2), it(:,1), 'gs'); +% +% distancesBetweenPoints = distance(it(1:end-1,1),it(1:end-1,2),... +% it(2:end,1),it(2:end,2), referenceEllipsoid('wgs 84')); +% +% figure(3); +% histogram(distancesBetweenPoints/1000,31) +% xlabel('kilometers between points using track2') +% mean(distancesBetweenPoints/1000) + +end \ No newline at end of file diff --git a/agate/utils/mapPlannedTrack.m b/agate/utils/mapPlannedTrack.m index 79ccbee..69242a5 100644 --- a/agate/utils/mapPlannedTrack.m +++ b/agate/utils/mapPlannedTrack.m @@ -1,11 +1,11 @@ function mapPlannedTrack(CONFIG, targetsFile, trackName, bathyOn, figNum) -%MAPPLANNEDTRACK Create static map of planned survey track +%MAPPLANNEDTRACK Create static map of planned mission track % % Syntax: % MAPPLANNEDTRACK(CONFIG, targetsFile, trackName, bathyOn, figNum) % % Description: -% Create a static map of the planned survey track from an input +% Create a static map of the planned mission track from an input % targets file. Optional argument to plot bathymetry (requires etopo % raster from NCEI) and land (requires shape files from Natural Earth % @@ -42,13 +42,13 @@ function mapPlannedTrack(CONFIG, targetsFile, trackName, bathyOn, figNum) % See also MAKETARGETSFILE % % TO DOs: -% - [ ] make possible to plot multiple gliders for one survey +% - [ ] make possible to plot multiple gliders for one mission % % Authors: % S. Fregosi % % FirstVersion: 22 March 2023 -% Updated: 25 May 2023 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/agate/utils/plotDiveProfile.m b/agate/utils/plotDiveProfile.m new file mode 100644 index 0000000..7434aba --- /dev/null +++ b/agate/utils/plotDiveProfile.m @@ -0,0 +1,54 @@ +function plotDiveProfile(locCalcT, savePath) + +if nargin < 2 + savePath = []; +end + +if nargin < 1 % no locCalcT loaded, so select it + [file, path] = uigetfile('.mat','Select locCalcT_pam.mat file'); + % path_in = uigetdir('G:\', 'Select instruments profiles folder'); + % path_in = [path_in '\']; + % path_in = ['G:\score\2015\profiles\' gldr '_' lctn '_' dplymnt '\']; + load([path file]); +end + + +ylims = [-1050 10]; +xlims = datenum([locCalcT.time(1) locCalcT.time(end)]); + +pamOFFC = [0.8 0.8 0.8]; +pamONC = [0 0 0]; +patchColor = [.8 .8 .8]; +% LW = .5; +% cd(path_in) +figure + +h = color_line3(locCalcT.time,-locCalcT.depth,locCalcT.pam,locCalcT.pam); +colormap([pamOFFC; pamONC]); +xData = [floor(locCalcT.time(1)):2:ceil(locCalcT.time(end))]; +set(gca,'xticklabel',{[]},'XTick',xData,'FontSize',14) +dateformat = 'mm/dd'; +datetick('x',dateformat,'keepticks') + +ylabel('depth (m)') +xlabel('date') +ylim(ylims); +xlim(xlims) + +titleString = input('Figure title string: ','s'); +title(titleString,'Interpreter','none'); +set(gca,'FontSize',14) +pbaspect([4 1 1]) +% set this for opening in illustrator + +if ~isempty(savePath) + savefig([savePath titleString '_diveProfile.fig']) + fprintf(1, 'pause to resize. resize fig and hit spacebar\n'); + pause + set(gcf,'Renderer','painters') + print([savePath titleString '_diveProfile.png'],'-dpng') + savefig([savePath titleString '_diveProfile.fig']) +end + +end + diff --git a/agate/utils/plotSoundSpeedProfile.m b/agate/utils/plotSoundSpeedProfile.m index bf45547..703e4da 100644 --- a/agate/utils/plotSoundSpeedProfile.m +++ b/agate/utils/plotSoundSpeedProfile.m @@ -35,7 +35,7 @@ function plotSoundSpeedProfile(CONFIG, locCalcT) global CONFIG figure; -plot(locCalcT.soundVelocity, -locCalcT.depth, 'Color', [0.8 0.8 0.8], ... +plot(locCalcT.soundVelocity, -locCalcT.depth, 'Color', [0.8 0.8 0.8 0.3], ... 'HandleVisibility', 'off') hold on ylim([-1010 10]); diff --git a/agate/utils/plotTrackBathyProfile.m b/agate/utils/plotTrackBathyProfile.m index 88c258c..7c7d077 100644 --- a/agate/utils/plotTrackBathyProfile.m +++ b/agate/utils/plotTrackBathyProfile.m @@ -1,29 +1,41 @@ -function plotTrackBathyProfile(CONFIG, targetsFile, bathyFile, yLine, figNum) +function plotTrackBathyProfile(CONFIG, targetsFile, yLine, figNum) % PLOTTRACKBATHYPROFILE Create bathymetric profile for planned targets % % Syntax: -% OUTPUT = PLOTTRACKBATHYPROFILE(CONFIG, TARGETSFILE) +% OUTPUT = PLOTTRACKBATHYPROFILE(CONFIG, TARGETSFILE, YLINE, FIGNUM) % % Description: % Create a plot of the bathymetric profile along a targets file to % get an overview of the bathymetry the planned track will cover and % identify areas where the bathymetry is less than 1000 m. A targets -% file is read in, interpolated at 0.1 decimal degree resolution, and -% bathymetric depths are extracted from an etopo raster (or other -% specified bathymetric raster). Interpolated points as well as -% depths at actual targets waypoints are plotted. +% file and bathymetry raster are loaded and interpolated to pull the +% seafloor depth along the targets file trackline. Depths along the +% trackline are plotted as well as actual targets waypoints. +% Interpolation between the waypoints is done using the great circle +% distance and the reference ellipsoid specified by the loaded raster +% (WGS84 if using an NCEI ETOPO tiff) +% +% Note: The total distance is slightly different than the distance +% calculated using lldistkm. lldistkm uses the Haversine formula +% which calculates great circle distance of a sphere with radius 6371 +% km. The mapprofile function calculates great circle distance using +% a reference ellipsoid (WGS84 for NCEI ETOPO tiffs) so is slightly +% more accurate but should only be off a few km at the scale of +% typical glider missions. % % Inputs: -% CONFIG [struct] agate global configuration settings from .cnf +% CONFIG [struct] agate configuration settings from .cnf % targetsFile [string] optional argument to targets file. If no file % specified, will prompt to select one, and if no path % specified, will prompt to select path -% yLine [vector] optional argument to set depth to place +% [table] alternatively can just reference a targets +% table that has already been read in to the workspace +% yLine [vector] optional argument to set depth to place % horizontal indicator line; default is 990 m % figNum optional argument defining figure number so it % doesn't keep making new figs but refreshes existing % -% Outputs: +% Outputs: % none, creates figure % % Examples: @@ -34,36 +46,19 @@ function plotTrackBathyProfile(CONFIG, targetsFile, bathyFile, yLine, figNum) % S. Fregosi % % FirstVersion: 10 May 2023 -% Updated: 01 June 2023 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -global CONFIG - % argument checks -if nargin < 5 - figNum = 211; -end if nargin < 4 figNum = 211; - yLine = -990; end -% use default bathy if none specified -if nargin < 3 || isempty(bathyFile) - % try this default - if isfield(CONFIG.path, 'shp') - shpDir = CONFIG.path.shp; - else - shpDir = 'C:\'; - end - bathyFile = fullfile(shpDir, 'etopo', 'ETOPO2022_v1_60s_N90W180_surface.tif'); - if nargin < 3 % also need to set these - figNum = 211; - yLine = -990; - end +if nargin < 3 + figNum = 211; + yLine = -990; end % select targetsFile if none specified @@ -76,71 +71,49 @@ function plotTrackBathyProfile(CONFIG, targetsFile, bathyFile, yLine, figNum) end % check that targetsFile exists if specified, otherwise prompt to select -if ~exist(targetsFile, 'file') - fprintf(1, 'Specified targetsFile does not exist. Select targets file to continue.\n'); - [fileName, filePath] = uigetfile([CONFIG.path.mission, '*.*'], ... - 'Select targets file'); - targetsFile = fullfile(filePath, fileName); - fprintf('targets file selected: %s\n', fileName); - -end - -% read in targets file -[targets, targetsFile] = readTargetsFile(CONFIG, targetsFile); - -% estimate cumulative track length -targets.cumDist_km = zeros(height(targets), 1); -for f = 2:height(targets) - targets.cumDist_km(f) = targets.cumDist_km(f-1) + ... - lldistkm([targets.lat(f-1) targets.lon(f-1)], [targets.lat(f) targets.lon(f)]); +if ischar(targetsFile) + if ~exist(targetsFile, 'file') + fprintf(1, 'Specified targetsFile does not exist. Select targets file to continue.\n'); + [fileName, filePath] = uigetfile([CONFIG.path.mission, '*.*'], ... + 'Select targets file'); + targetsFile = fullfile(filePath, fileName); + fprintf('targets file selected: %s\n', fileName); + end + % read in targets file + [targets, ~] = readTargetsFile(CONFIG, targetsFile); +elseif istable(targetsFile) + targets = targetsFile; end -% interpolate between targets at 0.1 dec deg resolution -ti = table; -ti.lat = interp1(targets.lat, [1:0.1:length(targets.lat)])'; -ti.lon = interp1(targets.lon, [1:0.1:length(targets.lon)]'); -ti.cumDist_km = interp1(targets.cumDist_km, [1:0.1:length(targets.lat)])'; -ti.depth = nan(height(ti), 1); - - -% check that specified bathymetric file exists -if ~exist(bathyFile, 'file') - [fn, path] = uigetfile([shpDir '*.tif;*.tiff'], 'Select etopo .tif file'); +% check for bathy file or select if not specified/doesn't exist +if isfield(CONFIG.map, 'bathyFile') + bathyFile = CONFIG.map.bathyFile; +elseif ~isfield(CONFIG.map, 'bathyFile') || ~exist(bathyFile, 'file') % prompt to choose file + if isfield(CONFIG.path, 'shp') + shpDir = CONFIG.path.shp; + else + shpDir = 'C:\'; + end + [fn, path] = uigetfile(fullfile(shpDir, '*.tif;*.tiff'), ... + 'Select bathymetry raster file'); bathyFile = fullfile(path, fn); end -% read in and crop bathymetry data + +% read in bathymetry data [Z, refvec] = readgeoraster(bathyFile, 'OutputType', 'double', ... 'CoordinateSystemType', 'geographic'); -[Z, refvec] = geocrop(Z, refvec, CONFIG.map.latLim, CONFIG.map.lonLim); - -% Pull out lat/lon vectors from refvec -% use 0.5*cell extent to get midpoints of each cell -Zlat = [refvec.LatitudeLimits(1)+0.5*refvec.CellExtentInLatitude: ... - refvec.CellExtentInLatitude:refvec.LatitudeLimits(2)]'; -Zlat = flipud(Zlat); % have to flip bc small latitudes are at poles -Zlon = [refvec.LongitudeLimits(1)+0.5*refvec.CellExtentInLongitude: ... - refvec.CellExtentInLongitude:refvec.LongitudeLimits(2)]'; - - -% loop through interpolated lat/lons and pull depth at closest Z cell -for f = 1:height(ti) - [mLat, idxLat] = min(abs(Zlat-ti.lat(f))); - [mLon, idxLon] = min(abs(Zlon-ti.lon(f))); - % make sure the mins are below the cell extent - if mLat <= refvec.CellExtentInLatitude && mLon <= refvec.CellExtentInLongitude - ti.depth(f) = Z(idxLat, idxLon); - end -end -% repeat for just the waypoints -targets.depth = nan(height(targets), 1); +% interpolate locations/depths +[zq, distq, latq, lonq] = mapprofile(Z, refvec, targets.lat, targets.lon); +distq_km = distq/1000; + +% find indices of waypoints only +targets.dist_km = zeros(height(targets), 1); +targets.depth_m = nan(height(targets), 1); for f = 1:height(targets) - [mLat, idxLat] = min(abs(Zlat-targets.lat(f))); - [mLon, idxLon] = min(abs(Zlon-targets.lon(f))); - % make sure the mins are below the cell extent - if mLat <= refvec.CellExtentInLatitude && mLon <= refvec.CellExtentInLongitude - targets.depth(f) = Z(idxLat, idxLon); - end + wpIdx = find(latq == targets.lat(f) & lonq == targets.lon(f)); + targets.dist_km(f) = distq_km(wpIdx); + targets.depth_m(f) = zq(wpIdx); end % set up figure @@ -152,21 +125,74 @@ function plotTrackBathyProfile(CONFIG, targetsFile, bathyFile, yLine, figNum) clf cla reset; -plot(ti.cumDist_km, ti.depth, 'k:'); +plot(distq_km, zq, 'k:'); hold on; -scatter(targets.cumDist_km, targets.depth, 10, 'k', 'filled') +scatter(targets.dist_km, targets.depth_m, 10, 'k', 'filled') % label the waypoints -text(targets.cumDist_km + sum(targets.cumDist_km)*.006, targets.depth - 100, ... - targets.name, 'FontSize', 10); +text(targets.dist_km + max(targets.dist_km)*.006, targets.depth_m - 100, ... + targets.name, 'FontSize', 10); yline(yLine, '--', 'Color', '#900C3F'); grid on; hold off; xlabel('track length [km]') -ylim([-round((max(Z(:)) + max(Z(:))*.1)) 10]) +ylim([round(min(zq) + min(zq)*.1) 10]) +xlim([0 round(targets.dist_km(end) + targets.dist_km(end)*.05)]) ylabel('depth [m]') set(gca, 'FontSize', 12) title(sprintf('%s %s %s', CONFIG.glider, CONFIG.mission, ... 'Targets Bathymetry Profile'), 'Interpreter', 'none') + + + +% OLD MANUAL METHOD +% [Z, refvec] = geocrop(Z, refvec, CONFIG.map.latLim, CONFIG.map.lonLim); + +% % Pull out lat/lon vectors from refvec +% % use 0.5*cell extent to get midpoints of each cell +% Zlat = [refvec.LatitudeLimits(1)+0.5*refvec.CellExtentInLatitude: ... +% refvec.CellExtentInLatitude:refvec.LatitudeLimits(2)]'; +% Zlat = flipud(Zlat); % have to flip bc small latitudes are at poles +% Zlon = [refvec.LongitudeLimits(1)+0.5*refvec.CellExtentInLongitude: ... +% refvec.CellExtentInLongitude:refvec.LongitudeLimits(2)]'; +% +% % estimate cumulative track length +% targets.cumDist_km = zeros(height(targets), 1); +% for f = 2:height(targets) +% targets.cumDist_km(f) = targets.cumDist_km(f-1) + ... +% lldistkm([targets.lat(f-1) targets.lon(f-1)], [targets.lat(f) targets.lon(f)]); +% end +% +% % interpolate between targets at 0.1 dec deg resolution +% ti = table; +% ti.lat = interp1(targets.lat, 1:0.1:length(targets.lat))'; +% ti.lon = interp1(targets.lon, (1:0.1:length(targets.lon))'); +% ti.cumDist_km = interp1(targets.cumDist_km, 1:0.1:length(targets.lat))'; +% ti.depth = nan(height(ti), 1); +% +% +% % loop through interpolated lat/lons and pull depth at closest Z cell +% for f = 1:height(ti) +% [mLat, idxLat] = min(abs(Zlat-ti.lat(f))); +% [mLon, idxLon] = min(abs(Zlon-ti.lon(f))); +% % make sure the mins are below the cell extent +% if mLat <= refvec.CellExtentInLatitude && mLon <= refvec.CellExtentInLongitude +% ti.depth(f) = Z(idxLat, idxLon); +% end +% end +% +% % repeat for just the waypoints +% targets.depth = nan(height(targets), 1); +% for f = 1:height(targets) +% [mLat, idxLat] = min(abs(Zlat-targets.lat(f))); +% [mLon, idxLon] = min(abs(Zlon-targets.lon(f))); +% % make sure the mins are below the cell extent +% if mLat <= refvec.CellExtentInLatitude && mLon <= refvec.CellExtentInLongitude +% targets.depth(f) = Z(idxLat, idxLon); +% end +% end + + + end \ No newline at end of file diff --git a/agate/utils/printTravelMetrics.m b/agate/utils/printTravelMetrics.m index 4843111..841b6a2 100644 --- a/agate/utils/printTravelMetrics.m +++ b/agate/utils/printTravelMetrics.m @@ -8,7 +8,7 @@ % Summarize and print out several metrics about mission distances % covered (over ground and along trackline), average speeds (over % ground and along trackline), and estimates of remaining days to -% reach the end of the survey trackline. +% reach the end of the mission trackline. % % Inputs: % CONFIG Global config variable from agate mission configuration @@ -42,7 +42,7 @@ % S. Fregosi % % FirstVersion: 25 April 2023 -% Updated: 27 May 2023 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/agate/utils/readTargetsFile.m b/agate/utils/readTargetsFile.m index 6d196a4..5cbecf4 100644 --- a/agate/utils/readTargetsFile.m +++ b/agate/utils/readTargetsFile.m @@ -5,9 +5,9 @@ % TARGETS = TREADTARGETSFILE(CONFIG, TARGETSFILE) % % Description: -% Read in a Seaglider formatted targets file to a table variable. +% Read in a Seaglider formatted targets file to a table variable. % Fullfile name can be specified as input argument, or can be left -% blank to prompt to select targets file to read. +% blank to prompt to select targets file to read. % % Inputs: % CONFIG global CONFIG variable from agate mission @@ -34,18 +34,34 @@ % Created with MATLAB ver.: 9.9.0.1524771 (R2020b) Update 2 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -if nargin < 2 - [fileName, filePath] = uigetfile([CONFIG.path.mission, '\*.*'], ... - 'Select targets file to read'); - targetsFile = fullfile(filePath, fileName); - fprintf('targets file selected: %s\n', fileName); +switch nargin + case 0 + [fileName, filePath] = uigetfile('*.*', ... + 'Select targets file to read'); + targetsFile = fullfile(filePath, fileName); + fprintf('targets file selected: %s\n', fileName); + case 1 + if isstruct(CONFIG) + if isfield(CONFIG.path, 'mission') + [fileName, filePath] = uigetfile([CONFIG.path.mission, '\*.*'], ... + 'Select targets file to read'); + else + [fileName, filePath] = uigetfile('*.*', ... + 'Select targets file to read'); + end + targetsFile = fullfile(filePath, fileName); + fprintf('targets file selected: %s\n', fileName); + elseif isstring(CONFIG) + targetsFile = CONFIG; + end end + % check that is fullfile name [path, ~, ~] = fileparts(targetsFile); if isempty(path) - fprintf(1, ['No filepath specified, re-enter targetsFile argument with '... - 'path included. Exiting\n']); + fprintf(1, ['No filepath specified, re-enter targetsFile argument with '... + 'path included. Exiting\n']); end % read in file @@ -61,18 +77,18 @@ targets = table; for t = 1:numTargets - idxPeriod = regexp(x(idxBreak(t):end), '\.'); - idxLat = regexp(x(idxBreak(t):end), 'lat=', 'once'); - idxLon = regexp(x(idxBreak(t):end), 'lon=', 'once'); - idxRad = regexp(x(idxBreak(t):end), 'radius=', 'once'); - - if ~isempty(idxLat) - targets.name{t} = deblank(x(idxBreak(t):idxBreak(t) + idxLat - 2)); - targets.lat(t) = str2num(x(idxBreak(t) + idxLat + 3:idxPeriod(1) + idxBreak(t) - 4)) ... - + str2num(deblank(x(idxPeriod(1) + idxBreak(t) - 3:idxBreak(t) + idxLon - 2)))/60; - targets.lon(t) = str2num(x(idxLon + idxBreak(t) + 3:idxPeriod(2) + idxBreak(t) - 4)) ... - - str2num(deblank(x(idxPeriod(2) + idxBreak(t) - 3:idxBreak(t) + idxRad - 2)))/60; - end + idxPeriod = regexp(x(idxBreak(t):end), '\.'); + idxLat = regexp(x(idxBreak(t):end), 'lat=', 'once'); + idxLon = regexp(x(idxBreak(t):end), 'lon=', 'once'); + idxRad = regexp(x(idxBreak(t):end), 'radius=', 'once'); + + if ~isempty(idxLat) + targets.name{t} = deblank(x(idxBreak(t):idxBreak(t) + idxLat - 2)); + targets.lat(t) = str2num(x(idxBreak(t) + idxLat + 3:idxPeriod(1) + idxBreak(t) - 4)) ... + + str2num(deblank(x(idxPeriod(1) + idxBreak(t) - 3:idxBreak(t) + idxLon - 2)))/60; + targets.lon(t) = str2num(x(idxLon + idxBreak(t) + 3:idxPeriod(2) + idxBreak(t) - 4)) ... + - str2num(deblank(x(idxPeriod(2) + idxBreak(t) - 3:idxBreak(t) + idxRad - 2)))/60; + end end diff --git a/agate/utils/setCONFIG.m b/agate/utils/setCONFIG.m index a136bfc..302f4d3 100644 --- a/agate/utils/setCONFIG.m +++ b/agate/utils/setCONFIG.m @@ -1,8 +1,8 @@ -function setCONFIG(missionCnf) -%SETCONFIG Set up global CONFIG structure for agate +function CONFIG = setCONFIG(CONFIG) +%SETCONFIG Set up CONFIG structure for agate % % Syntax: -% CONFIG = SETCONFIG +% CONFIG = SETCONFIG(CONFIG) % % Description: % Called from the agate initialization. Sets the default @@ -10,11 +10,11 @@ function setCONFIG(missionCnf) % present in the settings folder % % Inputs: -% input none -% +% CONFIG [struct] containing all the user-set configurations such as +% paths, basestation login info, etc % Outputs: -% CONFIG Global structure containing all the user-set configurations -% such as paths, basestation login info, etc +% CONFIG [struct] containing all the user-set configurations such as +% paths, basestation login info, etc % % Examples: % @@ -27,25 +27,11 @@ function setCONFIG(missionCnf) % https://github.com/MarineBioAcousticsRC/Triton/ % % FirstVersion: 06 April 2023 -% Updated: 11 April 2024 +% Updated: 07 August 2024 % % Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -global CONFIG - -% set defaults in case no config file -% % paths -% CONFIG.path.shp = 'C:\Users\User.Name\Documents\GIS\'; -% CONFIG.path.survey = 'C:\Desktop\glider_mission\'; -% % basestation configuration -% CONFIG.bs.cnfFile = 'basestation.cnf'; -% CONFIG.bs.host = 'url.com'; -% CONFIG.bs.username = 'pilot'; -% CONFIG.bs.password = 'PsWrD'; - - % update based on user-defined configuration file if it exists % if none specified, prompt to locate one @@ -57,22 +43,19 @@ function setCONFIG(missionCnf) else % check if full file or just parts [path, ~, ~] = fileparts(CONFIG.missionCnf); - if ~isempty(path) - CONFIG.path.cnfFid = fopen(CONFIG.missionCnf,'r'); - parseCnf(CONFIG.missionCnf); - else % no path specified + if isempty(path) % no path specified % default location is within agate\settings folder, so try that CONFIG.missionCnf = fullfile(CONFIG.path.settings, CONFIG.missionCnf); % otherwise prompt to select one - if ~exist(missionCnf, 'file') + if ~exist(CONFIG.missionCnf, 'file') [name, path] = uigetfile([CONFIG.path.agate, '\*.cnf'], ... - 'Select survey configuration file'); + 'Select mission configuration file'); CONFIG.missionCnf = fullfile(path, name); end end end -CONFIG.path.cnfFid = fopen(CONFIG.missionCnf,'r'); -parseCnf(CONFIG.missionCnf); +% CONFIG.path.cnfFid = fopen(CONFIG.missionCnf, 'r'); +CONFIG = parseCnf(CONFIG.missionCnf, CONFIG); CONFIG.gmStr = [CONFIG.glider '_' CONFIG.mission]; % if basestation 'bs' configurations exist @@ -84,8 +67,7 @@ function setCONFIG(missionCnf) 'Select basestation configuration file'); CONFIG.bs.cnfFile = fullfile(path, name); end - % CONFIG.bs.cnfFid = fopen(CONFIG.bs.cnfFile,'r'); - parseCnf(CONFIG.bs.cnfFile); + CONFIG = parseCnf(CONFIG.bs.cnfFile, CONFIG); end % if pm configurations exist @@ -96,7 +78,7 @@ function setCONFIG(missionCnf) 'Select PMAR convert configuration file'); CONFIG.pm.cnfFile = fullfile(path, name); end - parseCnf(CONFIG.pm.cnfFile); + CONFIG = parseCnf(CONFIG.pm.cnfFile, CONFIG); end end @@ -104,9 +86,7 @@ function setCONFIG(missionCnf) function CONFIG = parseCnf(userCnf, CONFIG) % parse info from .cnf text files -global CONFIG - -fid = fopen(userCnf,'r'); +fid = fopen(userCnf, 'r'); if fid == -1 fprintf(1, 'No file selected. Exiting.\n') return diff --git a/agate/scratch/tritonLogToEventLog.m b/agate/utils/tritonLogToEventLog.m similarity index 100% rename from agate/scratch/tritonLogToEventLog.m rename to agate/utils/tritonLogToEventLog.m diff --git a/agate/workflows/README-workflow.md b/agate/workflows/README-workflow.md deleted file mode 100644 index e589121..0000000 --- a/agate/workflows/README-workflow.md +++ /dev/null @@ -1 +0,0 @@ -This folder is ignored by GitHub. Any files stored here will ONLY be kept locally and will not be uploaded to GitHub.com, but they will not be under version control. \ No newline at end of file diff --git a/docs/_quarto.yml b/docs/_quarto.yml index a7426f5..543c2f7 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -38,8 +38,8 @@ website: text: Configuration guide - href: plotting-functions.qmd text: Plots - - href: survey-planning.qmd - text: Survey planning + - href: mission-planning.qmd + text: Mission planning - href: piloting-functions.qmd text: Piloting - href: convert-acoustics.qmd diff --git a/docs/configuration.qmd b/docs/configuration.qmd index 42cc4a0..4d7e772 100644 --- a/docs/configuration.qmd +++ b/docs/configuration.qmd @@ -7,29 +7,150 @@ subtitle: 'Guide for creating and modifying necessary configuration files' library(fontawesome) ``` -Running `agate` requires a few configuration files. Each of these are plain text files that end with .cnf and can be edited in any text editor or in MATLAB's Editor window. The **mission configuration file** is required for all processes, but the others are optional depending on the goals of a specific task. +Running __*agate*__ requires a few configuration files. Each of these are plain text files that end with .cnf and can be edited in any text editor or in MATLAB's Editor window. The **mission configuration file** is required for all processes, but the others are optional depending on the goals of a specific task. 1. An overview [mission configuration file](#mission-configuration-file) for a specific glider/mission -2. A [basestation configuration file](#basestation-configuration-file) with SSH login info - only required if using `agate` to interact directly with the basestation +2. A [basestation configuration file](#basestation-configuration-file) with SSH login info - only required if using __*agate*__ to interact directly with the basestation 3. A conversion configuration (for either [PMAR](#pmar-conversion-configuration-file) or [WISPR](#wispr-conversion-configuration-file)) used when converting raw (.dat) acoustic files to .wav or .flac ## Mission configuration file An example configuration file is located in the `agate/settings` folder: [`agate_config_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf) -Lines starting with `%` are comments. The configuration file has settings for the glider and mission, paths to relevant input and output folders, map extent and plotting settings, and acoustic system settings. The top section is required to initialize `agate` and use the most basic functions. The following sections are optional depending on what agate functionality is desired, including interfacing with the basestation, working with acoustic data outputs, and plotting maps. Save this file with a unique name for each glider and mission. Descriptions of each configuration item are included in the example file as comments. +The configuration file has settings for the glider and mission, paths to relevant input and output folders, map extent and plotting settings, and acoustic system settings. Lines starting with `%` are comments. + +The top section is [required](#required-configuration-settings) to initialize __*agate*__ and use the most basic functions. The remaining sections are optional depending on what __*agate*__ functionality is desired, including interfacing with the [basestation](), working with [acoustic]() data outputs, and [plotting maps](). Save this file with a unique name for each glider and mission. Descriptions of each configuration item are included in the example file as comments. To suggest additional configuration items, please open an [issue](https://github.com/sfregosi/agate-public/issues/new){target='_blank'}. [Back to top](#) +#### Required configuration settings + +The top section `% %%%% REQUIRED CONFIG PARAMETERS %%%%%%%%%%%` is required to initialize __*agate*__ and use the most basic functions. Explanations for each parameter are in the example configuration file as comments. + +```matlab +CONFIG.glider = 'sgXXX'; % glider serial number +CONFIG.mission = 'Location_Mon20XX'; % mission identifier defined in sg_calib_constants.m +CONFIG.sgVer = XX.XX; % 66.12 or 67.0, firmware version displayed on glider start up +CONFIG.tmd = XX; % target mission duration in days + +% define paths +% mission path - path to umbrella folder that either contains previously +% downloaded basestation and output files or will contain +% downloaded and processed/produced files and figures +CONFIG.path.mission = 'C:\Users\User.Name\Desktop\sgXXX_Location_Mon20XX\'; +% analysis path - optional path to an 'analysis' folder that would typically +% contain the outputs of acoustic analyses (e.g., Pamguard +% or Triton) +CONFIG.path.analysis = 'C:\Users\User.Name\project\analysis'; +``` + +#### Optional basestation settings + +```matlab +% %%%% OPTIONAL - basestation %%%%%%%%%%%%%%% +% specify basestation configuration file +CONFIG.bs.cnfFile = 'C:\Users\User.Name\Documents\gliders\basestation.cnf'; +% specify path to store downloaded basestation files +CONFIG.path.bsLocal = fullfile(CONFIG.path.mission, 'basestationFiles\'); +``` + +#### Optional acoustics settings + +Various settings required to operate an acoustic system or to process acoustic data after a mission is over. Depending on which acoustic system is installed, only PMAR or WISPR settings are needed. Unneeded lines can be deleted or ignored by adding a `%` at the start of the line. + +```matlab +% %%%% OPTIONAL - acoustics %%%%%%%%%%%%%%%%% +``` + +For PMAR, some settings are required during a mission to track remaining memory available. The conversion settings are only required after a mission is complete to convert .dat to .wav and should be turned off after conversion is done. The processing and analysis settings are only needed after a mission is complete. + +```matlab +% PMAR settings +CONFIG.pm.loggers = 0; % 1 for active, 0 for inactive +CONFIG.pm.activeCard = 0; % current active card; optional for sgVer 66.12 + % (will be updated in log file automatically); + % necessary to update throughout mission for sgVer 67.0 +CONFIG.pm.numCards = 4; % total number loaded SD cards +% after mission is complete, set below for raw .dat file conversion +CONFIG.pm.convert = 0; % 0 during mission or analysis, set 1 to run conversion +CONFIG.pm.convCnfFile = fullfile(CONFIG.path.mission, 'PMARConvert_sgXXX_Location_MonYYYY.cnf'); +% after mission is complete, set below for data processing/analysis +CONFIG.pm.fileLength = 600; % in seconds +CONFIG.pm.sampleRate = 180260; % sample rate +CONFIG.pm.dateFormat = 'yyMMdd-HHmmss.SSS'; +CONFIG.pm.dateStart = 19; % what character of file name starts the date format +% example file name: sg639_MHI_Apr2023_230411-185050.484.wav +``` + +WISPR settings are only required for processing and analysis after a mission is complete; there is no piloting interaction with the WISPR system at this time. + +```matlab +% WISPR settings +% only needed after mission is complete (no piloting functionality for WISPR) +CONFIG.ws.loggers = 0; % 1 for active, 0 for inactive +CONFIG.ws.fileLength = 60; % in seconds +CONFIG.ws.sampleRate = 180000; % sample rate +CONFIG.ws.dateFormat = 'yyMMdd_HHmmss'; +CONFIG.ws.dateStart = 7; % what character of file name starts the date format +% example file name: WISPR_230504_202206.wav +``` + +#### Optional plotting settings + +If using __*agate*__ for plotting during a mission or for analysis and processing after, it is useful to set some plot settings and paths so they don't have to be repeatedly manually modified. + +```matlab +% %%%% OPTIONAL - plotting %%%%%%%%%%%%%%%%%% +``` + +For mapping: + +```matlab +% maps +% path to basemap files (land shape files and bathymetry rasters) +CONFIG.path.shp = 'C:\Users\User.Name\Documents\GIS\'; +% optional: can specify full path to bathymetry file, otherwise agate +% will prompt to select file +CONFIG.map.bathyFile = 'C:\Users\User.Name\Documents\GIS\etopo\etopo1_ice_g_i2.bin'; +% define map extents and location of map elements +CONFIG.map.latLim = [000.00 000.00]; +CONFIG.map.lonLim = [-000.00 -000.00]; +CONFIG.map.naLat = 00.00; % optional arguments for north arrow location on map; + % leave empty '[]' for no north arrow +CONFIG.map.naLon = 000.00; +CONFIG.map.scalePos = [0 0]; % optional arguments for scale bar location + % and ticks, leave empty '[]' for no scale bar +CONFIG.map.scaleMajor = [0:50:100]; +CONFIG.map.scaleMinor = [0:12.5:25]; +``` + +For piloting related plots: + +```matlab +% piloting plot settings +% pre allocate fig numbers so figures are updated rather than created new +CONFIG.plots.figSeed = 111; % suggest using the glider serial e.g., 639 +CONFIG.plots.figNumList = CONFIG.plots.figSeed + [0 1 2 3 4 5 6 7 8 9]; +% figNumList (1) = battery, (2) = voltagePackUse, (3) = humidity +% and pressure, (4) = map, (5) = zoomed map, (6) = minimum voltage, +% (7) = voltage normalized, (8) = ERMA detections, (9) = ERMA reference + +% load plot positions +% load('C:\Users\User.Name\Documents\gliders\figPositions.mat'); +% CONFIG.plots.positions = positions; +``` + +[Back to top](#) + ## Basestation configuration file -The path and filename for basestation configuration file is specified in the mission configuration file. An example is located in the 'agate/settings' folder: [`basestation.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/basestation.cnf). +The path and filename for basestation configuration file is specified in the mission configuration file. An example is located in the `agate/settings` folder: [`basestation_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/basestation_example.cnf). This is a separate configuration file that typically does not change between missions and gliders, and contains potentially sensitive information for the SSH connection to a research group's basestation. This file should be stored somewhere central and safe, preferably outside of the GitHub repository for security reasons. This file must contain the following lines, with the inputs updated for a particular basestation: -```default +```matlab CONFIG.bs.host = 'url.com'; CONFIG.bs.username = 'pilot'; CONFIG.bs.password = 'PsWrD'; @@ -39,14 +160,14 @@ CONFIG.bs.password = 'PsWrD'; ## Acoustic conversion configuration files -A conversion configuration file is necessary for converting raw acoustic data to .wav or .flac formats. There are different configuration files for the PMAR and for the WISPR systems. They provide information on conversion settings and file paths. The conversion configuration file is specified within the mission configuration file, and so is parsed by the initial call to `agate` and the mission configuration. +A conversion configuration file is necessary for converting raw acoustic data to .wav or .flac formats. There are different configuration files for the PMAR and for the WISPR systems. They provide information on conversion settings and file paths. The conversion configuration file is specified within the mission configuration file, and so is parsed by the initial `agate` call and the mission configuration. ### PMAR conversion configuration file -An example configuration file is located in the 'agate/settings' folder: [`pmarConvert_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/pmarConvert_example.cnf) +An example configuration file is located in the `agate/settings` folder: [`pmarConvert_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/pmarConvert_example.cnf) -Lines starting with `%` are comments. All parameters are added to the existing `CONFIG` structure, under a nested `pm` structure. All the changeable parameters are listed/grouped at the top,but there is additional detail about each parameter as comments below. These detailed descriptions include some additional example inputs and the default settings. +Lines starting with `%` are comments. All parameters are added to the existing `CONFIG` structure, under a nested `pm` structure. All the changeable parameters are listed/grouped at the top, but there is additional detail about each parameter as comments below. These detailed descriptions include some additional example inputs and the default settings. - `CONFIG.pm.inDir`: path to raw PMAR `.dat` files. These can be within subdirectories by dive (the default write method of PMAR) @@ -63,7 +184,9 @@ Lines starting with `%` are comments. All parameters are added to the existing ` ### WISPR conversion configuration file -An example configuration file is located in the 'agate/settings' folder: [`wisprConvert_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/wisprConvert_example.cnf) +**UNDER CONSTRUCTION** + +An example configuration file is located in the `agate/settings` folder: [`wisprConvert_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/wisprConvert_example.cnf) Lines starting with `%` are comments. All parameters are added to the existing `CONFIG` structure, under a nested `ws` structure. All the changeable parameters are listed/grouped at the top,but there is additional detail about each parameter as comments below. These detailed descriptions include some additional example inputs and the default settings. diff --git a/docs/contribute.qmd b/docs/contribute.qmd new file mode 100644 index 0000000..7515c82 --- /dev/null +++ b/docs/contribute.qmd @@ -0,0 +1,89 @@ +--- +title: "How to contribute" +page-layout: full +--- + +```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) +library(fontawesome) +``` + +We welcome contributions from the passive acoustic glider community! If you would like to contribute to this repository, you can do so in a few ways: + +### **If you find a bug...** + +Please [report an issue on GitHub](https://github.com/sfregosi/agate-public/issues/new){target='_blank'}. Please use the 'Bug report' template. + +### **If you'd like to request a feature or suggest an enhancement...** + +Please [report an issue on GitHub](https://github.com/sfregosi/agate-public/issues/new){target='_blank'}. There is a 'Feature request' template just for this purpose. + +### **If you'd like to add a feature or fix a bug yourself...** + +[Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) the `agate-public` repository. This will create a copy of the repository in your own GitHub account. You can clone this fork to your local machine to work with the toolbox and make changes directly to the code, but also continue to pull changes from the primary repository to stay up to date. Then, when you have a feature you'd like to contribute back to the main repository, you can use a [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) to incorporate those changes. + +We are happy to help get folks set up with this process, so please reach out with any questions! + +For consistency in documentation, we ask that you use the following templates to create any new functions or scripts: + +#### New function + +Paste the below code into the top of the new function and updated as needed. Refer to existing functions in the `utils` folder for examples of what kind of detail to include. For consistency, use 'camelCase' for the function name. Include your name, contact info, and MATLAB version info which can be checked with `version`. + +```matlab +function output = newFunction(input) +% NEWFUNCTION One-line description here, please +% +% Syntax: +% OUTPUT = NEWFUNCTION(INPUT) +% +% Description: +% Detailed description here, please +% Inputs: +% input describe, please +% +% Outputs: +% output describe, please +% +% Examples: +% +% See also +% +% Authors: +% F. Last +% +% FirstVersion: 05 August 2024 +% Updated: +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +``` + +#### New script + +Paste the below code into the top of the new script and fill in the detail as needed. Refer to existing scripts in the `example_workflows` folder. Please include your name, contact info, and MATLAB version info which can be checked with `version`. + +```matlab +% NEWSCRIPT +% One-line description here, please +% +% Description: +% Detailed description here, please +% +% Notes +% +% See also +% +% +% Authors: +% F. Last +% +% FirstVersion: 05 August 2024 +% Updated: +% +% Created with MATLAB ver.: 9.13.0.2166757 (R2022b) Update 4 +% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +``` diff --git a/docs/convert-acoustics.qmd b/docs/convert-acoustics.qmd index f902132..e618614 100644 --- a/docs/convert-acoustics.qmd +++ b/docs/convert-acoustics.qmd @@ -4,10 +4,14 @@ subtitle: 'Guide for converting raw acoustic .dat files to .wav' --- ```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) library(fontawesome) ``` -The Seaglider can be equipped with a variety of acoustic systems. Two - the PMARXL and WISPR systems - both record raw data as .dat files that need to be converted to .wav or .flac for analysis. `agate` includes tools to do this for both of these two systems. All conversion code is located in the `convertAcoustics` folder. +The Seaglider can be equipped with a variety of acoustic systems. Two - the PMARXL and WISPR systems - both record raw data as .dat files that need to be converted to `.wav` or `.flac` for analysis. __*agate*__ includes tools to do this for both of these two systems. All conversion code is located in the `convertAcoustics` folder. Additionally, following a mission in Spring 2023, a bug in the WISPR firmware was discovered that led to variable gain over the duration of the deployment. We developed a systematic way to 'fix' this gain in an attempt to standardize it across all files. The code needed to apply this fix is in the `convertAcoustics/fixWispr` folder. @@ -16,11 +20,11 @@ Additionally, following a mission in Spring 2023, a bug in the WISPR firmware wa ## Convert PMAR raw files -See [`workflow_convertPMAR.m`](https://github.com/sfregosi/agate-public/blob/main/agate/example_workflows/workflow_convertPMAR.m) in the `example_workflows` folder for a simple example workflow for converting a mission's PMAR .dat files to .wav. +See [`workflow_convertPMAR.m`](https://github.com/sfregosi/agate-public/blob/main/agate/example_workflows/workflow_convertPMAR.m) in the `example_workflows` folder for a simple example workflow for converting a mission's PMAR `.dat` files to `.wav`. -Like other workflows in `agate` it must first be initialized with a mission-level configuration file (e.g., [agate_config_example.cnf]( https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf)). Only the top **REQUIRED** and `CONFIG.pm` settings in the **OPTIONAL - acoustics** sections of the configuration file are needed for file conversion. +Like other workflows in __*agate*__ it must first be initialized with a mission-level configuration file (*e.g.*, [`agate_config_example.cnf`]( https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf)). Only the top `REQUIRED` and `CONFIG.pm` settings in the `OPTIONAL - acoustics` sections of the configuration file are needed for file conversion. -Additionally, a PMAR conversion configuration file (e.g., [pmarConvert_example.cnf](https://github.com/sfregosi/agate-public/blob/main/agate/settings/pmarConvert_example.cnf)) is needed. This contains the specific paths and is where you can set your desired conversion settings. Detailed descriptions of each parameter setting are in the example configuration file or can be found on the [Configuration](./configuration.html#pmar-conversion-configuration-file) page. +Additionally, a PMAR conversion configuration file (*e.g.*, [`pmarConvert_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/pmarConvert_example.cnf)) is needed. This contains the specific paths and is where you can set your desired conversion settings. Detailed descriptions of each parameter setting are in the example configuration file or can be found on the [Configuration - PMAR conversion](./configuration.html#pmar-conversion-configuration-file) page. @@ -34,13 +38,13 @@ Additionally, a PMAR conversion configuration file (e.g., [pmarConvert_example.c ## Fix WISPR gain bug -A bug was discovered in WISPR in spring 2023 (should be fixed now??) that leads to sound recordings that look like they have variable gain settings applied on different files. One file would be half as loud as the next (6 dB gain change). For more info on the source/issue with this bug, contact Chris Jones. An imperfect, but workable fix was developed with C. Jones and is included here just in case someone else would find it useful or if the issue arises again in the future. Because this is hopefully one-off issue, the below instructions are an outline/draft of documentation and the included code is not generalized (*e.g.*, hard coded paths, etc) and is meant to guide a similar effort. Please reach out if you encounter the same problem or need further assistance. +A bug was discovered in WISPR in spring 2023 (should be fixed now??) that leads to sound recordings that look like they have variable gain settings applied on different files. One file would be half as loud as the next (6 dB gain change). For more info on the source/issue with this bug, contact Chris Jones. An imperfect, but workable fix was developed with C. Jones and is included here just in case someone else would find it useful or if the issue arises again in the future. Because this is hopefully one-off issue, the below instructions are an outline/draft of documentation and the included code is not generalized (*e.g.*, hard coded paths, etc.) and is meant to guide a similar effort. Please reach out if you encounter the same problem or need further assistance. -This process assumes you have initially converted the .dat files to .wavs, and in looking at those .wavs, noticed an issue. Those 'bad' .wavs will be used to make decisions about file gain adjustments so those need to be retained. Then, this process will *reconvert* the .dat files into .wavs, making adjustments to the expected gain if needed, as assessed following a set of rules or according to manual user input. +This process assumes you have initially converted the `.dat` files to `.wavs`, and in looking at those .wavs, noticed an issue. Those 'bad' `.wavs` will be used to make decisions about file gain adjustments so those need to be retained. Then, this process will *reconvert* the `.dat` files into `.wavs`, making adjustments to the expected gain if needed, as assessed following a set of rules or according to manual user input. -The general workflow was to work on one dive's worth of files at a time. That dive's 'bad' .wav files would be opened in Raven so the spectrograms could be visually compared. Jumps in gain are vary obvious as stark changes in noise level from one file to the next. Quieter files are gain settings of '1' and are not changed. The louder files are assumed to have a gain setting of '2' and are reduced to align more with the quieter files. +The general workflow was to work on one dive's worth of files at a time. That dive's 'bad' `.wav` files would be opened in Raven so the spectrograms could be visually compared. Jumps in gain are vary obvious as stark changes in noise level from one file to the next. Quieter files are gain settings of '1' and are not changed. The louder files are assumed to have a gain setting of '2' and are reduced to align more with the quieter files. -A MATLAB script is run for that dive's worth of files, where .dat files are read in and the noise levels in a specified band are compared. If they are within a similar level, then the gain is assumed to *not* change and the files are written to .wav as is. If the second file is louder, then it's gain is adjusted down and this adjusted file is written to .wav format. There are special catches in the script for instances where the noise level between the two files is not a clear 1:1 or 1:2 ratio. In those cases, the script prompts the user to manually assess and specify an adjustment. The user can use the script-generated plots and the spectrograms of the 'bad' files to make this decision. +A MATLAB script is run for that dive's worth of files, where `.dat` files are read in and the noise levels in a specified band are compared. If they are within a similar level, then the gain is assumed to *not* change and the files are written to `.wav` as is. If the second file is louder, then it's gain is adjusted down and this adjusted file is written to `.wav` format. There are special catches in the script for instances where the noise level between the two files is not a clear 1:1 or 1:2 ratio. In those cases, the script prompts the user to manually assess and specify an adjustment. The user can use the script-generated plots and the spectrograms of the 'bad' files to make this decision. All code needed to apply the fix is located in the `agate/processAcoustics/fixWispr` folder. @@ -52,7 +56,7 @@ All code needed to apply the fix is located in the `agate/processAcoustics/fixWi `plotRMSSpec.m`: Function to plot RMS level for two files being compared -`read_wispr_file.m`: Function to read a raw WISPR .dat file, extracting all included header info +`read_wispr_file.m`: Function to read a raw WISPR `.dat` file, extracting all included header info ### Running `fix_gain` @@ -60,15 +64,15 @@ All code needed to apply the fix is located in the `agate/processAcoustics/fixWi The `fix_gain` script is meant to be run over a folder of files. Depending on deployment length, it may be beneficial to work in smaller batches of files. For example, for a 5 week glider mission, we had 40k files which was unreasonable to work with all at once. Plus, when the glider was at the surface there were short gaps in time and so comparing noise levels of files across that time gap may not be appropriate. We found that working on a folder of just the files for a single dive at a time worked well. You could use a different approach but it would require reworking the paths in this script. -Create a 'working' folder where you will incrementally move the .dat files for just a single dive, process those, then move them out of that folder and replace with the files for the next dive. +Create a 'working' folder where you will incrementally move the `.dat` files for just a single dive, process those, then move them out of that folder and replace with the files for the next dive. -Load the dive's worth of 'bad' .wavs into Raven (use the Page option to load large amounts of data without crashing Raven) to view the spectrograms in cases where manual assessment is needed. +Load the dive's worth of 'bad' `.wavs` into Raven (use the Page option to load large amounts of data without crashing Raven) to view the spectrograms in cases where manual assessment is needed. #### Set paths and other settings Within the script, several lines of code will need to be changed for your local paths and settings preferences. -- **Line 48**: Specify path to `agate` so all required functions are added to the path +- **Line 48**: Specify path to __*agate*__ folder so all required functions are added to the path - **Line 54**: Set `verbose` to either `'true'` or `'false'` - `verbose = 'true'`: All prompts and all plots will be generated. The user will need to manually click through every file comparison. This is very useful when first fixing this problem to get a feel for how the files compare, but is time consuming. It is also useful in time periods with a lot of animal noise that could lead to incorrect assumptions of gain changes. - `verbose = 'false'`: Plots and prompts only appear if a non-standard gain adjustment is detected (not 1 or 2). This enables the script to run mostly hands off except in cases that aren't clear @@ -81,7 +85,7 @@ Within the script, several lines of code will need to be changed for your local - **Line 71**: Set `adc_vref`. Default for WISPR is 5 - **Line 72**: Set `nbufs` to specify number of buffers to compare (from end of first file to start of second file). For our recording settings, a buffer was 5632 samples, sample rate was 180 kHz, so 96 buffers is just over 3 seconds - **Line 73**: Set `max_thresh` to remove spikes in waveform that could skew the RMS calculations. This is adaptive, so if it is set at 1 and that removes too many datapoints, the process will try to incrementally increase it. This is necessary for periods of glider pumping which are all very loud -Lines 78:79, 81: Set spectrum parameters to provide resonable resolution for a given sampling rate. Larger fft_size will give finer resolution but may lead to 'spiky' spectra that aren't good for comparison +Lines 78:79, 81: Set spectrum parameters to provide reasonable resolution for a given sampling rate. Larger `fft_size` will give finer resolution but may lead to 'spiky' spectra that aren't good for comparison - **Lines 84:85**: Set frequency range (`f1` lower limit, `f2` upper limit) to compare. Ideal is to select a frequency range a lot of animal or anthropogenic (intermittent/variable) sounds would not be present The remaining lines set up the log file, so don't need to be changed unless the user wants additional information printed out @@ -98,7 +102,7 @@ The only valid inputs should be '1' or '2'. Values that are not one of these wil In periods with a lot of dolphins, the script may get confused (noise levels in subsequent 3 seconds can be very different!) so those may require lots of manual assessment. -If a wrong key is accidentally pressed or a mistake is made, the best approach is to Quit at the next available prompt. Navigate to the log files and rename them with 'part1' or something similar. Move the .dat files that were correctly processed out of the working folder, and then re-run the reduced working folder. This will create new log files at this new starting location, and will overwrite any incorrectly adjusted .wavs. A record of what was adjusted for the earlier files will be saved in the original logs with the appended names. +If a wrong key is accidentally pressed or a mistake is made, the best approach is to Quit at the next available prompt. Navigate to the log files and rename them with 'part1' or something similar. Move the `.dat` files that were correctly processed out of the working folder, and then re-run the reduced working folder. This will create new log files at this new starting location, and will overwrite any incorrectly adjusted `.wavs`. A record of what was adjusted for the earlier files will be saved in the original logs with the appended names. diff --git a/docs/get-started.qmd b/docs/get-started.qmd index a720fa0..9a8b508 100644 --- a/docs/get-started.qmd +++ b/docs/get-started.qmd @@ -1,5 +1,5 @@ --- -title: "Get started with agate" +title: "Get started with *agate*" subtitle: 'Installation and quick start guide' --- @@ -7,29 +7,28 @@ subtitle: 'Installation and quick start guide' library(fontawesome) ``` -This page is meant to help you get started with `agate` by getting it properly 'installed' on your MATLAB path, setting up the necessary configuration files and folder structure, and a quick start guide for some basic commands. +This page is meant to help you get started with __*agate*__ by getting it properly 'installed' on your MATLAB path, setting up the necessary configuration files and folder structure, and a quick start guide for some basic commands. -It page is not a detailed list of all available functions and their specific documentation (that might be coming later!). Those details are available within the standard MATLAB-type documentation in the header of each function and include a detailed description, info on input and output arguments, and examples. These details can be pulled up by typing `doc function` or `help function` within the MATLAB Command Window. +It page is not a detailed list of all available functions and their specific documentation (that might be coming later!). Those details are available within the standard MATLAB-type documentation in the header of each function and include a detailed description, info on input and output arguments, and examples. These details can be pulled up by typing `doc ` or `help ` in the MATLAB Command window, replacing `` with the actual name of the function. ## Installation +*See the [Dependencies](https://sfregosi.github.io/agate-public/#dependencies) section of the home page for more info on the required Mathworks File Exchange packages (copies of these come packaged with __agate__) and MATLAB Toolbox requirements.* -*See the [Dependencies](https://sfregosi.github.io/agate-public/#dependencies) section of the home page for more info on the required Mathworks File Exchange packages (copies of these come packaged with `agate`) and MATLAB Toolbox requirements.* - -- Download `agate` from [GitHub](https://github.com/sfregosi/agate-public) +- Download __*agate*__ from [GitHub](https://github.com/sfregosi/agate-public) - **Option 1:** Download the latest release - This option ensures a stable release and removes the requirement of working with GitHub but will need manual updating - Visit the [Releases](https://github.com/sfregosi/agate-public/releases) page and download the latest release source code as a zip or tar.gz file - **Option 2:** Clone the repository using GitHub or GitHub Desktop - This package is actively being developed and the easiest way stay up to date with the latest improvements is to regularly check for updates from GitHub, but this comes with risks as it may be buggy - Click on the green *Code* button and select *Open with GitHub Desktop* - - Specify where to clone the cloned local copy. Suggest the default MATLAB directory (e.g., `C:\Users\User.Name\Documents\MATLAB\`) + - Specify where to clone the cloned local copy. Suggest the default MATLAB directory (*e.g.*, `C:\Users\User.Name\Documents\MATLAB\`) - For more help with GitHub see this [Git Started Doc](https://github.com/PIFSC-Protected-Species-Division/PSDOS/blob/main/files/git_started.md) - **Option 3:** Download the repository as a zip file - This provides the latest functionality before an official release and removes requirement of working with GitHub but will need manual updating - Click the green *Code* button at the landing page of the repository and choose *Download ZIP* - - Unzip the downloaded folder and place within the default MATLAB directory (e.g., `C:\Users\User.Name\Documents\MATLAB\`) -- Add `agate` to the MATLAB path + - Unzip the downloaded folder and place within the default MATLAB directory (*e.g.*, `C:\Users\User.Name\Documents\MATLAB\`) +- Add __*agate*__ to the MATLAB path - Open MATLAB and click on *Set Path* in the Environment section of the Home tab (@fig-screenshot-matlab-setpath) - In the Set Path dialog box, choose *Add with Subfolders...*, select the `agate-public` folder, and click *Save*, then *Close* (@fig-screenshot-matlab-setpath-save) - This will now be saved for future MATLAB sessions, but would need to be repeated for any installation on a new computer @@ -45,42 +44,40 @@ It page is not a detailed list of all available functions and their specific doc ### Create configuration files -Running `agate` requires a few configuration files. Both of these are plain text files that end with .cnf and can be edited in any text editor or in MATLAB's Editor window +Running __*agate*__ requires a few configuration files. Both of these are plain text files that end with .cnf and can be edited in any text editor or in MATLAB's Editor window 1. An overview [mission configuration file](#mission-configuration-file) for a specific glider/mission 2. A [basestation configuration file](#basestation-configuration-file) with SSH login info #### Mission configuration file -An example configuration file is located in the 'agate/settings' folder: [`agate_config_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf) +An example configuration file is located in the `agate/settings` folder: [`agate_config_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/agate_config_example.cnf) + +The configuration file has settings for the glider and mission, paths to relevant input and output folders, map extent and plotting settings, and acoustic system settings. The top section is required to initialize __*agate*__ and use the most basic functions. The remaining sections are optional depending on what __*agate*__ functionality is desired. Make a copy and save this file with a unique name for each glider and mission. Descriptions of each configuration item are included in the example file as comments that start with `%`. -Lines starting with `%` are comments. The configuration file has settings for the glider and mission, paths to relevant input and output folders, map extent and plotting settings, and acoustic system settings. The top section is required to initialize `agate` and use the most basic functions. The following sections are optional depending on what agate functionality is desired, including interfacing with the basestation, working with acoustic data outputs, and plotting maps. Save this file with a unique name for each glider and mission. Descriptions of each configuration item are included in the example file as comments. +Additional info on setting up configuration files can be found in the [Configuration file guide](https://sfregosi.github.io/agate-public/configuration.html). To suggest additional configuration items, please open an [issue](https://github.com/sfregosi/agate-public/issues/new){target='_blank'}. #### Basestation configuration file -The path and filename for basestation configuration file is specified in the mission configuration file. An example is located in the 'agate/settings' folder: [`basestation.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/basestation.cnf). +The path and filename for a basestation configuration file is specified in the mission configuration file. An example is located in the `agate/settings` folder: [`basestation_example.cnf`](https://github.com/sfregosi/agate-public/blob/main/agate/settings/basestation_example.cnf). -This is a separate configuration file that typically does not change between missions and gliders, and contains potentially sensitive information for the SSH connection to a research group's basestation. This file should be stored somewhere central and safe, preferably outside of the GitHub repository for security reasons. This file must contain the following lines, with the inputs updated for a particular basestation: +This is a separate configuration file that typically does not change between missions and gliders, and contains potentially sensitive information for the SSH connection to a research group's basestation. *This file should be stored somewhere central and safe, preferably outside of the GitHub repository for security reasons.* -```default -CONFIG.bs.host = 'url.com'; -CONFIG.bs.username = 'pilot'; -CONFIG.bs.password = 'PsWrD'; -``` +Additional info on setting up configuration files can be found in the [Configuration file guide](https://sfregosi.github.io/agate-public/configuration.html). [Back to top](#) ### Set up folder structure -The suggested folder structure for working with `agate` is to specify a 'mission' folder, and then within that have a standardized set of nested folders for the various `agate` inputs and outputs. The path of the 'mission' folder is specified by `CONFIG.path.mission` in the [mission configuration file](#mission-configuration-file) and typically follows the Seaglider naming scheme (e.g., `C:\Users\User.Name\Desktop\sg###_Location_MonYYYY`). +The suggested folder structure for working with __*agate*__ is to specify a 'mission' folder, and then within that have a standardized set of nested folders for the various __*agate*__ inputs and outputs. The path of the 'mission' folder is specified by `CONFIG.path.mission` in the [mission configuration file](#mission-configuration-file) and typically follows the Seaglider naming scheme (*e.g.*, `C:\Users\User.Name\Desktop\sg###_Location_MonYYYY`). -Within that, should be a 'flightStatus' and a 'basestationFiles' folder; 'flightStatus' will house output figures and tables, and 'basestationFiles' is where downloaded basestation files will be saved. +Within that, should be a `flightStatus` and a `basestationFiles` folder; `flightStatus` will house output figures and tables, and `basestationFiles` is where downloaded basestation files will be saved. These folders can be set up manually, or created in MATLAB: -```default +```matlab % specify the local piloting folder for this trip path_status = fullfile(CONFIG.path.mission, 'flightStatus'); % where to store status outputs path_bsLocal = fullfile(CONFIG.path.mission, 'basestationFiles'); % local copy of basestation files @@ -92,13 +89,13 @@ mkdir(path_bsLocal); [Back to top](#) -### Initialize `agate` +### Initialize __*agate*__ - Open MATLAB -- Add `agate` to the path (with subfolders), if not already done -- Type `agate agate_config.cnf` in the command window and hit enter - - replace `agate_config.cnf` with the name of your configuration file (e.g., `agate_config_sg639_MHI_Apr2022.cnf`) - - if the configuration file is located within the `agate/settings` folder, just the name is sufficient. If it is located elsewhere, specify the fullfile path (e.g., `C:/Users/User.Name/Desktop/agate_config_sgXXX.cnf`) +- Add the `agate` folder to the path (with subfolders), if not already done +- Type `agate ` in the command window and hit enter + - replace `` with the name of your configuration file (*e.g.*, `agate_config_sg639_MHI_Apr2022.cnf`) + - if the configuration file is located within the `agate/settings` folder, just the name is sufficient. If it is located elsewhere, specify the fullfile path (*e.g.*, `C:/Users/User.Name/Desktop/agate_config_sgXXX.cnf`) - Alternatively, simply type `agate` and you will be prompted to select a configuration file [Back to top](#) @@ -110,7 +107,7 @@ Use `downloadBasestationFiles` to automatically download various basestation fil Below is example code to run this step. This can be saved as a script that makes it easy to re-run each time the glider surfaces, or it can be typed directly into the MATLAB Command Window. -```default +```matlab % ensure agate folder is added to the matlab path before proceeding % initialize agate; will prompt for configuration file agate @@ -134,7 +131,7 @@ downloadBasetationFiles(CONFIG, path_bsLocal) ### Extract select piloting parameters and flight metrics -``` +```matlab %% extract various piloting parameters and values reported by .log file pp = extractPilotingParams(CONFIG, path_bsLocal, path_status); diff --git a/docs/images/screenshot_googleEarth_path.PNG b/docs/images/screenshot_googleEarth_path.PNG new file mode 100644 index 0000000..7571c99 Binary files /dev/null and b/docs/images/screenshot_googleEarth_path.PNG differ diff --git a/docs/index.qmd b/docs/index.qmd index 7086c09..5cb79e3 100644 --- a/docs/index.qmd +++ b/docs/index.qmd @@ -1,6 +1,6 @@ --- -title: 'agate' -subtitle: 'Documentation for agate: Acoustic Glider Tools and Environment' +title: '*agate*: Acoustic Glider Analysis Tools and Environment' +# subtitle: page-layout: full --- @@ -13,41 +13,38 @@ library(fontawesome) ``` -# Acoustic Glider Analysis Tools and Environment + `r fa(name = "calendar-check")` *Last Update: `r format(Sys.Date(), "%d %b %Y")`* A collection of MATLAB-based tools for piloting passive acoustic gliders, processing glider positional and environmental data, and analyzing glider-collected passive acoustic data, particularly for surveys of marine mammals. -This package is very much under development, but my hope is that I will maintain a [stable release](https://github.com/sfregosi/agate-public/releases) and then for those interested in the latest functionality, the repository can be cloned. For those interested in contributing to the package, I suggest creating a fork and using pull requests to contribute. - -This code was either developed or most recently updated and tested with MATLAB version 2022b, but has undergone some testing with 2020b. +This package is very much under development and is hosted and version controlled on [GitHub](https://github.com/sfregosi/agate-public). My hope is that I will maintain a [stable release](https://github.com/sfregosi/agate-public/releases) and then for those interested in the latest functionality, the GitHub repository can be cloned. For those interested in [contributing](contribute.html) to the package, I suggest creating a fork and using pull requests to contribute. Please contact me if you have any questions, feedback, or suggestions! +selene [dot] fregosi [at] noaa.gov or [report an issue on GitHub](https://github.com/sfregosi/agate-public/issues/new){target='_blank'} ## Background -These tools were initially developed for use with passive acoustic glider surveys conducted by the OSU/NOAA CIMERS Bioacoustics Lab [bioacoustics.us](https://bioacoustics.us){index='_blank'}. Initial development was for surveys using Seaglider platforms and either the Wideband Intelligent Signal Processor and Recorder (WISPR) or the PMAR-XL recording systems. - -Through NOAA's Uncrewed Systems Initiative (UxS) we received funding to develop and improve these tools into a more broadly applicable and user-friendly tool box that could be used by all interested in conducting glider-based passive acoustic surveys for marine mammals. - -Ready? [Get started](https://sfregosi.github.io/agate-public/get-started.html) +These tools were initially developed for use with passive acoustic glider surveys conducted by the [OSU/NOAA CIMERS Bioacoustics Lab](https://cimers.oregonstate.edu/our-research){index='_blank'}. Initial development was for surveys using Seaglider platforms and either the Wideband Intelligent Signal Processor and Recorder (WISPR) or the Passive Miniaturized Acoustic Recorder XL (PMARXL) recording systems. -Please contact me if you have any questions, feedback, or suggestions! +NOAA's [Uncrewed Systems Operations Center (UxSOC)](https://www.omao.noaa.gov/uncrewed-systems/about-noaa-uncrewed-systems-operations) funded additional development into a more broadly applicable and user-friendly tool box that could be used by all interested in conducting glider-based passive acoustic surveys for marine mammals. -selene [dot] fregosi [at] noaa.gov or [Report an issue on GitHub](https://github.com/sfregosi/agate-public/issues/new){target='_blank'} +**Ready? [Get started](https://sfregosi.github.io/agate-public/get-started.html)** -![diagram showing the different components of agate](images/tool_workflow_v5-01.png){#fig-components width=75%} +![Diagram showing the different components of __*agate*__. Tools can be broken up into two main categories: piloting tools, which are used for survey planning and execution, and processing tools which are used in post-mission analyses. The processing tools can further be broken up into flight, acoustic, and cetacean encounter processing tools. The cetacean encounter processing tools can incorporate externally identified cetacean detections (white balloon) or process on-board detections if available.](images/tool_workflow_v5-01.png){#fig-components width=75%} ## Dependencies This code was either developed or most recently updated and tested with MATLAB version 2022b, but has undergone some testing with 2020b. -MATLAB toolboxes: +#### MATLAB toolboxes - Mapping Toolbox -`agate` requires the MATLAB Mapping toolbox. In the future, we may try modify the mapping tools to work with the freely available `m_map` package to make it more accessible, but are not there yet! +__*agate*__ requires the MATLAB Mapping toolbox. In the future, we may try modify the mapping tools to work with the freely available `m_map` package to make it more accessible, but are not there yet! -This package requires a few resources from MATLAB File Exchange. They come packaged within the utils/fileExchange folder. More info about each can be found below: +#### MATLAB File Exchange + +This package requires a few resources from MATLAB File Exchange. They come packaged within the `utils/fileExchange` folder. More info about each can be found below: - [SSH/SFTP/SCP For MATLAB (v2)](https://www.mathworks.com/matlabcentral/fileexchange/35409-ssh-sftp-scp-for-matlab-v2) *David Freedman (2023). SSH/SFTP/SCP For Matlab (v2), MATLAB Central File Exchange. Retrieved April 24, 2023.* @@ -60,10 +57,14 @@ This package requires a few resources from MATLAB File Exchange. They come packa *Gabriel Ruiz-Martinez (2023). Seawater density from salinity, temperature and pressure, MATLAB Central File Exchange. Retrieved April 24, 2023.* -Basemap rasters: +#### Basemap rasters + +A raster file is needed to plot bathymetry data on any of the maps. Detailed basemaps are available from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). + +There are tradeoffs in the resolution of basemap raster selected. -- A raster file is needed to plot bathymetry data on any of the maps. Detailed basemaps are available from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). -Depending on the needed resolution, the 60 arc second .tiff file may be sufficient, and is a reasonable size for download and plotting, but is a bit slow to load. Alternatively, finer resolution (15-60 arc second resolution options) for the specific area of interest can be extracted using the [ETOPO Grid Extract tool](https://www.ncei.noaa.gov/maps/grid-extract/) to download .tiff files in a smaller file size for faster loading and plotting. +- High resolution basemaps are necessary for fine-scale piloting to ensure the glider does not hit the bottom, but they require manual export of a .tiff from NCEI. 15 arc second is the finest resolution option and can be extracted for the specific area of interest using the [ETOPO Grid Extract tool](https://www.ncei.noaa.gov/maps/grid-extract/). The smaller the area extracted, the faster it will load and plot. +- Lower resolution may be sufficient for post-mission plotting. The global 60 arc second tiff is a good mid-resolution option that is easy, although slow, to download, import, and plot. Alternatively, 60 arc second resolution for just a specific area of interest can be extracted with the [ETOPO Grid Extract tool](https://www.ncei.noaa.gov/maps/grid-extract/) providing a smaller file size and faster loading/plotting. ## Disclaimer diff --git a/docs/survey-planning.qmd b/docs/mission-planning.qmd similarity index 55% rename from docs/survey-planning.qmd rename to docs/mission-planning.qmd index 692a9bb..6c37352 100644 --- a/docs/survey-planning.qmd +++ b/docs/mission-planning.qmd @@ -1,28 +1,30 @@ --- -title: "Survey planning" -subtitle: 'Suggested workflows for the survey planning tools' +title: "Mission planning" +subtitle: 'Suggested workflows for the mission planning tools' --- ```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) library(fontawesome) ``` -This page is meant to provide some examples of how the survey planning piloting functions maybe be used ahead of a planned mission. The below sections include examples for creating a targets file, creating a planned survey map, plotting a bathymetry profile for the planned track, and summarizing total mission distance and duration. +This page is meant to provide some examples of how the mission planning piloting functions maybe be used ahead of a planned mission. The below sections include examples for creating a targets file, creating a planned mission map, plotting a bathymetry profile for the planned track, and summarizing total mission distance and duration. -All code on this page combined in the [`workflow_surveyTrackPlanning.m`](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/workflow_surveyTrackPlanning.m) in the 'example_workflows' folder within the 'agate-public/agate' folder. +All code on this page combined in the [`workflow_missionTrackPlanning.m`](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/workflow_missionTrackPlanning.m) in the `example_workflows` folder within the `agate-public/agate` folder. Details for each function used below (inputs, outputs, etc) are available within the standard MATLAB-type documentation in the header of each function and include a detailed description, info on input and output arguments, and examples. These details can be pulled up by typing `doc function` or `help function` within the MATLAB Command Window. ### Initialization -To run any of the `agate` functions, the toolbox must be initialized with a configuration file. +To run any of the __*agate*__ functions, the toolbox must be initialized with a configuration file. -No configuration file yet? Go back to [Get started - Create configuration files](get-started.html#create-configuration-files). At a minimum, for `agate` to initialize, the configuration file must include the top *required* portion. +No configuration file yet? Go to the [Configuration Guide](configuration.html). The examples on this page include some plotting, so the ` OPTIONAL - plotting ` section must be complete. -The examples on this page include some plotting, so the ` OPTIONAL - plotting ` section should be complete. - -```default -% !ensure agate is on the path! +```matlab +% !make sure agate is on the path! % initialize with specified configuration file, 'agate_config.cnf' agate agate_config.cnf @@ -38,9 +40,32 @@ global CONFIG [Back to top](#) +### Create planned track using Google Earth Pro + +Use Google Earth Pro (the desktop app) to plan a mission trackline and save it as a `.kml`. + +- Create a path object by navigating to the *Add* menu and selecting *Path* +- A popup will appear where you can name the path +- While the popup is still the top window, the cursor will become a square; can click anywhere on the map to make a waypoint +- Continue to click to make the desired waypoints +- Waypoints can be moved by clicking and dragging +- Right-clicking on a given waypoint will delete it +- View the total track length under the *Measurements* tab +- Once you are happy with the track, hit *Ok* in the popup + +The path must be saved as a `.kml` containing just a single path/track. To properly save: + +- Within Google Earth, right-click on the path name in the panel on the left hand side +- Select *Save place as* +- Change the file type from `.kmz` to `.kml` and hit *Save* + +An example is located in [`agate/example_workflows/exampleTrack.kml`](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/exampleTrack.kml) + +![Example path in Google Earth saved as a `.kml` and ready to be converted to a `targets` file.](images/screenshot_googleEarth_path.PNG){#fig-example-kml fig-align='center' width=90%} + ### Create targets file from .kml -The `makeTargetsFile` function will read in an existing .kml (that contains a single path) and use it to create a properly formatted Seaglider 'targets' file. The generated targets file will be named 'targets_' plus the name of the .kml file (e.g., 'targets_exampleTrack'). It will contain header information with the glider and mission information defined in `CONFIG`, the date it was created, and the specified radius. +The `makeTargetsFile` function will read in an existing `.kml` (that contains a single path) and use it to create a properly formatted Seaglider `targets` file. The generated targets file will be named `targets_` plus the name of the `.kml` file (*e.g.*, `targets_exampleTrack`). It will contain header information with the glider and mission information defined in `CONFIG`, the date it was created, and the specified radius. ```default / Targets file for mission sgXXX_Location_Mon20XX @@ -49,18 +74,15 @@ The `makeTargetsFile` function will read in an existing .kml (that contains a si / template WPxx lat=DDMM.MMMM lon=DDDMM.MMMM radius=XXXX goto=WPzz ``` -The .kml can be made in Google Earth [MORE INFO ON THAT TO COME LATER!] -Track must be saved as a kml containing just a single track/path. To properly save: within Google Earth, right click on track name in left panel, select save place as, change file type from .kmz to .kml, save. -An example is located in ['agate/example_workflows/exampleTrack.kml'](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/exampleTrack.kml) - Waypoint names can be generated one of three ways: -(1) prefix: specify a character string prefix in the function call and alphanumeric names will be created automatically. -(2) file: list the desired waypoint names within a simple .txt files, with one name per line and the number of waypoint names must equal the number of waypoints in the .kml; the function will prompt to select the file. An example can be found in ['agate/example_workflows/waypointNames.txt'](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/waypointNames.txt) -(3) manual: waypoint names are manually entered in the command window within the function call +**1. prefix**: specify a character string prefix in the function call and alphanumeric names will be created automatically. -```default +**2. file**: list the desired waypoint names within a simple `.txt` file, with one name per line and the number of waypoint names must equal the number of waypoints in the `.kml`; the function will prompt to select the file. An example can be found in [`agate/example_workflows/waypointNames.txt`](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/waypointNames.txt) + +**3. manual**: waypoint names are manually entered in the Command window within the function call +```matlab % specify file name to .kml path kmlFile = fullfile(CONFIG.path.mission, 'exampleTrack.kml'); % OR @@ -121,7 +143,7 @@ export_fig(fullfile(CONFIG.path.mission, [CONFIG.glider '_' CONFIG.mission, ... ``` -![](images/examplePlots/sg639_MHI_Apr2023_plannedTrack.png){#planned-map fig-align='center' width=60%} +![Planned track map with each waypoint labeled.](images/examplePlots/sg639_MHI_Apr2023_plannedTrack.png){#fig-planned-map fig-align='center' width=60%} [Back to top](#) @@ -129,10 +151,11 @@ export_fig(fullfile(CONFIG.path.mission, [CONFIG.glider '_' CONFIG.mission, ... It can be useful to have a profile of the bathymetry the planned track will traverse, to highlight periods where the glider's target dive depth may need to be adjusted more shallow, or can be extended deeper. -An indicator line (dashed red line) will be plotted at 990 m as the default (max $D_TGT for Seagliders) but can be specified to a different value with the `yLine` argument. +See [Dependecies - Basemap rasters](index.html#basemap-rasters) for more info on how to select and download bathymetry rasters. -```default +An indicator line (dashed red line) will be plotted at 990 m as the default (max `$D_TGT` for Seagliders) but can be specified to a different value with the `yLine` argument. +```matlab % can specify bathymetry file bathyFile = 'C:\GIS\etopo\ETOPO2022_bedrock_30arcsec_MHI.tiff'; plotTrackBathyProfile(CONFIG, targetsFile, bathyFile) @@ -146,14 +169,14 @@ exportgraphics(gcf, fullfile(CONFIG.path.mission, [CONFIG.glider '_' ... 'Resolution', 300) ``` -![](images/examplePlots/sg639_MHI_Apr2023_targetsBathymetryProfile_targets.png){#track-bathy fig-align='center' width=80%} +![Bathymetry profile for set targets file which shows the seafloor depth at each target and in the straight line between each target. The 990 m depth limit is marked with a dashed line.](images/examplePlots/sg639_MHI_Apr2023_targetsBathymetryProfile_targets.png){#fig-track-bathy fig-align='center' width=80%} ### Summarize planned track The below code reads in an existing (or newly created!) targets file and will loop through to calculate the distance between each waypoint and then print out the total planned track distance. If an estimate of glider speed (in km/day) is available, that can be used to estimate mission duration. -```default +```matlab % if no targetsFile specified, will prompt to select [targets, targetsFile] = readTargetsFile(CONFIG); % OR specify targetsFile variable from above @@ -178,12 +201,9 @@ fprintf(1, 'Estimated mission duration, at %i km/day: %.1f days\n', avgSpd, ... The output will look something like this: -``` ->> +```matlab Total tracklength for targets_exampleTrack: 54 km Estimated mission duration, at 15 km/day: 3.6 days ->> - ``` [Back to top](#) diff --git a/docs/piloting-functions.qmd b/docs/piloting-functions.qmd index f9d11e5..faa4bba 100644 --- a/docs/piloting-functions.qmd +++ b/docs/piloting-functions.qmd @@ -4,10 +4,14 @@ subtitle: 'Suggested workflows for using the piloting functions' --- ```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) library(fontawesome) ``` -This page is meant to demonstrate some suggested workflows and examples of how the piloting functions may be used in an active survey to help the pilot download basestation files, create monitoring plots, and monitor mission progress. +This page is meant to demonstrate some suggested workflows and examples of how the piloting functions may be used in an active mission to help the pilot download basestation files, create monitoring plots, and monitor mission progress. All code on this page combined in the [`workflow_downloadScript.m`](https://github.com/sfregosi/agate-public/tree/main/agate/example_workflows/workflow_downloadScript.m) in the 'example_workflows' folder within the 'agate-public/agate' folder. @@ -15,14 +19,14 @@ Details for each piloting function (inputs, outputs, etc) are available within t ### Initialization -To run any of the `agate` piloting functions, the toolbox must be initialized with a configuration file. +To run any of the __*agate*__ functions, the toolbox must be initialized with a configuration file. -No configuration file yet? Go back to [Get started - Create configuration files](get-started.html#create-configuration-files). At a minimum, for `agate` to initialize, the configuration file must include the top *required* portion. If `agate` will be used to download files from the basestation to the local computer, a `basestation.cnf` file is required and the ` OPTIONAL - working with the basestation ` section must be completed. +No configuration file yet? Go to the [Configuration Guide](configuration.html). If __*agate*__ will be used to download files from the basestation to the local computer, a `basestation.cnf` file is required and the ` OPTIONAL - working with the basestation ` section of the mission configuration file must be completed. The examples on this page include some plotting, so the ` OPTIONAL - plotting ` section should be complete. Examples below will also include some outputs for both the **PMAR** (`pm`) and **WISPR** (`ws`) acoustic systems. Those are specified in the configuration file and the examples below will indicate which system is 'active' for each example. This is controlled with the `CONFIG.pm.loggers` and `CONFIG.ws.loggers` options in the ` OPTIONAL - acoustics ` section of the configuration file. -```default -% !ensure agate is on the path! +```matlab +% !make sure agate is on the path! % initialize with specified configuration file, 'agate_config.cnf' agate agate_config.cnf @@ -37,9 +41,9 @@ agate ### Downloading files from the basestation -`agate` can use SSH to automatically download any new files present on the basestation to a local computer for further processing. The below folder structure is the suggested approach for working with `agate` and will ensure later functions are looking in the right spots. It may be possible to customize the folder structure but may result in bugs; if that happens, please report them and we can try to fix it! +__*agate*__ can use SSH to automatically download any new files present on the basestation to a local computer for further processing. The below folder structure is the suggested approach for working with __*agate*__ and will ensure later functions are looking in the right spots. It may be possible to customize the folder structure but may result in bugs; if that happens, please report them and we can try to fix it! -```default +```matlab % specify the local piloting folder for this trip in CONFIG.path.mission % set up nested folders for basestation files and piloting outputs path_status = fullfile(CONFIG.path.mission, 'flightStatus'); % where to store output plots/tables @@ -56,11 +60,11 @@ downloadBasetationFiles(CONFIG, path_bsLocal) ### Creating a piloting parameters (`pp`) variable -Many of the plotting and other piloting-related functions within `agate` rely on a `pp` (piloting parameters) variable - a large table with various outputs from the .nc and .log files compiled in one place. Use the `extractPilotingParams` to create this table. +Many of the plotting and other piloting-related functions within __*agate*__ rely on a `pp` (piloting parameters) variable - a large table with various outputs from the .nc and .log files compiled in one place. Use the `extractPilotingParams` to create this table. The last argument, `preload`, is used to specify if the table should be made from scratch (does not load any previously created table), or if a previous table should be loaded and any new dives appended to that table. As more dives occur during a mission, creating a new table each time can get slow, so saving the previous table and setting `preload` to `1` can save processing time. If the piloting parameters table is saved in the default location shown in the `save()` step below, then it will be loaded automatically. If it is saved elsewhere, the function will prompt to select the correct .mat to load. -```default +```matlab % create piloting parameters (pp) table from downloaded basestation files pp = extractPilotingParams(CONFIG, fullfile(CONFIG.path.mission, 'basestationFiles'), ... fullfile(CONFIG.path.mission, 'flightStatus'), 0); @@ -68,32 +72,31 @@ fullfile(CONFIG.path.mission, 'flightStatus'), 0); % save it to the default location save(fullfile(CONFIG.path.mission, 'flightStatus', ['diveTracking_' CONFIG.glider '.mat']), 'pp'); - ``` [Back to top](#) ### Mid-mission plots -For detail on all available plotting functions, see the [Plots](plotting-functions){target='_blank'} page. A highlight of those most useful for piloting are below. +For detail on all available plotting functions, see the [Plots](plotting-functions.html){target='_blank'} page. A highlight of those most useful for piloting are below. The below steps require a `pp` variable. If it was created already, this first step is not needed. -```default +```matlab % load existing pp table load(fullfile(CONFIG.path.mission, 'flightStatus', ['diveTracking_' CONFIG.glider '.mat'])) ``` #### Map -...with target waypoints, dives completed thus far, and vector arrows for the currents. +*...with target waypoints, dives completed thus far, and vector arrows for the currents.* In this example, bathymetry is plotted, specified by `CONFIG.map.bathyFile`. That last argument can be left out to not plot bathymetry (which can be slow depending on the resolution of the selected bathymetry raster). If the last argument is set to `1`, a prompt will appear to select the correct bathymetry file. -The below example code saves the map both as a .fig file and a .png. The .fig version will be a very large file, if bathymetry is included, but it is useful for reopening in MATLAB and being able to zoom and move around in the plot. The .png is good for a quick easy overview and is sharable, but doesn't allow the interactive zooming that a pilot may need. +The below example code saves the map both as a `.fig` file and a `.png`. The `.fig` version will be a very large file, if bathymetry is included, but it is useful for reopening in MATLAB and being able to zoom and move around in the plot. The .png is good for a quick easy overview and is sharable, but doesn't allow the interactive zooming that a pilot may need. -High resolution bathymetry TIFF files can be downloaded from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). Depending on the needed resolution, the 60 arc second TIFF may be sufficient, and is a reasonable size for download and plotting, but is slow to load because it covers the whole globe. Alternatively, finer resolution (15-60 arc second resolution options) for the specific area of interest can be extracted using the [ETOPO Grid Extract tool](https://www.ncei.noaa.gov/maps/grid-extract/) to download a TIFF that covers a smaller area, but in a smaller sized file for faster loading and plotting. +High resolution bathymetry TIFF files can be downloaded from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). See [Dependecies - Basemap rasters](index.html#basemap-rasters) for more info on how to select and download bathymetry rasters. -```default +```matlab % print map **SLOWISH** - figNumList(1) targetsFile = fullfile(CONFIG.path.mission, 'targets'); plotGliderPath_etopo(CONFIG, pp, targetsFile, CONFIG.map.bathyFile); @@ -107,11 +110,11 @@ exportgraphics(gca, fullfile(path_status, [CONFIG.glider '_map.png']), ... [Back to top](#) #### Monitoring plots -...for humidity, internal pressure, battery consumption, power draw, and acoustic system status. +*...for humidity, internal pressure, battery consumption, power draw, and acoustic system status.* Individual pilots may find some or all or none of these plots useful, but here are just a few examples. The example code has the option to save the figures with the `print()` function, but that is optional. To automatically plot, save, and then close the figures, just add a `close` command after print (see example with third and fifth plots below). -```default +```matlab % humidity and pressure - figNumList(2) plotHumidityPressure(CONFIG, pp) print(fullfile(path_status, [CONFIG.glider '_humidityPressure.png']), '-dpng') @@ -138,14 +141,14 @@ print(fullfile(path_status, [CONFIG.glider '_minimumVoltage.png']), '-dpng') If the glider is running a PMAR acoustic system (and `CONFIG.pm.loggers = 1` is set in the configuration file), the free space remaining on each SD card will be plotted by `plotBattUseFreeSpace`. Additionally, storage space used per minute, by dive, and over time can be plotted with `plotPmUsed`: -```default +```matlab % PMAR space used per minute and over time plotPmUsed(CONFIG, pp) ``` If the glider is running a WISPR acoustic system (and `CONFIG.ws.loggers = 1` is set in the configuration file), and using the on-board ERMA sperm whale detector, detection events can be plotted for a single dive. The plots show ICI (inter-click-interval) over the event duration and as a histogram. This plot is interactive and allows the user to click backwards through previous dives; a specific dive can be specified in the function call or `end` can be used to plot the most recent dive. The plot also provides a 'reference plot' showing what a true detection event of both a group of sperm whales or an individual sperm whale would look like. This plot can be used by the pilot to compare and validate the incoming detections. -```default +```matlab % plot detection events from the most recent dive plotErmaDetections(CONFIG, path_bsLocal, pp.diveNum(end)) ``` @@ -154,16 +157,16 @@ plotErmaDetections(CONFIG, path_bsLocal, pp.diveNum(end)) ### Printing errors and mission speed and duration information -The Seaglider .log file provides a summary of any errors that occurred during each dive, but they are just a list of integers that then have to be compared to a manual (and differ by Rev B vs Rev E!) so there is a function to print out all non-zero errors and a short descriptor of the type. Any dive can be specified in the second argument, or just the most recent dive as shown below. +The Seaglider `.log` file provides a summary of any errors that occurred during each dive, but they are just a list of integers that then have to be compared to a manual (and differ by Rev B vs Rev E!) so there is a function to print out all non-zero errors and a short descriptor of the type. Any dive can be specified in the second argument, or just the most recent dive as shown below. -```default +```matlab % print out errors with info on type, for the most recent dive printErrors(CONFIG, size(pp,1), pp) ``` The `printTravelMetrics` and `printRecoveryMetrics` functions calculate several summary values for the glider's average speed, progress along the trackline, and estimated time of arrival at the recovery point (last waypoint in targets file). These outputs can be printed to the MATLAB Command Window if the last argument is set to `1`, otherwise they will be stored in a structure `tm`. -```default +```matlab % print avg speed and rough estimate of total mission duration tm = printTravelMetrics(CONFIG, pp, fullfile(CONFIG.path.mission, 'targets'), 1); diff --git a/docs/plotting-functions.qmd b/docs/plotting-functions.qmd index 13e9cfb..d56e4fd 100644 --- a/docs/plotting-functions.qmd +++ b/docs/plotting-functions.qmd @@ -4,20 +4,24 @@ subtitle: 'Overview of all plotting functions' --- ```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) library(fontawesome) ``` -This page provides example calls and outputs for each of the plotting functions contained within `agate`. There are plotting functions that may be useful during survey planning, active piloting, and in post-processing and analysis, and the plots are grouped by those categories below. +This page provides example calls and outputs for each of the plotting functions contained within __*agate*__. There are plotting functions that may be useful during mission planning, active piloting, and in post-processing and analysis, and the plots are grouped by those categories below. -Basic details for each plotting function are available within the standard MATLAB-type documentation in the header of each function and include a detailed description, info on input and output arguments, and examples. These details can be pulled up by typing `doc function` or `help function` within the MATLAB Command Window. +Basic details for each plotting function are available within the standard MATLAB-type documentation in the header of each function and include a detailed description, info on input and output arguments, and examples. These details can be pulled up by typing `doc ` or `help ` within the MATLAB Command Window. ## Initialization -To run any of the `agate` plotting functions, the toolbox must be initialized with a configuration file. +To run any of the __*agate*__ plotting functions, the toolbox must be initialized with a configuration file. No configuration file yet? Go back to [Get started - Create configuration files](get-started.html#create-configuration-files). At a minimum, the configuration file must include the top *required* portion, as well as the *optional - plotting* section. -```default +```matlab % ensure agate is on the path % initialize with specified configuration file, 'agate_config.cnf' @@ -29,13 +33,12 @@ agate Most of the plotting functions rely on a `pp` (piloting parameters) variable that is a large table with various outputs from the .nc and .log files compiled in one place. If this has already be created, it can be loaded directly. If not, use `extractPilotingParams` to build it. -```default +```matlab % load existing pp table load(fullfile(CONFIG.path.mission, 'flightStatus', ['diveTracking_' CONFIG.glider '.mat'])) % create new pp table -pp = extractPilotingParams(CONFIG, fullfile(CONFIG.path.mission, 'basestationFiles'), ... -fullfile(CONFIG.path.mission, 'flightStatus'), 0); +pp = extractPilotingParams(CONFIG, fullfile(CONFIG.path.mission, 'basestationFiles'), fullfile(CONFIG.path.mission, 'flightStatus'), 0); % zero as last argument creates it from scratch (does not load any previous tables) save(fullfile(CONFIG.path.mission, 'flightStatus', ['diveTracking_' CONFIG.glider '.mat']), 'pp'); @@ -45,28 +48,28 @@ save(fullfile(CONFIG.path.mission, 'flightStatus', ['diveTracking_' CONFIG.glide ## Piloting plots -Some of the below piloting plots are also available withing the Seaglider Piloting Tools, but `agate` allows for more automated plotting (without the GUI) and tries to improve on some of those existing plots, either with better labels, more detail, or addition of acoustic inputs. +Some of the below piloting plots are also available within the Seaglider Piloting Tools, but __*agate*__ allows for more automated plotting (without the GUI) and tries to improve on some of those existing plots, either with better labels, more detail, or addition of acoustic inputs. ### Humidity and pressure -```default +```matlab plotHumidityPressure(CONFIG, pp) ``` -![](images/examplePlots/sg639_humidityPressure.png){#humid-press fig-align='center' width=50%} +![Example humidity and pressure plot.](images/examplePlots/sg639_humidityPressure.png){#humid-press fig-align='center' width=50%} ### Minimum voltage -Rev B gliders (e.g., SG639 - left side) will have two lines - one each for the '24V' and '10V' batteries (even if both are 15V), while Rev E gliders (e.g., SG679 - right side) will also have two lines, but the `10V` line will primarily remain at 15V and only the `24V` line will decrease over the mission. +Rev B gliders (*e.g.*, SG639 - left side) will have two lines - one each for the '24V' and '10V' batteries (even if both are 15V), while Rev E gliders (*e.g.*, SG679 - right side) will also have two lines, but the `10V` line will primarily remain at 15V and only the `24V` line will decrease over the mission. -```default +```matlab plotMinVolt(CONFIG, pp) ``` ::: {#min-volt layout-ncol=2} -![](images/examplePlots/sg639_minimumVoltage.png){#min-volt-639} +![Minimum voltage plot for SG639 (Rev B glider).](images/examplePlots/sg639_minimumVoltage.png){#min-volt-639} -![](images/examplePlots/sg679_minimumVoltage.png){#min-volt-679} +![Minimum voltage plot for SG679 (Rev E glider).](images/examplePlots/sg679_minimumVoltage.png){#min-volt-679} ::: ### Battery remaining and PAM free space @@ -75,52 +78,52 @@ If the glider is using a PMAR acoustic system, the free space remaining on each The 30% battery line and 35 GB PMAR line are hard coded into the plot, but could be modified within the function. The 'target mission duration' line is defined by `CONFIG.tmd`, as set in the configuration file. -```default +```matlab plotBattUseFreeSpace(CONFIG, pp) ``` -![](images/examplePlots/sg639_battUseFreeSpace.png){#batt-free-639 fig-align='center' width=50%} +![Remaining battery and PAM storage space for SG639 with PMARXL installed and operating.Note glider battery was not full at start of mission so remaining battery starts at just over 60%.](images/examplePlots/sg639_battUseFreeSpace.png){#batt-free-639 fig-align='center' width=50%} ### Voltage use by device -If the glider is using a PMAR or WISPR acoustic system, power draw of the acoustic system will be included with measures of pitch, roll, and VBD. +If the glider is using a PMAR or WISPR acoustic system, power draw of the acoustic system will be included with measures of pitch, roll, and VBD (variable buoyancy device). -```default +```matlab plotVoltagePackUse(CONFIG, pp) ``` ::: {#use layout-ncol=2} -![](images/examplePlots/sg639_usageByDevice.png){#use-639} +![Energy use by device, reported for each dive for SG639 (Rev B glider).](images/examplePlots/sg639_usageByDevice.png){#use-639} -![](images/examplePlots/sg679_usageByDevice.png){#use-679} +![Energy use by device, reported for each dive for SG679 (Rev E glider).](images/examplePlots/sg679_usageByDevice.png){#use-679} ::: ### Voltage use by device, normalized by dive duration -If the glider is using a PMAR or WISPR acoustic system, power draw of the acoustic system will be included with measures of pitch, roll, and VBD. +If the glider is using a PMAR or WISPR acoustic system, power draw of the acoustic system will be included with measures of pitch, roll, and VBD (variable buoyancy device). -```default +```matlab plotVoltagePackUse_norm(CONFIG, pp) ``` ::: {#use-norm layout-ncol=2} -![](images/examplePlots/sg639_usageByDevice_normalized.png){#use-norm-639} +![Energy use by device, normalized by dive duration for SG639 (Rev B glider).](images/examplePlots/sg639_usageByDevice_normalized.png){#use-norm-639} -![](images/examplePlots/sg679_usageByDevice_normalized.png){#use-norm-679} +![Energy use by device, normalized by dive duration for SG679 (Rev E glider).](images/examplePlots/sg679_usageByDevice_normalized.png){#use-norm-679} ::: ### Map -The map will include the waypoints and track from the targets file and will plot the surface positions of each dive (yellow dots), with red lines connecting surface positions, currents plotted as blue vector arrows, and the next target waypoint with a green circle. +The map will include the waypoints and track from the `targets` file and will plot the surface positions of each dive (yellow dots), with red lines connecting surface positions, currents plotted as blue vector arrows, and the next target waypoint with a green circle. -Map extent (latitude and longitude limits), the location of the north arrow, and the location and scale of the scale bar are all set within the configuration file, with example values below. `CONFIG.latLim` and `CONFIG.map.lonLim` are required to make the map; the north arrow and scale bar are optional and if those settings do not exist in `CONFIG` they will not be included in the plot. +Map extent (latitude and longitude limits), the location of the north arrow, and the location and scale of the scale bar are all set within the [mission configuration file](configuration.html#optional-plotting-settings), with example values below. `CONFIG.latLim` and `CONFIG.map.lonLim` are required to make the map; the north arrow and scale bar are optional and if those settings do not exist in `CONFIG` they will not be included in the plot. In this example, bathymetry is plotted, specified by `CONFIG.map.bathyFile`. That last argument can be left out to not plot bathymetry (which can be slow depending on the resolution of the selected bathymetry raster). If the last argument is set to `1`, a prompt will appear to select the correct bathymetry file. -High resolution bathymetry TIFF files can be downloaded from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). Depending on the needed resolution, the 60 arc second TIFF may be sufficient, and is a reasonable size for download and plotting, but is slow to load because it covers the whole globe. Alternatively, finer resolution (15-60 arc second resolution options) for the specific area of interest can be extracted using the [ETOPO Grid Extract tool](https://www.ncei.noaa.gov/maps/grid-extract/) to download a TIFF that covers a smaller area, but in a smaller sized file for faster loading and plotting. +High resolution bathymetry TIFF files can be downloaded from [NCEI](https://www.ncei.noaa.gov/products/etopo-global-relief-model). See [Dependecies - Basemap rasters](index.html#basemap-rasters) for more info on how to select and download bathymetry rasters. -```default +```matlab targetsFile = fullfile(CONFIG639.path.mission, 'targets'); % map set up configurations - all should be specified in configuration file @@ -142,10 +145,23 @@ plotGliderPath_etopo(CONFIG, pp, targetsFile, 1); plotGliderPath_etopo(CONFIG, pp, targetsFile); ``` -![](images/examplePlots/sg639_map.png){#map fig-align='center' width=80%} +![Example piloting map with planned track and waypoints in black, realized track in red with surface positions in yellow, and depth averaged currents as blue vector arrows. The current target waypoint is circled in green. ](images/examplePlots/sg639_map.png){#map fig-align='center' width=80%} [Back to top](#) +## Mission summary plots + +### Final trackline map + +### Dive profile + +### Sound speed profile + +## Analysis plots + +### Cetacean event maps + + diff --git a/docs/theme-dark.scss b/docs/theme-dark.scss index 9e645de..2ee1f2b 100644 --- a/docs/theme-dark.scss +++ b/docs/theme-dark.scss @@ -4,10 +4,26 @@ $font-family: "Atkinson Hyperlegible", sans-serif; +/*-- color definitions --*/ +$beaver-orange: #D73F09 !default; +$stratosphere: #006A8E !default; +$star-canvas: #003B5C !default; +$rogue-wave: #0D5257 !default; +$crater: #8E9089 !default; +$primary: $beaver-orange !default; +$success: $star-canvas !default; +$link-color: $stratosphere !default; +$progress-bg: #e8e8e7 !default; +$code-color: $rogue-wave !default; +$code-bg: #e8e8e7 !default; +$code-block-bg: #e8e8e7 !default; +$btn-code-copy-color: $rogue-wave !default; + + // Base document colors $body-bg: #181818; $body-color: white; -$link-color: #75AADB; +$link-color: #D73F09; $light: #525252; @@ -24,3 +40,9 @@ $popover-bg: #242424; // Bootstrap inputs $input-bg: #242424; + +/* code blocks */ +pre { + color: $rogue-wave; + background-color: #e8e8e7; +} diff --git a/docs/theme.scss b/docs/theme.scss index bdefa89..aef3a5d 100644 --- a/docs/theme.scss +++ b/docs/theme.scss @@ -38,13 +38,19 @@ main ul ul ul{ margin-bottom: 0; } - +/*-- SIDEBAR NAVIGATION --*/ +div.sidebar-title { + color: white; +} +div.sidebar-tools-main { + color: white; +} /*-- SECTION HEADER STYLING --*/ h1 { -letter-spacing: 0.8px; -line-height: 1.1em; -margin-top: 1.5rem; -margin-bottom: 0.5rem; + letter-spacing: 0.8px; + line-height: 1.1em; + margin-top: 1.5rem; + margin-bottom: 0.5rem; } h2 {