Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
218 commits
Select commit Hold shift + click to select a range
a9639ad
New XBT format code (xbt_parse.py) and all the changes to config and …
BecCowley Dec 8, 2023
c8b19d9
Change check for test probe to earlier in the code to skip unnecessar…
BecCowley Dec 11, 2023
cf4f6fe
Add check & handling for PL code where profile lengths are different …
BecCowley Dec 11, 2023
accd732
Add code to handle other parameters such as sound speed and salinity.…
BecCowley Dec 19, 2023
4cb8efd
Update utilities to handle other profiles such as sound velocity and …
BecCowley Dec 19, 2023
5bf4580
Improve byte array handling code, some minor comment edits. More debu…
BecCowley Dec 19, 2023
bf7ecff
Add some code to handle the older files where longitude was changed f…
BecCowley Dec 19, 2023
0c3d9a6
Make improvements to the PLA handling.
BecCowley Dec 19, 2023
1f6a549
Check if TEMP profile is in the file before reading and processing.
BecCowley Dec 19, 2023
20848fb
Improve TEMP uncertainty variable creation to make sure it is the sam…
BecCowley Dec 19, 2023
c6adc36
Handle the case where more than one profile in the file and read thes…
BecCowley Dec 19, 2023
5907f14
Platform code was not being passed in if the variable was empty. Now …
BecCowley Dec 19, 2023
b36ad13
Update time handling as sometimes zero padding is missing from the le…
BecCowley Dec 19, 2023
028f5f4
Create a QC field for additional RAW variables where QC flags exist (…
BecCowley Dec 19, 2023
2effa77
Fix bug created from previous change, handle string trimming for stri…
BecCowley Dec 19, 2023
1def839
Add a TODO comment
BecCowley Jan 22, 2024
62cfc3d
Add rounding to latitude, longitude and depth variables
BecCowley Mar 18, 2024
2bc9e1a
Add try catch around some of the variables to handle empty fields.
BecCowley Mar 18, 2024
5091b1b
Comment out the SYST recorder type list for now as it gives incorrect…
BecCowley Mar 18, 2024
677ca49
Add shell information for bash
BecCowley Apr 24, 2024
3cc0e64
Better handling of previous values as they are numbers.
BecCowley Apr 24, 2024
b8bd1fe
Combine DATE and TIME correction flags into one TIME variable.
BecCowley Apr 24, 2024
b5f71d8
If no depth data, skip this file conversion.
BecCowley Apr 24, 2024
38617dd
Include sorting of keys data at reading time.
BecCowley Apr 24, 2024
905d6e5
Add missing geospatial information into global attributes.
BecCowley Apr 24, 2024
0757acd
Adjust histories handling to allow files with empty history records t…
BecCowley Apr 24, 2024
6be03c8
If incorrect recorder type in the file, log the issue and change RCT$…
BecCowley Apr 24, 2024
d0e4b30
Add Scripps as an institute.
BecCowley Apr 24, 2024
4fe3822
Note to operator when QC flags are found in the profile, but continue…
BecCowley Apr 24, 2024
c249733
Minor formatting changes.
BecCowley Apr 26, 2024
785a31e
Trim variable arrays to match the number of depths if the array is no…
BecCowley Apr 26, 2024
ad6d50f
Fix bug in handling of TE flag.
BecCowley Apr 26, 2024
670c6a2
Merge branch 'aodn:master' into XBTnewNCFormat
BecCowley Apr 26, 2024
b3222f2
Some more updates from another repo
Apr 26, 2024
a571f76
Merge branch 'XBTnewNCFormat' into XBTnewFormat2
BecCowley Apr 26, 2024
8d3ad3a
Merge pull request #9 from BecCowley/XBTnewFormat2
BecCowley Apr 26, 2024
a5f1498
Ensure ship_name is always included in the file, even if unknown.
BecCowley May 10, 2024
b95ce86
Ensure recorder type global variable included even if unknown.
BecCowley May 10, 2024
589bf99
LONGITUDE is in degrees_east, adjust code to calculate correctly if d…
BecCowley May 10, 2024
9f63a85
Change the N_HISTORY dimension to unlimited.
BecCowley May 27, 2024
3a10c28
Add first go at Turo netcdf to iMOS netcdf conversion. Code courtesy …
BecCowley Jul 22, 2024
594d6d6
Some major re-writes to turo2IMOSnetcdf code to work more flexibly an…
BecCowley Aug 26, 2024
66363db
Add the test probe code (104) to the XBT config file and adjust the u…
BecCowley Aug 26, 2024
bd248f0
Adjust the project name and some attributes required in the nc attrib…
BecCowley Aug 26, 2024
4dfed1b
Include ship IMO information from vocabularies.
BecCowley Aug 26, 2024
36113bc
Update the accept/reject code handling to read a table. Debug as requ…
BecCowley Sep 24, 2024
041c043
Update flag file creation allowing for two code variables, xbt_accept…
BecCowley Sep 24, 2024
413c24b
Update the accept and reject csv tables.
BecCowley Sep 24, 2024
a0104f9
Add the accept and reject code variables.
BecCowley Sep 24, 2024
77b3f83
Bug fix LONGITUDE was being read from the RAW file.
BecCowley Sep 26, 2024
162fab2
Move insertion of 'A' and 'R' labels for history codes
BecCowley Sep 26, 2024
be70e93
Improve and debug handling of position errors and surface transient e…
BecCowley Sep 26, 2024
a3208a1
Improve and debug handling of position errors and surface transient e…
BecCowley Sep 26, 2024
2bc6ebf
First attempt at checking the consistency of the flags between HISTOR…
BecCowley Sep 26, 2024
09368be
Move the section that removes duplicated CS values.
BecCowley Sep 26, 2024
f0f7292
Add the expected flag quality value (flag severity equivalent) to the…
BecCowley Sep 27, 2024
40ae592
Change handling of depth dimensioned data variables to one big datafr…
BecCowley Sep 27, 2024
4fa519b
Change handling to dataframe for all DEPTH dimensioned variables and …
BecCowley Sep 27, 2024
9e9facc
Extract institution code information.
BecCowley Sep 27, 2024
ca247e1
Deep copy of dataframe.
BecCowley Sep 27, 2024
2df3962
Handle QC codes in raw file and not in edited.
BecCowley Sep 27, 2024
53d9846
Fix some deprecation bug warnings and other bugs.
BecCowley Sep 27, 2024
83c5e6a
Update global attributes to be consistent with new and old files and …
BecCowley Sep 30, 2024
ad7a559
Debug for case where number of histories is not reflective of actual …
BecCowley Sep 30, 2024
a1f0673
Update filename convention for conversion of Turo files to match IMOS…
BecCowley Sep 30, 2024
159f6ec
Update XBT_input_filename global attribute to be the cruise folder an…
BecCowley Sep 30, 2024
fe3321e
Update Turo conversion to make accept_codes and reject_codes
BecCowley Sep 30, 2024
59442e4
Update accept and reject codes list.
BecCowley Oct 8, 2024
6a4507b
Update handling of global attributes, standardise between the old con…
BecCowley Oct 8, 2024
81c5128
Change writing out to create parquet files and move conversion to net…
BecCowley Oct 8, 2024
74f9e87
Add all probe types to config file and change some formatting
BecCowley Oct 13, 2024
8ebde01
Remove exit on errors and improve formatting.
BecCowley Oct 13, 2024
bd97c71
Assign unknown platform code to missing ship information, convert onl…
BecCowley Oct 14, 2024
bdf580a
Improve and debug TIME and LAT/LON flag handlling.
BecCowley Oct 14, 2024
54ad911
Check for duplicated depths and mismatches in DEPTH and DEPTH_RAW, lo…
BecCowley Oct 14, 2024
368564a
Debug probe type determination, add fuzzy logic to older style text e…
BecCowley Oct 14, 2024
85b6180
Debug move hunk inside if statement.
BecCowley Oct 14, 2024
75685e8
Improve and debug replacement of CS flagged TEMP values and identific…
BecCowley Oct 14, 2024
bf5fef3
Debug ship_name code introduced when IMO added.
BecCowley Oct 14, 2024
a050462
Improve LOGGER handling to assist with grep of xbt.log.
BecCowley Oct 14, 2024
40c829c
Debug variable name error.
BecCowley Oct 14, 2024
b647c66
More streamlining of error code messages.
BecCowley Oct 14, 2024
b2d0d8e
Error fixed in string.
BecCowley Oct 14, 2024
f9b67a9
Error fixed in string.
BecCowley Oct 14, 2024
e27524c
Merge branch 'XBTnewNCFormat' of github.com:BecCowley/data-services i…
BecCowley Oct 14, 2024
8bc2409
Debug assignment of LONG error.
BecCowley Oct 14, 2024
652e255
Add control character removal and streamline handling of control char…
BecCowley Oct 15, 2024
ff4757f
Change lat and long tolerance checking to 6 digits and 0.01 tolerance.
BecCowley Oct 15, 2024
a5acebc
formatting
BecCowley Oct 15, 2024
d2d8c8f
Remove debugging lines.
BecCowley Oct 15, 2024
969cac1
Fix string bug and longitude raw histories if there aren't any.
BecCowley Oct 15, 2024
7b6e195
Debug for string type
BecCowley Oct 15, 2024
b6152fd
Refactor the QC codes to use the csv files rather than config files. …
BecCowley Oct 18, 2024
e0ae9d8
For profiles where the raw and final are not the same length due to a…
BecCowley Oct 18, 2024
37c025b
Refactor data parsing to account for different number of records (eg,…
BecCowley Oct 23, 2024
383f238
Fix bug in duplicate history removal.
BecCowley Oct 23, 2024
c6808cd
Fix institute code bug.
BecCowley Oct 23, 2024
c3a1d9e
Handle situation where the probe type is not an XBT type.
BecCowley Oct 23, 2024
12b7616
Bug fix
BecCowley Oct 23, 2024
68d791e
Another bug fix
BecCowley Oct 23, 2024
bb467c4
Fix array sizing for situations where flags are longer or shorter tha…
BecCowley Oct 27, 2024
772a17a
Fix for bad variable types in history records.
BecCowley Oct 27, 2024
e1702c4
Fix TIME handling of updating TIME_RAW from histories.
BecCowley Oct 27, 2024
9098fe1
Handle PER and TER codes to avoid updating *_RAW values for these
BecCowley Oct 28, 2024
8c08997
Fix bugs in temperature restoration for CSR flags and update reading …
BecCowley Oct 28, 2024
dea1238
Fix histories indexing bug for TIME and be more specific in if statem…
BecCowley Oct 28, 2024
b498deb
Re-do testing for problems between profile data.
BecCowley Oct 28, 2024
c4bc34d
Update error message about missing CS flags only if some good data in…
BecCowley Oct 28, 2024
3b996b2
Handle situation where 99.99 values in TEMP array.
BecCowley Oct 28, 2024
fababb4
More control character bugs.
BecCowley Oct 28, 2024
6959564
Fix handling of time for noqc profile in histories.
BecCowley Oct 29, 2024
5fd7707
Fix errors in TEMP_quality_control flag handling.
BecCowley Oct 29, 2024
07904d1
Fix situation where the flag_severity value is incorrect.
BecCowley Oct 29, 2024
934608f
Fix where a good flag is below a bad one.
BecCowley Oct 29, 2024
677ce79
Account for zero index in history testing.
BecCowley Oct 29, 2024
2e2212a
Improve logic for flag quality checking.
BecCowley Oct 29, 2024
cdc0450
Account for CSR flag used as a single flag to flag all data as bad be…
BecCowley Oct 30, 2024
66c7c74
Change interpolated data where 99.99 is present and adjacent to CSA f…
BecCowley Oct 30, 2024
314e999
Check for missing values in TEMP and TEMP_RAW and rescue from previou…
BecCowley Oct 30, 2024
7ca4c97
Move a hunk to outside a loop in SPA adjacent to CSR flag code.
BecCowley Oct 30, 2024
bb06fcb
Fix error message.
BecCowley Oct 30, 2024
e63d2b9
Fix indexing in histories table to avoid errors in SPA to CSR flag ha…
BecCowley Oct 30, 2024
9690e5a
Refactor the history reading to handle poor formats in the netcdf files.
BecCowley Oct 30, 2024
764472a
Clean up extra SPA, HFA or IPA flags where they overlap with CSR that…
BecCowley Oct 31, 2024
bfa30f6
Fix bugs with data type in histories and removing overlapping CSR and…
BecCowley Oct 31, 2024
f39b9c4
Remove warning message for IOTA codes
BecCowley Oct 31, 2024
a9c41c1
Remove redundant code and change flag removal to remove codes that ar…
BecCowley Oct 31, 2024
063368f
Fix bug where forgot to change the A or R for histories where just on…
BecCowley Oct 31, 2024
17505bc
Remove specific FSR handling as now dealt with in create_flag_feature
BecCowley Oct 31, 2024
0f7a75a
Fix error in flag fixing.
BecCowley Oct 31, 2024
69099f7
Handle the PLA flag surface temperature restoration and remove redund…
BecCowley Oct 31, 2024
38b9f20
Add back duplicate removal of histories where one previous value is 9…
BecCowley Oct 31, 2024
1954a18
Improve testing for LONGITUDE_RAW/LATITUDE_RAW and previous value com…
BecCowley Oct 31, 2024
1436d93
Remove info line from log file.
BecCowley Oct 31, 2024
150f5ba
Handle no history records better.
BecCowley Oct 31, 2024
9655eda
Refine removal of unknown history codes to also refer to the paramete…
BecCowley Oct 31, 2024
9ed15e5
Improve warnings on duplicated history values.
BecCowley Oct 31, 2024
04a1293
Remove redundant column from histories table.
BecCowley Oct 31, 2024
cbd33a6
Fix bug in ship name decoding for l'Astrolabe.
BecCowley Nov 4, 2024
7777801
HISTORY_STOP_DEPTH was not being created. Fixed.
BecCowley Nov 4, 2024
7d0090f
Add 0 quality to REA in flag table.
BecCowley Nov 4, 2024
9b1fcb5
Include BA datatypes.
BecCowley Nov 5, 2024
9e94395
Add flag 0 to accept range in create_flag_feature
BecCowley Nov 5, 2024
4ce5d66
Bug fix to account for no CSR flags.
BecCowley Nov 5, 2024
09f40c4
Change error to warning and account for LOA flag application
BecCowley Nov 5, 2024
b9be52d
Debug to account for extra 3.7m depth where depth in qc file has been…
BecCowley Nov 5, 2024
ce01460
Improve HISTORY_PARAMETER fixing for all cases.
BecCowley Nov 6, 2024
265e715
Reformat, no code change.
BecCowley Nov 6, 2024
c8afaf5
Account for 9999 values in previous value for time and date.
BecCowley Nov 6, 2024
de2dbed
Handle 9999 values in TIME previous value where conversion to datetim…
BecCowley Nov 6, 2024
5138ac0
Update BDA flags, and handle strict assignment of flags where there i…
BecCowley Dec 2, 2024
ffc02d7
Run combine_histories even if none in raw file so all checking is com…
BecCowley Dec 2, 2024
efc6c31
Check for extra 3.7m depth for all profiles, not just ones with one e…
BecCowley Dec 2, 2024
ccf4677
Improve time decoding by inserting zeros where missing in time string…
BecCowley Dec 2, 2024
21b1f01
Alter combine_histories to remove requirement for raw file to have hi…
BecCowley Dec 2, 2024
dcf5213
Handle SetCopyWarning for pandas dataframes.
BecCowley Dec 3, 2024
00e232d
Try again to handle SetCopyWarning.
BecCowley Dec 3, 2024
07faa3d
Fix code assignment bug.
BecCowley Dec 3, 2024
d670ad7
Change TEMP_quality_control value to 9 for any remaining missing TEMP…
BecCowley Dec 4, 2024
fc14470
Handle cases where missing TEMP values might be found in histories wh…
BecCowley Dec 4, 2024
c57124e
Move utilities out of main code and update XBT_line information retri…
BecCowley Dec 11, 2024
437af12
More edits for the XBT_line vocabulary changes.
BecCowley Dec 11, 2024
0d66770
Fix date/time histories coding to avoid copies of slices of dataframe…
BecCowley Dec 11, 2024
8d2a536
Update variable name HISTORY_TEMP_QC_CODE_VALUE to HISTORY_QC_CODE_VALUE
BecCowley Jan 16, 2025
676b817
Add metadata information from configuration file to the parquet table…
BecCowley Jan 16, 2025
02ab0cd
Add parameter information to our standard tables and replace bespoke …
BecCowley Jan 30, 2025
786ffd7
Make a time string handling utility and update time fixes.
BecCowley Jan 30, 2025
8314bb0
Fixes for prof_parm, TEMP values replaced with 99* values and handle …
BecCowley Jan 30, 2025
ae13bfd
HISTORY record improvements with CS codes and duplicate history recor…
BecCowley Jan 30, 2025
4f7a869
Return pattern string that was removed in previous commit.
BecCowley Jan 30, 2025
4754ecf
Move many global attributes into variables.
BecCowley Jan 31, 2025
1665238
Updates to variable names and data types for new variables for XBT fo…
BecCowley Feb 14, 2025
ef0a391
Fix a whole lot of logic and errors due to move of global attributes …
BecCowley Feb 14, 2025
af500fc
Fix variable name bug.
BecCowley Feb 14, 2025
2850c51
Rewrite Turo conversion to match historic conversion with data moved …
BecCowley Feb 18, 2025
a8d8ae3
Add some more XBT variables to the imosParameters.txt file
BecCowley Feb 18, 2025
9e24861
Turo conversion, add some new variables and remove from globals and f…
BecCowley Feb 19, 2025
cf54d1c
Turo conversion code, fix some bugs and add variables.
BecCowley Feb 19, 2025
e63cee1
Add code to export netcdf format information to a csv file.
BecCowley Feb 19, 2025
cf88b0e
Update Turo conversion comments and some import calls.
BecCowley Feb 19, 2025
0eb036d
Fix some bugs and add missing variables to match both Turo and Histor…
BecCowley Feb 20, 2025
8b1988d
Add a tidy-up of redundant columns and print statement to historical …
BecCowley Feb 20, 2025
835b335
Update vocab handling to use a local copy of the vocabularies in the …
BecCowley Mar 6, 2025
bf2733d
Add local check of XBT_line_vocabulary if URL is inaccessible
BecCowley Mar 6, 2025
afa6ea8
Change fill value for XBT-accept and reject codes to 0 from 99
BecCowley Mar 27, 2025
071f297
Add text versions of online vocabularies for instance where code cann…
BecCowley Mar 27, 2025
94f0458
Update error message and bug fix.
BecCowley Apr 28, 2025
6fe3fa1
Fix some bugs identified by AW and LK.
BecCowley Apr 30, 2025
43e5a15
Add FV00 file creation and recorder serial number to the uniqueid.
BecCowley May 2, 2025
6075305
Update Turo converter to just include the latest codes, not all histo…
BecCowley May 9, 2025
b7b0ee8
Handle time conversion better and account for dates in other time var…
BecCowley Jun 6, 2025
e4fe274
Fix errors to make files netcdf compliant for version 1.8 (using onli…
BecCowley Jun 6, 2025
1e90ce9
Remove a file and add a TODO
BecCowley Jun 6, 2025
0356362
Add automatic CSR flag addition on conversion
BecCowley Jun 6, 2025
1e83907
Merge pull request #10 from BecCowley/addCSA
BecCowley Jun 6, 2025
52f33cf
Add wire break test to conversion. Also fix some typos
BecCowley Jun 12, 2025
c8a019b
Merge pull request #11 from BecCowley/addWBR
BecCowley Jun 12, 2025
eace8a2
Tidy variable names to match the ones in the XBT-community format and…
BecCowley Jun 17, 2025
7ac7a08
Update code so it uses the netcdfVars.csv file.
BecCowley Jun 19, 2025
354d89b
Make uncertainties creation, probe type information and writing netcd…
BecCowley Jun 19, 2025
667956e
Fix issues to make CF compliant. Add cf_role column to netcdfVars.csv…
BecCowley Jun 19, 2025
8cf3993
Merge pull request #12 from BecCowley/fix_varnames
BecCowley Jun 20, 2025
0fc7a31
Change data type of some history records to match requirements in nc …
BecCowley Jun 20, 2025
bf9af7b
Debug handling of HISTORY_PREVIOUS_VALUE data type.
BecCowley Jun 20, 2025
819399a
Fix issues with HISTORY_PREVIOUS_VALUE handling as it contains differ…
BecCowley Jun 23, 2025
60b6fb9
Merge pull request #13 from BecCowley/fix_history_prevValue
BecCowley Jun 24, 2025
ee9e78c
Merge branch 'master' of github.com:BecCowley/data-services into XBTn…
BecCowley Jun 24, 2025
90764da
Update reading of surface codes and some other variables to use the n…
BecCowley Jun 24, 2025
06bf256
Merge pull request #14 from BecCowley/update_var_handling
BecCowley Jun 24, 2025
5f8e53a
Update handling of edited and raw profiles with different lengths, eg…
BecCowley Jun 25, 2025
7a79829
Merge pull request #15 from BecCowley/fix_depthhandling
BecCowley Jun 25, 2025
1a5cf62
Fix bug with launcher assigment
BecCowley Jun 25, 2025
4cecc85
Update float types for consistency.
BecCowley Jun 25, 2025
7ac7809
Remove station limiting tester
BecCowley Jun 25, 2025
4e70cde
Adjust probe type assignment to account for test probe.
BecCowley Jun 25, 2025
3d57b12
Remove redundant import.
BecCowley Jun 25, 2025
537175b
Fix nan handling when outputting to netcdf files so fill values inser…
BecCowley Jun 30, 2025
39c4616
Fix output folder handling
BecCowley Jun 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
429 changes: 429 additions & 0 deletions SOOP/SOOP_XBT/DELAYED/TuroXBT2IMOSnc.py

Large diffs are not rendered by default.

1,700,898 changes: 1,700,898 additions & 0 deletions SOOP/SOOP_XBT/DELAYED/check_parquet.ipynb

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions SOOP/SOOP_XBT/DELAYED/create_xbtfft.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# create a text file with the XBT_fault_and_feature_type binary values and their corresponding labels
import pandas as pd
from decode_flags import convert


def create_xbtfft():
# Read the CSV file and convert it to a DataFrame
file_path = 'flag_quality_table.csv'
df = pd.read_csv(file_path)
# drop the rows with 'selected' in the rule_direction column
df = df[df['rule_direction'] != 'selected']
# keep some of the columns
df = df[['name', 'full_code', 'XBT_accept_code', 'XBT_reject_code', 'TEMP_quality_control','depth', 'Parameter']]
# replace the NaN values with 0 in depth column
df['depth'] = df['depth'].fillna(1)
# convert the depth and rule_direction columns to match categories in the xbt_config file
df['depth'] = df['depth'].map({0: 'ACT_CODES_FULL_PROFILE', 1: 'ACT_CODES_TO_NEXT_FLAG', 3.6: 'ACT_CODES_SINGLE_POINT'})
# drop the rows with NaN values in the XBT_accept_code column
dfa = df.dropna(subset=['XBT_accept_code'])
# drop the rows with NaN values in the XBT_reject_code column
dfr = df.dropna(subset=['XBT_reject_code'])

# convert the name and full_code
labels = dfa['name'].tolist()
full_code = dfa['full_code'].tolist()
tempqc = dfa['TEMP_quality_control'].tolist()
depth = dfa['depth'].tolist()
parameter = dfa['Parameter'].tolist()
with open('xbt_accept_code.csv', 'w') as f:
# write the header
f.write('label,code,byte_value,tempqc,group_label,parameter\n')
for i in range(len(dfa)):
# write the labels, code and the binary value with a comma in between each
f.write(f'{convert(2**i, labels)},{full_code[i]},{2**i},{tempqc[i]},{depth[i]},{parameter[i]}\n')
labels = dfr['name'].tolist()
full_code = dfr['full_code'].tolist()
tempqc = dfr['TEMP_quality_control'].tolist()
depth = dfr['depth'].tolist()
parameter = dfr['Parameter'].tolist()
with open('xbt_reject_code.csv', 'w') as f:
# write the header
f.write('label,code,byte_value,tempqc,group_label,parameter\n')
for i in range(len(dfr)):
# write the labels, code and the binary value with a comma in between each
f.write(f'{convert(2**i, labels)},{full_code[i]},{2**i},{tempqc[i]},{depth[i]},{parameter[i]}\n')
return dfa, dfr


dfa, dfr = create_xbtfft()
22 changes: 22 additions & 0 deletions SOOP/SOOP_XBT/DELAYED/decode_flags.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
""" Use this snippet of code to decode binary flag codes into meaningful values
Have to pass in flag labels or names as type list of strings.
And i has to be an unsigned integer
"""


def convert(i, labels):
output = ''
pos = 0
while i:
if i & 1:
output += labels[pos]
pos += 1
i >>= 1
return output


# test it:
val = 8388641
labels = ["scientific_qc_applied","wire_stretch","leakage","electrical_interference_interpolated","high_frequency_noise_filtered","repeat_profile","temperature_inversion_confirmed","temperature_inversion_unconfirmed","temperature_offset","temperature_eddy_or_front","temperature_steps_or_structure_confirmed","temperature_steps_or_structure_unconfirmed","depth_offset","constant_temperature","time_error_corrected","latitude_error_corrected","longitude_error_corrected","probe_type_changed","spike_interpolated","fine_structure","insulation_penetration_interpolated","nub_inversion","hit_bottom","premature_launch","surface_temperature_anomaly","surface_offset","temperature_anomaly","temperature_difference_at_depth","unique_id_changed","bowing_BathySystems","cusping_BathySystems_leakage","sippicanMK_timing_delay_driver_error","BathySystem_software_fault_modulo_10_spikes_filtered","protecho_systems_leakage_fault","sippicanMK9_sticking_bit_19_point_filtered","depth_corrected_multiplied_by_10","depth_corrected_update_fall_rate_equation","bathy_data_low_resolution"]
print(convert(val, labels))

69 changes: 69 additions & 0 deletions SOOP/SOOP_XBT/DELAYED/flag_quality_table.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name,historic_extra_code,old_code,code,full_code,depth,condition,rule_direction,QC_reject_code,QC_accept_code,TIME_quality_control,LATITUDE_quality_control,LONGITUDE_quality_control,PROBE_TYPE_quality control,TEMP_quality_control,DEPTH_quality_control,Parameter
scientific_qc_applied,0,,QC,QCA,0,,down,,1,1,1,1,1,1,1,TEMP
wire_stretch,0,,WS,WSA,,,down,,2,,,,,2,1,TEMP
leakage,0,,LE,LEA,,,down,,4,,,,,2,1,TEMP
electrical_interference_interpolated,0,"SP, IP",EI,EIA,,,selected,,8,,,,,5,1,TEMP
electrical_interference_interpolated,0,,EI,EIA,,,down,,8,,,,,2,1,TEMP
high_frequency_noise_filtered,0,,HF,HFA,,,selected,,16,,,,,5,1,TEMP
high_frequency_noise_filtered,0,,HF,HFA,,,down,,16,,,,,2,1,TEMP
repeat_profile,0,,RE,REA,0,,down,,32,,,,,0,0,TEMP
temperature_inversion_confirmed,0,,IV,IVA,,,down,,64,,,,,1,1,TEMP
temperature_inversion_unconfirmed,0,,PI,PIA,,,down,,128,,,,,2,1,TEMP
temperature_offset,0,"TO, TD, SA, SO",TO,TOA,,,down,,256,,,,,2,1,TEMP
temperature_eddy_or_front,0,,EF,EFA,0,,down,,512,,,,,1,1,TEMP
temperature_steps_or_structure_confirmed,0,FS,ST,STA,0,,down,,1024,,,,,1,1,TEMP
temperature_steps_or_structure_unconfirmed,0,,PS,PSA,0,,down,,2048,,,,,2,1,TEMP
depth_offset,0,"DO, PL",DO,DOA,0,,down,,4096,,,,,5,1,TEMP
constant_temperature,0,,CT,CTA,,,down,,8192,,,,,2,1,TEMP
time_error_corrected,0,,TE,TEA,0,,down,,16384,5,,,,2,1,TIME
latitude_error_corrected,0,PE,LA,LAA,0,,down,,32768,,5,,,2,1,LATITUDE
longitude_error_corrected,0,PE,LO,LOA,0,,down,,65536,,,5,,2,1,LONGITUDE
probe_type_changed,0,,PR,PRA,0,,down,,131072,,,,5,5,5,TEMP
spike_interpolated,1,,SP,SPA,,,selected,,262144,,,,,5,1,TEMP
spike_interpolated,1,,SP,SPA,,,down,,262144,,,,,2,1,TEMP
fine_structure,1,,FS,FSA,0,,down,,524288,,,,,2,1,TEMP
insulation_penetration_interpolated,1,,IP,IPA,,,selected,,1048576,,,,,5,1,TEMP
insulation_penetration_interpolated,1,,IP,IPA,,,down,,1048576,,,,,2,1,TEMP
nub_inversion,1,,NU,NUA,0,,down,,2097152,,,,,1,1,TEMP
hit_bottom,1,,HB,HBA,,,down,,4194304,,,,,2,1,TEMP
premature_launch,1,,PL,PLA,0,,down,,8388608,,,,,2,1,TEMP
surface_temperature_anomaly,1,,SA,SAA,0,,down,,16777216,,,,,2,1,TEMP
surface_offset,1,,SO,SOA,0,,down,,33554432,,,,,2,1,TEMP
temperature_anomaly,1,,TA,TAA,,,down,,67108864,,,,,2,1,TEMP
temperature_difference_at_depth,1,,TD,TDA,,,down,,134217728,,,,,2,1,TEMP
unique_id_changed,1,,ID,IDA,0,,down,,268435456,,,,,1,1,TEMP
bowing_BathySystems,1,,BO,BOA,0,,down,,536870912,,,,,2,1,TEMP
cusping_BathySystems_leakage,1,,CU,CUA,,,down,,1073741824,,,,,2,1,TEMP
sippicanMK_timing_delay_driver_error,1,,DR,DRA,0,,down,,2147483648,,,,,2,1,TEMP
BathySystem_software_fault_modulo_10_spikes_filtered,1,,MO,MOA,,,down,,4294967296,,,,,2,1,TEMP
protecho_systems_leakage_fault,1,,PF,PFA,0,,down,,8589934592,,,,,2,1,TEMP
sippicanMK9_sticking_bit_19_point_filtered,1,,SB,SBA,,,down,,17179869184,,,,,2,1,TEMP
depth_corrected_multiplied_by_10,1,,DE,DEA,0,,down,,34359738368,,,,,1,1,TEMP
depth_corrected_update_fall_rate_equation,1,,DP,DPA,0,,down,,68719476736,,,,,0,0,TEMP
bathy_data_low_resolution,1,,BD,BDA,0,,down,,1.37E+11,,,,,2,1,TEMP
surface_transient,0,,CS,CSR,3.6,,up,1,,,,,,3,1,TEMP
wire_break,0,,WB,WBR,,,down,2,,,,,,4,1,TEMP
wire_stretch,0,,WS,WSR,,,down,4,,,,,,3,1,TEMP
hit_bottom,0,,HB,HBR,,,down,8,,,,,,3,3,TEMP
leakage,0,,LE,LER,,,down,16,,,,,,3,1,TEMP
electrical_interference,0,"SP, IP",EI,EIR,,,down,32,,,,,,4,1,TEMP
high_frequency_noise,0,,HF,HFR,,,down,64,,,,,,4,1,TEMP
no_good,0,,NG,NGR,,,down,128,,,,,,4,1,TEMP
temperature_offset,0,"TO, TD, SA, SO",TO,TOR,,,down,256,,,,,,3,1,TEMP
depth_offset,0,"DO, PL",DO,DOR,0,,down,512,,,,,,3,3,TEMP
constant_temperature,0,,CT,CTR,,,down,1024,,,,,,3,3,TEMP
time_error,0,,TE,TER,0,,down,2048,,3,,,,3,1,TIME
position_error,0,,PE,PER,0,,down,4096,,,3,3,,3,1,LATITUDE LONGITUDE
duplicate_profile,0,,DU,DUR,0,,down,8192,,,,,,4,1,TEMP
test_probe,0,,TP,TPR,0,,down,16384,,,4,4,,4,4,TEMP
probe_type,0,,PR,PRR,0,,down,32768,,,,,3,3,3,TEMP
spike,1,,SP,SPR,,,down,65536,,,,,,4,1,TEMP
insulation_penetration,1,,IP,IPR,,,down,131072,,,,,,3,1,TEMP
surface_temperature_anomaly,1,,SA,SAR,0,,down,262144,,,,,,3,1,TEMP
surface_offset,1,,SO,SOR,0,,down,524288,,,,,,3,1,TEMP
no_trace,1,,NT,NTR,0,,down,1048576,,,,,,4,1,TEMP
bowing_BathySystems,1,,BO,BOR,0,,down,2097152,,,,,,3,1,TEMP
sippicanMK_timing_delay_driver_error,1,,DR,DRR,,,down,4194304,,,,,,3,1,TEMP
protecho_systems_leakage_fault,1,,PF,PFR,0,,down,8388608,,,,,,3,1,TEMP
contact_lost_with_probe,1,,CL,CLR,,,down,16777216,,,,,,3,1,TEMP
bathy_data_low_resolution,1,,BD,BDR,0,,down,33554432,,,,,,3,1,TEMP
Loading
Loading