DDL improvements (data and waterwebservices)

[veenstrajelmer]

General

• rws docs: https://rijkswaterstaatdata.nl/waterdata and https://rijkswaterstaat.github.io/wm-ws-dl/?python#introduction
• There is also https://pypi.org/project/rws-waterinfo/#data (and sourcecode on gitlab), created in 2023 but has no useful features (yet?) compared to ddlpy.
• what is the meaning of KwaliteitswaardecodeLijst values, documented as Kwaliteitsoordeel_code in https://rijkswaterstaatdata.nl/publish/pages/223004/uitleg-attributen-bij-bulkwaarnemingenservice-dd-06-11-2023-.pdf
• future waterwebservices: https://rijkswaterstaatdata.nl/projecten/beta-waterwebservices
• register all wm-ws-dl issues at servicedesk data?
• https://community.developer.overheid.nl/t/dl-api-waterdata-rijkswaterstaat/270

ddlpy improvements

• catch switched start/end dates Deltares/ddlpy#44
• Sum up duplicated periods in ddlpy.measurements_amount() Deltares/ddlpy#92
• Empty response instead of error if request is too large Deltares/ddlpy#96
• Add support for different timezones Deltares/ddlpy#40
• Add yearly frequency to ddlpy.measurements() Deltares/ddlpy#94
• consider simplifying try/except for Tijdstip Keyerror Deltares/ddlpy#102
• remove Parameter_Wat_Omschrijving in _combine_waarnemingenlijst Deltares/ddlpy#103
• consider dropping Hoedanigheid from catalog Deltares/ddlpy#89

DDL/waterwebservices api improvements

• allow distinction between historic and realtime stations Rijkswaterstaat/wm-ws-dl#20
• prevent duplicated ddl station Code Rijkswaterstaat/wm-ws-dl#12
• inconsistency between requested/returned time extent in case of summertime Rijkswaterstaat/wm-ws-dl#9
• support for requesting with unique Locatie_MessageID instead of 9-decimal X/Y + Code. >> not urgent since ddlpy handles it automatically. Would still be convenient though, since Locatie_MessageID is always unique. This also simplifies getting hist or realtime data once you know which station is which. Or is Locatie_MessageID already used in the station filtering? >> new Wadar WaterWebservices have unique location codes and do not require passing X/Y in the request anymore.
• Flatten StatuswaardeLijst/KwaliteitswaardecodeLijst (and rename to Statuswaarde/Kwaliteitswaardecode) in ddl. This also goes for other keys under WaarnemingMetadata like Bemonsteringshoogte/Referentievlak/OpdrachtgevendeInstantie. Additionally also always return kwaliteitswaarde as int instead of str? >> not urgent since nested lists are flattened in ddlpy >> new Wadar Waterwebservices has no lists anymore for these columns
• split BioTaxonType in Code and Omschrijving like all others >> resolved in new Wadar Waterwebservices
• minimize query/disk size of WaardeBepalingsmethode.Code (e.g. "other:F009") by dropping "other:" from codes >> or do not all start with ":other"? >> Will be stored different in wadar-waterwebservices, so not relevant
• nodata returned for a query where there is extreme data Deltares/ddlpy#46 >> CheckWaarnemingenAanwezig and OphalenAantalWaarnemingen are not always consistent with OphalenWaarnemingen Rijkswaterstaat/wm-ws-dl#11
• difficult to get the latest time since multiple or no results are returned per location Rijkswaterstaat/wm-ws-dl#15
• simplify distinction between HW and LW extremes Rijkswaterstaat/wm-ws-dl#19
• align NVT in Omschrijving columns Rijkswaterstaat/wm-ws-dl#36
• sort returned data on time >> not urgent since data is sorted in ddlpy
• drop Meetwaarde.Waarde_Alfanumeriek if Meetwaarde.Waarde_Numeriek column is present, contains duplicate information. Would be cleaner if ddl would not return this duplicate column. Or consider simplify distinction between HW and LW extremes Rijkswaterstaat/wm-ws-dl#19
• merge metadata in WaarnemingMetadata (e.g. StatuswaardeLijst or KwaliteitswaardecodeLijst) with rest (e.g. Hoedanigheid or Groepering), and also split in Code/Omschrijving
• split ProcesType in Code and omschrijving (was added in new Wadar WaterWebservices)
• minimize query/disk size of WaarnemingMetadata.StatuswaardeLijst by replacing Gecontroleerd by G, Ongecontroleerd by O, Definitief by D. Or better: use Code and Omschrijving separately
• minimize query/disk space by excluding columns with duplicated information (like textual description of grootheid/eenheid/hoedanigheid/etc combination in Parameter_Wat_Omschrijving column)
• also add attrs to kwaliteitswaardecode (e.g. in netcdf created by create to_xarray function Deltares/ddlpy#85), although this metadata is currently only available in a pdf >> add in ddl instead
• accept larger requests (and prevent timeout), so we can safely retrieve data per year instead of per month. This will increase the retrieval process significantly
• "Omschrijving" column was added in new Wadar WaterWebservices. It contains almost the same information as "Naam" (station name), so it is either duplicate or confusing information. Column can be removed?
• OphalenCatalogus does not give error when passing incorrect CatalogusFilter entries Rijkswaterstaat/wm-ws-dl#44

DDL data improvements (donar or ddl)

• incorrect metadata for IJMDBTHVN Rijkswaterstaat/wm-ws-dl#21
• Retrieve salinity measurements on multiple heights Rijkswaterstaat/wm-ws-dl#31
• hoedanigheid (vertical reference level) is defined in multiple metadata fields Rijkswaterstaat/wm-ws-dl#42
• invalid Y-coordinate for location KOMo Rijkswaterstaat/wm-ws-dl#13
• prevent None in KwaliteitswaardecodeLijst Rijkswaterstaat/wm-ws-dl#14
• unexpected large data gaps Rijkswaterstaat/wm-ws-dl#16
• data is distributed over NAP/MSL hoedanigheid for some stations Rijkswaterstaat/wm-ws-dl#17
• unmasked outliers present in data for some stations Rijkswaterstaat/wm-ws-dl#18
• Duplicated data with different statusses for WALSODN Rijkswaterstaat/wm-ws-dl#22
• Clean up duplicates for EURPFM extremes Rijkswaterstaat/wm-ws-dl#23
• Duplicates for NIEWSTZL extremes (different MeetApparaat) Rijkswaterstaat/wm-ws-dl#24
• unflagged outliers for HUIBGT Rijkswaterstaat/wm-ws-dl#25
• TEXNZE has large gap in 2007 data Rijkswaterstaat/wm-ws-dl#26
• invalid 170cm values for HUIBGT Rijkswaterstaat/wm-ws-dl#27
• Clean duplicates with nan/99 values in HARVT10 Rijkswaterstaat/wm-ws-dl#28
• no ext data available for STELLDBTN after 1996 Rijkswaterstaat/wm-ws-dl#29
• negative outliers for UITHZWD1 Rijkswaterstaat/wm-ws-dl#30
• limited data extent and frequency available for HARTKWT and SCHEURHVN Rijkswaterstaat/wm-ws-dl#32
• comparison to model timeseries shows unexpected vertical shifts in measurements Rijkswaterstaat/wm-ws-dl#33 (add additional stations and updated slides after re-checking with new ddl data)
• Extremely low yearly mean for STELLDBTN 2013 Rijkswaterstaat/wm-ws-dl#34
• Duplicate times for BAALHK Rijkswaterstaat/wm-ws-dl#35
• Align RIKZ_WAT and RIKZMON_WAT Rijkswaterstaat/wm-ws-dl#37 >> closed as completed, but single source of truth is desireable
• waterlevel extremes missing for some stations/years Rijkswaterstaat/wm-ws-dl#38
• Missing/duplicated values in WATHTE timeseries and extremes Rijkswaterstaat/wm-ws-dl#39
• Since drop_duplicates was added to kenmerkendewaarden.read_measurements(), one can get logging for the amount of time-value-duplicates and time-duplicates in each dataset. Before adding this to Missing/duplicated values in WATHTE timeseries and extremes Rijkswaterstaat/wm-ws-dl#39, first re-retrieve all (corrected) data (after 14-8-2024, although this does not include cleaner extremes yet)
• phase difference VLISSGN due middentijd/eindtijd Rijkswaterstaat/wm-ws-dl#41
• Incorrect extreme values for BERGSDSWT and others Rijkswaterstaat/wm-ws-dl#43

Additional data validation

• Use DCSM data QA (tidal analysis and model-measurement differences) to provide additional feedback on data
• for cmems/kwk/getijverandering: how to get entire timeseries of approved waterlevel? Do we check status or kwaliteitscode? And are all statuses/kwaliteitscodes up to date? Maybe create overview of status/qc over time per station.
• maybe compare ddl available data to waterbase available data

ddlpy example for HOEKVHLD
Inspired by example notebook:

import ddlpy
# import logging
# logging.basicConfig()
# ddlpy.ddlpy.logger.setLevel(logging.DEBUG)

locations = ddlpy.locations()
bool_hoedanigheid = locations['Hoedanigheid.Code'].isin(['NAP'])
bool_stations = locations.index.isin(['HOEKVHLD', 'IJMDBTHVN','SCHEVNGN'])
bool_grootheid = locations['Grootheid.Code'].isin(['WATHTE'])
bool_groepering = locations['Groepering.Code'].isin(['NVT'])
selected = locations.loc[bool_grootheid & bool_hoedanigheid & bool_groepering & bool_stations]

start_date = "2019-01-01"
end_date = "2020-02-01"
# pass a single row of the locations dataframe to the measurements function to get the measurements for that location
measurements = ddlpy.measurements(selected.iloc[0], start_date, end_date)

measurements.plot(y='Meetwaarde.Waarde_Numeriek', linewidth=0.5, figsize=(13, 8))