CheckWaarnemingenAanwezig and OphalenAantalWaarnemingen are not always consistent with OphalenWaarnemingen

[veenstrajelmer]

This is a issue related to Deltares/ddlpy#46, but without ddlpy dependency in the example code.

Sometimes the ddl CheckWaarnemingenAanwezig returns false for a query where there is extreme data. We get available false and num meas 28 prints from the code below:

import datetime as dt
import pytz
import requests
import pandas as pd

# general
start_date = dt.datetime(1980,1,1)
end_date = dt.datetime(1980,1,5)
start_date_str = pytz.UTC.localize(start_date).isoformat(timespec='milliseconds')
end_date_str = pytz.UTC.localize(end_date).isoformat(timespec='milliseconds')

aquometadata_dict = {'Eenheid': {'Code': 'cm'},
                     'Grootheid': {'Code': 'WATHTE'},
                     'Hoedanigheid': {'Code': 'NAP'}}
locatie_dict = {'X': 576917.669784491, 
                'Y': 5759136.15818497,
                'Code': 'HOEKVHLD'}

#meas available
request = {
    "AquoMetadataLijst": [aquometadata_dict],
    "LocatieLijst": [locatie_dict],
    "Periode": {
        "Begindatumtijd": start_date_str,
        "Einddatumtijd": end_date_str
    }
}

url_aanwezig = "https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/CheckWaarnemingenAanwezig"
resp = requests.post(url_aanwezig, json=request, timeout=5)
result = resp.json()
if not result['Succesvol']:
    raise Exception(result.get('Foutmelding', 'No error returned'))

print("available", result['WaarnemingenAanwezig'])


# meas slice
request = {
    "AquoPlusWaarnemingMetadata": {
        "AquoMetadata": aquometadata_dict
        },
    "Locatie": locatie_dict,
    "Periode": {"Begindatumtijd": start_date_str, 
                "Einddatumtijd": end_date_str},
}

url_ophalen = "https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/OphalenWaarnemingen"
resp = requests.post(url_ophalen, json=request)
result = resp.json()
if not result["Succesvol"]:
    raise Exception(result.get("Foutmelding", "No error returned"))

rows = []
rows_meta = []
for waarneming in result["WaarnemingenLijst"]:
    for row in waarneming["MetingenLijst"]:
        rows.append(row)
    rows_meta.append(waarneming["AquoMetadata"])
# normalize and return
df = pd.json_normalize(rows)
df_meta = pd.json_normalize(rows_meta)

print("num meas", len(df))

This prints:

available false
num meas 28

This also happens the other way round, for VERDTLPNOT (WATHTE/MSL/NVT). There is data available according to OphalenAantalWaarnemingen, also according to ddlpy.measurements_available() but no data is retrieved:

import requests
import pandas as pd
import pytz

aquometadata_dict = {'Eenheid': {'Code': 'cm'},
                     'Grootheid': {'Code': 'WATHTE'},
                     'Hoedanigheid': {'Code': 'MSL'},
                     'Groepering': {'Code': 'NVT'}
                     }

locatie_dict = {'X': 569986.264615189, 
                'Y': 5768521.49708648,
                'Code': 'VERDTLPNOT'}


start_date = pd.Timestamp("2000-03-01")
end_date = pd.Timestamp("2000-03-02")
start_date_str = pytz.UTC.localize(start_date).isoformat(timespec="milliseconds")
end_date_str = pytz.UTC.localize(end_date).isoformat(timespec="milliseconds")

# get number of waarnemingen
url_ddl = 'https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/OphalenAantalWaarnemingen'
request_ddl = {
    "AquoMetadataLijst": [aquometadata_dict],
    "LocatieLijst": [locatie_dict],
    "Groeperingsperiode" : "Dag",
    "Periode": {
        "Begindatumtijd": start_date_str,
        "Einddatumtijd": end_date_str,
    }
}


resp = requests.post(url_ddl, json=request_ddl)
if not resp.ok:
    raise Exception('%s for %s: %s'%(resp.reason, resp.url, str(resp.text)))
result = resp.json()
if not result['Succesvol']:
    raise Exception('query not succesful, Foutmelding: %s from %s'%(result['Foutmelding'],url_ddl))
for one in result['AantalWaarnemingenPerPeriodeLijst']:
    # print(one['AquoMetadata']['Grootheid'])
    # print(one['AquoMetadata']['Hoedanigheid'])
    # print(one['AquoMetadata']['Groepering'])
    df = pd.json_normalize(one['AantalMetingenPerPeriodeLijst'])
    df2 = df[["AantalMetingen"]].copy()
    df2["datetime"] = (df["Groeperingsperiode.Jaarnummer"].apply(lambda x: f"{x:04d}") +
                      "-" + 
                      df["Groeperingsperiode.Maandnummer"].apply(lambda x: f"{x:02d}") +
                      "-" +
                      df["Groeperingsperiode.Dag"].apply(lambda x: f"{x:02d}"))
    print(df2)

# meas slice
request = {
    "AquoPlusWaarnemingMetadata": {
        "AquoMetadata": aquometadata_dict
        },
    "Locatie": locatie_dict,
    "Periode": {"Begindatumtijd": start_date_str, 
                "Einddatumtijd": end_date_str},
}

url_ophalen = "https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/OphalenWaarnemingen"
resp = requests.post(url_ophalen, json=request)
result = resp.json()
if result["Succesvol"]:
    rows = []
    for waarneming in result["WaarnemingenLijst"]:
        for row in waarneming["MetingenLijst"]:
            rows.append(row)
    # normalize and return
    meas = pd.json_normalize(rows)
else:
    print(result.get("Foutmelding", "No error returned"))
    meas = pd.DataFrame()
print("num meas retrieved:", len(meas))

This prints:

   AantalMetingen    datetime
0             138  2000-03-01
1               6  2000-03-02
Geen gegevens gevonden!
num meas retrieved: 0

[TvLoon-RWS]

Check for underlying issue, and check if the same happens in Wadar-WaterWebservices