update nb output for Thredds v5 #77
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb::Cell 0 _ Notebook cell execution failed Cell 0: Cell outputs differ Input: from siphon.catalog import TDSCatalog url = "https://boreas.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/cb-oura-1.0/catalog.xml" # TEST_USE_PROD_DATA # Create Catalog cat = TDSCatalog(url) # List of datasets print(f"Number of datasets: {len(cat.datasets)}") # Access mechanisms - here we are interested in OPENDAP, a data streaming protocol cds = cat.datasets[0] print(f"Access URLs: {tuple(cds.access_urls.keys())}") Traceback: mismatch 'stdout' assert reference_output == test_output failed: "Number of da...cdfSubset')\n" == "Number of da...cdfSubset')\n" Skipping 43 identical leading characters in diff, use -v to show Skipping 50 identical trailing characters in diff, use -v to show - erver', 'OpenDAP', 'NC ? ^^^ + erver', 'OPENDAP', 'NC ? ^^^ _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb::Cell 2 _ Notebook cell execution failed Cell 2: Cell outputs differ Input: # Extract a subset of the file # Again, this only creates an in-memory representation of the data sub = ds.pr.sel(time="2050").sel(lon=-80, lat=46, method="nearest") # The data is only downloaded when we actuall need it for a computation. sub.mean(keep_attrs=True).compute() Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data...[256 16 16]' == '<xarray.Data...[256 16 16]' Skipping 238 identical trailing characters in diff, use -v to show <xarray.DataArray 'pr' ()> - array(4.372211e-05, dty + array(3.975487e-05, dty _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb::Cell 3 _ Notebook cell execution failed Cell 3: Cell outputs differ Input: rcp45_data = [cat.datasets[x] for x in cat.datasets if "rcp45" in x] rcp45_data Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '[day_NorESM1...50-2100.ncml]' == '[day_ACCESS1...50-2100.ncml]' Skipping 66 identical trailing characters in diff, use -v to show - [day_ACCESS1-3_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, ? ^^^ ^ ^ + [day_NorESM1-M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, ? ^^^ ^ ^ + day_MPI-ESM-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_IPSL-CM5B-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_IPSL-CM5A-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_INM-CM4_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_HadGEM2-CC_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_GFDL-ESM2M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_CanESM2_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_CMCC-CMS_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, day_BNU-ESM_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_CMCC-CMS_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_CanESM2_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_GFDL-ESM2M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_HadGEM2-CC_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_INM-CM4_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_IPSL-CM5A-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_IPSL-CM5B-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_MPI-ESM-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_NorESM1-M_historic ? ^^^ ^ ^ + day_ACCESS1-3_historic ? ^^^ ^ ^ _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb::Cell 5 _ Notebook cell execution failed Cell 5: Cell outputs differ Input: datasets = {} for r in ["rcp45", "rcp85"]: expected = 11 datasets[r] = get_ncfilelist(r, expected, url) # Display search results display(datasets["rcp45"]) display(datasets["rcp85"]) Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '[day_NorESM1...50-2100.ncml]' == '[day_ACCESS1...50-2100.ncml]' Skipping 66 identical trailing characters in diff, use -v to show - [day_ACCESS1-3_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, ? ^^^ ^ ^ + [day_NorESM1-M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, ? ^^^ ^ ^ + day_MPI-ESM-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_IPSL-CM5B-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_IPSL-CM5A-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_INM-CM4_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_HadGEM2-CC_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_GFDL-ESM2M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_CanESM2_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, + day_CMCC-CMS_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, day_BNU-ESM_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_CMCC-CMS_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_CanESM2_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_GFDL-ESM2M_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_HadGEM2-CC_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_INM-CM4_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_IPSL-CM5A-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_IPSL-CM5B-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_MPI-ESM-LR_historical+rcp45_r1i1p1_na10kgrid_qm-moving-50bins-detrend_1950-2100.ncml, - day_NorESM1-M_historic ? ^^^ ^ ^ + day_ACCESS1-3_historic ? ^^^ ^ ^ _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 0 _ Notebook cell execution failed Cell 0: Cell outputs differ Input: import warnings import xarray as xr from IPython.display import display # Fancy representation of xarray objects from siphon.catalog import TDSCatalog warnings.simplefilter("ignore") url = "https://boreas.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/cb-oura-1.0/catalog.xml" # TEST_USE_PROD_DATA # Create Catalog cat = TDSCatalog(url) # DAP link for this demo ds_url = cat.datasets[0].access_urls["OPENDAP"] # xarray.Dataset ds = xr.open_dataset(ds_url, chunks=dict(time=256 * 2, lon=32, lat=32)) display(ds) a = ds.tasmin.isel(time=0).plot(figsize=(10, 4)) Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data...id: NCC' == '<xarray.Data... ACCESS1-3' Skipping 50 identical leading characters in diff, use -v to show - time: 55152) ? - + time: 55115) ? + Coordinates: * lat (lat) float32 66.62 66.54 66.46 66.37 ... 40.29 40.21 40.12 40.04 * lon (lon) float32 -120.8 -120.7 -120.6 -120.5 ... -54.63 -54.55 -54.46 - * time (time) datetime64[ns]/DATE/1950-01-02 ... 2100-12-31 + * time (time) object/DATE/TIME/.../DATE/00:00:00 Data variables: tasmin (time, lat, lon) float32 dask.array<chunksize=(512, 32, 32), meta=np.ndarray> tasmax (time, lat, lon) float32 dask.array<chunksize=(512, 32, 32), meta=np.ndarray> pr (time, lat, lon) float32 dask.array<chunksize=(512, 32, 32), meta=np.ndarray> - Attributes: (12/27) ? ^ + Attributes: (12/26) ? ^ Conventions: CF-1.5 title: Ouranos standard ensemble of bias-adjusted cl... - history: CMIP5 compliant file produced from raw ACCESS... + history: DATE_TIME_TZ CMOR rewrote data to com... institution: Ouranos Consortium on Regional Climatology an... - source: ACCESS1-3 2011. Atmosphere: AGCM v1.0 (N96 gr... - driving_model: ACCESS1-3 + source: NorESM1-M 2011 atmosphere: CAM-Oslo (CAM4-Os... + driving_experiment: historical,rcp85 ... ... + frequency: day modeling_realm: atmos target_dataset: CANADA : ANUSPLIN interpolated Canada daily 3... target_dataset_references: CANADA : https://doi.org/10.1175/2011BAMS3132... - driving_institution: Commonwealth Scientific and Industrial Resear... + driving_institution: Norwegian Climate Centre - driving_institute_id: CSIRO-BOM ? ^^^^^^^^^ + driving_institute_id: NCC ? + ^ - driving_model_id: ACCESS1-3 _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 1 _ Notebook cell execution failed Cell 1: Cell outputs differ Input: from clisops.core import subset lon = [-75.4, -85, -65.5] # Longitude lat = [46.67, 41, 55.3] # Latitude ds_gridpoint = subset.subset_gridpoint(ds, lon=lon, lat=lat) display(ds_gridpoint) # Plot first year of tasmax data a = ds_gridpoint.tasmax.isel(time=slice(0, 365)).plot.line(x="time", figsize=(10, 4)) Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data...id: NCC' == '<xarray.Data... ACCESS1-3' Skipping 39 identical leading characters in diff, use -v to show - time: 55152) ? - + time: 55115) ? + Coordinates: lat (site) float32 46.71 41.04 55.29 lon (site) float32 -75.38 -84.96 -65.46 - * time (time) datetime64[ns]/DATE/1950-01-02 ... 2100-12-31 + * time (time) object/DATE/TIME/.../DATE/00:00:00 Dimensions without coordinates: site Data variables: tasmin (time, site) float32 dask.array<chunksize=(512, 3), meta=np.ndarray> tasmax (time, site) float32 dask.array<chunksize=(512, 3), meta=np.ndarray> pr (time, site) float32 dask.array<chunksize=(512, 3), meta=np.ndarray> - Attributes: (12/27) ? ^ + Attributes: (12/26) ? ^ Conventions: CF-1.5 title: Ouranos standard ensemble of bias-adjusted cl... - history: CMIP5 compliant file produced from raw ACCESS... + history: DATE_TIME_TZ CMOR rewrote data to com... institution: Ouranos Consortium on Regional Climatology an... - source: ACCESS1-3 2011. Atmosphere: AGCM v1.0 (N96 gr... - driving_model: ACCESS1-3 + source: NorESM1-M 2011 atmosphere: CAM-Oslo (CAM4-Os... + driving_experiment: historical,rcp85 ... ... + frequency: day modeling_realm: atmos target_dataset: CANADA : ANUSPLIN interpolated Canada daily 3... target_dataset_references: CANADA : https://doi.org/10.1175/2011BAMS3132... - driving_institution: Commonwealth Scientific and Industrial Resear... + driving_institution: Norwegian Climate Centre - driving_institute_id: CSIRO-BOM ? ^^^^^^^^^ + driving_institute_id: NCC ? + ^ - driving_model_id: ACCESS1-3 _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 3 _ Notebook cell execution failed Cell 3: Cell outputs differ Input: # Specify the longitude and latitude boundaries lon_bnds = [-80.5, -60.2] lat_bnds = [44, 55] ds1 = subset.subset_bbox(ds, lon_bnds=lon_bnds, lat_bnds=lat_bnds) display(ds1) # Plot a map of first timestep a = ds1.tasmax.isel(time=0).plot() Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data...id: NCC' == '<xarray.Data... ACCESS1-3' Skipping 50 identical leading characters in diff, use -v to show - time: 55152) ? - + time: 55115) ? + Coordinates: * lat (lat) float32 54.96 54.87 54.79 54.71 ... 44.29 44.21 44.12 44.04 * lon (lon) float32 -80.46 -80.38 -80.3 -80.21 ... -60.38 -60.3 -60.21 - * time (time) datetime64[ns]/DATE/1950-01-02 ... 2100-12-31 + * time (time) object/DATE/TIME/.../DATE/00:00:00 Data variables: tasmin (time, lat, lon) float32 dask.array<chunksize=(512, 20, 28), meta=np.ndarray> tasmax (time, lat, lon) float32 dask.array<chunksize=(512, 20, 28), meta=np.ndarray> pr (time, lat, lon) float32 dask.array<chunksize=(512, 20, 28), meta=np.ndarray> - Attributes: (12/27) ? ^ + Attributes: (12/26) ? ^ Conventions: CF-1.5 title: Ouranos standard ensemble of bias-adjusted cl... - history: CMIP5 compliant file produced from raw ACCESS... + history: DATE_TIME_TZ CMOR rewrote data to com... institution: Ouranos Consortium on Regional Climatology an... - source: ACCESS1-3 2011. Atmosphere: AGCM v1.0 (N96 gr... - driving_model: ACCESS1-3 + source: NorESM1-M 2011 atmosphere: CAM-Oslo (CAM4-Os... + driving_experiment: historical,rcp85 ... ... + frequency: day modeling_realm: atmos target_dataset: CANADA : ANUSPLIN interpolated Canada daily 3... target_dataset_references: CANADA : https://doi.org/10.1175/2011BAMS3132... - driving_institution: Commonwealth Scientific and Industrial Resear... + driving_institution: Norwegian Climate Centre - driving_institute_id: CSIRO-BOM ? ^^^^^^^^^ + driving_institute_id: NCC ? + ^ - driving_model_id: ACCESS1-3 _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb::Cell 6 _ Notebook cell execution failed Cell 6: Cell outputs differ Input: # Align data to a starting year ds_sub = subset.subset_time(ds, start_date="1981") print( f"Subset time using start_date only\nstart: {ds_sub.time.min().values};\tend: {ds_sub.time.max().values}\n" ) # Select a temporal slice ds_sub = subset.subset_time(ds, start_date="1981-08-05", end_date="2084-06-15") print( f"Subset time using both start_date & end_date\nstart: {ds_sub.time.min().values};\tend: {ds_sub.time.max().values}" ) Traceback: mismatch 'stdout' assert reference_output == test_output failed: 'Subset time ...TE/00:00:00\n' == 'Subset time ...0.000000000\n' Subset time using start_date only - start: 1981-01-01T00:00:00.000000000; end: 2100-12-31T00:00:00.000000000 + start:/DATE/00:00:00; end:/DATE/00:00:00 Subset time using both start_date & end_date - start: 1981-08-05T00:00:00.000000000; end: 2084-06-15T00:00:00.000000000 + start:/DATE/00:00:00; end:/DATE/00:00:00 _ PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb::Cell 0 _ Notebook cell execution failed Cell 0: Cell outputs differ Input: import os os.environ["USE_PYGEOS"] = "0" # force use Shapely with GeoPandas import warnings import numba warnings.simplefilter("ignore", category=numba.core.errors.NumbaDeprecationWarning) import geopandas as gpd import matplotlib.pyplot as plt import xarray as xr from clisops.core import subset from dask.diagnostics import ProgressBar from siphon.catalog import TDSCatalog from xclim import atmos warnings.simplefilter("ignore") # TODO change address url = "https://boreas.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/cb-oura-1.0/catalog.xml" # TEST_USE_PROD_DATA # Create Catalog cat = TDSCatalog(url) # Subset over the Gasp�� peninsula in eastern Quebec gaspe = gpd.GeoDataFrame.from_file( "/notebook_dir/pavics-homepage/tutorial_data/gaspesie_mrc.geojson" ) ds = subset.subset_shape( xr.open_dataset(cat.datasets[0].access_urls["OPENDAP"], chunks=dict(time=256)), shape=gpd.GeoDataFrame(geometry=gaspe.buffer(0.05)), ) # What we see here is only a representation of the full content, the entire data set hasn't been loaded. display(ds) # plot of single day tasmin a = ds.tasmin.isel(time=0).plot(figsize=(10, 4)) Traceback: mismatch 'text/plain' assert reference_output == test_output failed: '<xarray.Data... EPSG:4326' == '<xarray.Data... EPSG:4326' Skipping 48 identical leading characters in diff, use -v to show - time: 55152) ? - + time: 55115) ? + Coordinates: * lat (lat) float32 49.29 49.21 49.12 49.04 ... 48.12 48.04 47.96 47.87 * lon (lon) float32 -67.71 -67.63 -67.55 -67.46 ... -64.3 -64.21 -64.13 - * time (time) datetime64[ns]/DATE/1950-01-02 ... 2100-12-31 + * time (time) object/DATE/TIME/.../DATE/00:00:00 Data variables: tasmin (time, lat, lon) float32 dask.array<chunksize=(256, 18, 44), meta=np.ndarray> tasmax (time, lat, lon) float32 dask.array<chunksize=(256, 18, 44), meta=np.ndarray> pr (time, lat, lon) float32 dask.array<chunksize=(256, 18, 44), meta=np.ndarray> crs int64 1 - Attributes: (12/28) ? ^ + Attributes: (12/27) ? ^ Conventions: CF-1.5 title: Ouranos standard ensemble of bias-adjusted cl... - history: CMIP5 compliant file produced from raw ACCESS... + history: DATE_TIME_TZ CMOR rewrote data to com... institution: Ouranos Consortium on Regional Climatology an... - source: ACCESS1-3 2011. Atmosphere: AGCM v1.0 (N96 gr... - driving_model: ACCESS1-3 + source: NorESM1-M 2011 atmosphere: CAM-Oslo (CAM4-Os... + driving_experiment: historical,rcp85 ... ... + modeling_realm: atmos target_dataset: CANADA : ANUSPLIN interpolated Canada daily 3... target_dataset_references: CANADA : https://doi.org/10.1175/2011BAMS3132... - driving_institution: Commonwealth Scientific and Industrial Resear... + driving_institution: Norwegian Climate Centre - driving_institute_id: CSIRO-BOM ? ^^^^^^^^ + driving_institute_id: NCC ? + ^ - driving_model_id: ACCESS1-3 crs: EPSG:4326 ```
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Most of the graphs are different, weird. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Only merge if bird-house/birdhouse-deploy#413 can be merged.
This is mostly to have a nice notebook diff.