Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 Aggregation Best Practice? #215

Open
dblodgett-usgs opened this issue Apr 9, 2022 · 13 comments
Open

S3 Aggregation Best Practice? #215

dblodgett-usgs opened this issue Apr 9, 2022 · 13 comments

Comments

@dblodgett-usgs
Copy link
Contributor

I'm testing with:
unidata/thredds-docker:5.3

I have a "content root":
TDS_CONTENT_ROOT_PATH=/mnt/content

And have mounted TDS configuration files there. Otherwise, I'm working in a stock TDS.

I have a /mnt/thredds that I use to store configuration files. I think of this as the "data root".

Side note:
In our non-AWS production system, /mnt/thredds also houses all our netcdf data content, some of which are joinExisting aggregations of many thousands of files, some of which also union a collection of joinExisting aggregations such that we have many variables in one OPeNDAP endpoint. e.g. https://cida.usgs.gov/thredds/dodsC/loca_future.html

In my testing for migration to S3, I am trying to work out what the best configuration pattern will be to get performance out of these data.

I've determined that a catalog/ncml pattern like either of the following will function and result in a joinExisting cache that contains the indices of the joinExisting dimension:

  <dataset name="test prism agg regexp (works but SLOW?)" ID="prism-agg-3" urlPath="prism-agg/agg3" dataType="Grid" serviceName="all">
    <ncml:netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" id="prism-agg-3">
      <aggregation dimName="time" type="joinExisting" timeUnitsChange="true">
        <scan location="cdms3://default@aws/nhgf-development?thredds/prism_v2/#delimiter=/" regExp=".*nc"/>
      </aggregation>
    </ncml:netcdf>
  </dataset>

or

  <dataset name="test prism agg explicit" ID="prism-agg-2" urlPath="prism-agg/agg2" dataType="Grid" serviceName="all">
    <ncml:netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" id="prism-agg-2">
      <aggregation dimName="time" type="joinExisting" timeUnitsChange="true">
        <netcdf location="cdms3://default@aws/nhgf-development?thredds/prism_v2/prism_2019.nc"/>
        <netcdf location="cdms3://default@aws/nhgf-development?thredds/prism_v2/prism_2020.nc"/>
      </aggregation>
    </ncml:netcdf>s
  </dataset>

Now to the crux of my issue. I'm getting very slow responses when querying these aggregations for coordinate variables which really need to be snappy.

In contrast to the stand along catalog dataset elements above, I currently have my catalog and ncml layed out using three tiers that help separate concerns.

  1. xml THREDDS catalog metadata containing a netcdf element with a location attribute pointing to an ncml file.
  2. The ncml file contains dataset metadata and a netcdf element with a location attribute pointing to
  3. an ncml aggregation file with the contents of the xml snips above.

My thought is to modify my ncml in the third tier so that, rather than pointing to data adjacent to it in a traditional file system, it points to a cdms3 location either using a scan or using explicit object paths.

But then I still run up against this issue where my reads of coordinate variables are super slow. In this particular case it is because the time bounds variable is not cached and has to be retrieved to read in the file. In other cases, I would really like to avoid scanning all the data to populate a cache in the first place.

So I guess I have three questions.

  1. Is this three-tiered catalog metadata -> ncml metadata -> ncml aggregation layout going to cause us issues with caching or in other ways I'm not aware of?
  2. Is it possible to pre-cache coordinate variables in joinExisting or other ncml collections? Where would I go to find information about that?
  3. long shot, but I've never used the TDM -- should I pursue deploying that for its utility in managing the caches?

Thanks -- and please tell me tho RTFM if one exists!
@haileyajohnson
Copy link

If only we had good enough documentation to just say RTFM :)

Overall, if you're finding S3 aggregations to be cumbersomely slow, that's bad and we'd like to work with you on that. But first, we should try some aggregation caching options. Caching is on by default in the TDS, and caches coordinate variable for joinExisting collections the first time they are read (see here. You can configure your cache policy in your threddsConfig.xml (see here).

If your aggregations are dynamic, you'll also need to add a "recheckEvery" property to your aggregation.
I also think you need to change regExp=".*nc" to suffix=".*nc".

Note that the first time accessing your datasets after starting your server might still be slow, but subsequent accesses should be pretty fast. If they're not, we need to look into that.

To answer your other questions:

  1. I'm not aware of any reason that layout would cause issues, but could I see what's going into the ncml metadata tier?
  2. The TDM creates indexes for GRIB files, so it wouldn't help in your case, unfortunately

@dblodgett-usgs
Copy link
Contributor Author

Thanks for following up @haileyajohnson -- I appreciate it.

My aggregation cache does appear to be working. I've set scour and maxAge to
"-1" and I'm seeing the expected slow initial scan then fast response.

The dataset in question is hosted on our old server here.

It is really just illustrative of this case where I'd really like a bit more than the joinExisting dimension to be cached. In this case, the time_bounds is slow from S3 because it has to be read out of every file. But more generally, I'd really like to be able to cache all 1D and maybe 2D variables such that queries for coordinate variables are always fast. Is this possible? Could I inject them into NCML such that they would never need to be retrieve from object store in the first place? I might just try that and see what happens!

Thanks for the TDM nuance - I'll keep ignoring that :)

@cofinoa
Copy link
Contributor

cofinoa commented Apr 13, 2022

@dblodgett-usgs,

As @haileyajohnson is pointing, you are facing some issues that are amplified 10-100 times, by the cdms3: terrible latency (i.e. HTTP). These issues are mitigated in your "local" system because the "low-latency" of the filesystem you are using.

The main issue is that, for every file you are aggregating, the joinExisting needs to be open and read how many elements has on the time dimension.

You can mitigate that by specifying the number of elements in time dimension are in each netcdf file, by using the ncoords attribute:

<dataset name="test prism agg explicit" ID="prism-agg-2" urlPath="prism-agg/agg2" dataType="Grid" serviceName="all">
    <ncml:netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" id="prism-agg-2">
      <aggregation dimName="time" type="joinExisting" timeUnitsChange="true">
        <netcdf location="cdms3://default@aws/nhgf-development?thredds/prism_v2/prism_2019.nc" ncoords="365"/>
        <netcdf location="cdms3://default@aws/nhgf-development?thredds/prism_v2/prism_2020.nc" ncoords="366"/>
      </aggregation>
    </ncml:netcdf>s
  </dataset>

then any variable on the resulting aggregation using the time dimension "knows" how many elements are in total, and each file. You can find more info on this, RTFM:
Defining coordinates on a JoinExisting aggregation

The second, and related issue, is the timeUnitsChange="true" which impose to open each file to read the time units attribute.

How I'm solving this issues, it's to create and external .ncml where all coordinates, with its values and metadata explicit created and just data aggregation on main variables it's perform, avoiding to read each file.

You can point to this .ncml from your catalog entry.

If you need "dynamical" aggregations, then a external cron can rebuild the external NCML and in combination with the recheckEvery in the catalog entry.

Hope this helps

@dblodgett-usgs
Copy link
Contributor Author

OK -- thanks @cofinoa -- I think this mostly answers my question.

to create and external .ncml where all coordinates, with its values and metadata explicit created and just data aggregation on main variables it's perform, avoiding to read each file.

Do you have a sample of how you're doing this? And can you confirm that THREDDS will not attempt to readd data that has already been declared in the ncml directly?

@cofinoa
Copy link
Contributor

cofinoa commented Apr 13, 2022

@dblodgett-usgs

I have cooked the following example for the "external" .ncml (with the help of @zequihg50).

For each remote netcdf dataset on the JoinExisting aggregation, I have added the ncoords attribute with the corresponding values.
Then I have added the time coordinate, with all values for the resulting time aggregation, avoiding to access "all" files.

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
    <attribute name="__info__" value="Virtual dataset generated by the ESGF Virtual Aggregation Project (https://github.com/SantanderMetGroup/EVA)."/>
    <attribute name="__license__" value="This is a derived dataset product from ESGF, same licenses from original datasets apply for this dataset."/>
    <attribute name="__eva_version__" value=""/>
    <variable name="time" shape="time" type="double">
      <attribute name="bounds" value="time_bnds" />
      <attribute name="units" value="days since 1850-1-1" />
      <attribute name="calendar" value="gregorian" />
      <attribute name="axis" value="T" />
      <attribute name="long_name" value="time" />
      <attribute name="standard_name" value="time" />
      <values start="60265.5" increment="1" />
    </variable>
    <aggregation type="joinExisting" dimName="time">
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20150101-20151231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20160101-20161231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20170101-20171231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20180101-20181231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20190101-20191231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20200101-20201231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20210101-20211231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20220101-20221231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20230101-20231231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20240101-20241231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20250101-20251231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20260101-20261231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20270101-20271231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20280101-20281231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20290101-20291231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20300101-20301231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20310101-20311231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20320101-20321231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20330101-20331231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20340101-20341231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20350101-20351231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20360101-20361231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20370101-20371231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20380101-20381231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20390101-20391231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20400101-20401231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20410101-20411231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20420101-20421231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20430101-20431231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20440101-20441231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20450101-20451231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20460101-20461231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20470101-20471231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20480101-20481231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20490101-20491231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20500101-20501231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20510101-20511231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20520101-20521231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20530101-20531231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20540101-20541231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20550101-20551231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20560101-20561231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20570101-20571231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20580101-20581231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20590101-20591231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20600101-20601231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20610101-20611231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20620101-20621231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20630101-20631231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20640101-20641231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20650101-20651231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20660101-20661231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20670101-20671231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20680101-20681231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20690101-20691231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20700101-20701231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20710101-20711231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20720101-20721231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20730101-20731231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20740101-20741231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20750101-20751231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20760101-20761231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20770101-20771231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20780101-20781231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20790101-20791231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20800101-20801231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20810101-20811231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20820101-20821231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20830101-20831231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20840101-20841231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20850101-20851231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20860101-20861231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20870101-20871231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20880101-20881231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20890101-20891231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20900101-20901231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20910101-20911231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20920101-20921231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20930101-20931231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20940101-20941231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20950101-20951231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20960101-20961231.nc" ncoords="366"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20970101-20971231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20980101-20981231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_20990101-20991231.nc" ncoords="365"/>
        <netcdf location="http://esgf-data2.diasjp.net/thredds/dodsC/esg_dataroot/CMIP6/ScenarioMIP/MIROC/MIROC-ES2L/ssp126/r1i1p1f2/day/tas/gn/v20200318/tas_day_MIROC-ES2L_ssp126_r1i1p1f2_gn_21000101-21001231.nc" ncoords="365"/>
    </aggregation>
</netcdf>

If you access the .ncml there will be only just file open, the first one, to capture all metadata for the resulting file.

@dblodgett-usgs
Copy link
Contributor Author

Got it. Thank you!

@dblodgett-usgs
Copy link
Contributor Author

A follow up question here - I have some large union aggregations, some set up with a scan others with an explicit list. The first time the union is accessed after a restart, it takes a long time to come back. Long enough that we are seeing 504 time outs. Is there any way to persist whatever is retrieved when a large union is scanned like that?

@cofinoa
Copy link
Contributor

cofinoa commented May 18, 2022

you can persist the resulting aggregation dataset to a external NCML and ...

You can point to this .ncml from your catalog entry.

If you need "dynamical" aggregations, then a external cron can rebuild the external NCML and in combination with the recheckEvery in the catalog entry.

@dblodgett-usgs
Copy link
Contributor Author

Thanks for following up, @cofinoa. It's not clear to me how this would work for a union aggregation. Each file has one variable and I'm unioning hundreds or thousands of these files. e.g. https://cida.usgs.gov/thredds/dodsC/cmip5_bcca/future.html

On a traditional storage device the latency to assemble the basic metadata for the union is very fast but on S3 it takes enough time to cause a time out.

@cofinoa
Copy link
Contributor

cofinoa commented May 18, 2022

@dblodgett-usgs after the TDS start ... and the first access to the Union aggregation based on scan .... the second and so on ... are timing out?

From your URL example, my guess is that TDS has cached the aggregation after the first access, and the following access are much faster.

can you share the NCML used in your example? the S3 endpoints are public accesible?

@dblodgett-usgs
Copy link
Contributor Author

The first access times out and it is much faster while the aggregation is "fresh". But the result of the scan doesn't seem to be persisted any where so when the server is restarted or enough time and/or other usage has elapsed, the end point is subject to time out due to a rescan. I'm trying to figure out if there is a way to avoid this scan in the first place.

@cofinoa
Copy link
Contributor

cofinoa commented May 18, 2022

there is a config parameter related with aggregation cache:
https://docs.unidata.ucar.edu/tds/current/userguide/tds_config_ref.html#aggregation-cache

but It say that it's only for JoinExisting aggregations, but you can give a try.

@dblodgett-usgs
Copy link
Contributor Author

I've verified that the aggregation cache does not work.

Out of curiosity, I used toolsui to generate an ncml template of the dataset with location="dods://..."

If I remove the location attribute and include the union aggregation within the netcdf element, I thought I was seeing a speed up of that initial scan, but upon further testing, it's not a panacea. The initial scan for some of these is still far too long.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dblodgett-usgs @cofinoa @haileyajohnson