Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail more quickly on too large ncss requests #366

Closed
haileyajohnson opened this issue May 8, 2023 · 2 comments
Closed

Fail more quickly on too large ncss requests #366

haileyajohnson opened this issue May 8, 2023 · 2 comments

Comments

@haileyajohnson
Copy link

To avoid long, unsuccessful ncss requests, look into calculating the data download size as soon as possible.

@edesfandiari
Copy link

A netcdf subset service (ncss) request for a large file download can almost take an
hour to return a "Request Too Large"! Following is an example and maybe a possible
solution to shorten that time.

The approximate final file size may be determined from the ncss cache file that
is produced almost immediately. For example, a ncss query immediately produced
ncss-grid218435382.nc (50980 bytes). Fifty five minutes later, it grow to over 9GB
before returning a "Request Too Large: Too large, max size = 5192000000".

Examining the dimensions in the short 50980 bytes file (e.g. ncdump, below) may
be used to estimate the final file size.

ncdump -h /var/lib/tomcats/ops/content/thredds/cache/ncss/ncss-grid218435382.nc
netcdf ncss-grid218435382 {
dimensions:
      lon = 3600 ;
      time = 365 ;
      lat = 1800 ;

This yields a file size of 3600 x 365 x 1800 x 4bytes = ~9.5GB which can be compared
the allowed maximum size.

@haileyajohnson
Copy link
Author

This should be fixed by Unidata/netcdf-java#1213

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants