data manager issue with sunpy.instr.goes

[hayesla]

Description

I know we are planning to move the instr out of core, but I'm playing around with sunpy.instr. calculate_temperature_em(), and there seems to be a bug with the data manager to get the required chianti files.

Expected behavior

>>> goes_ts = TimeSeries('go1520120514.fits').truncate('2012-05-14 13:32', '2012-05-14 13:45')
>>> ts_new = goes.calculate_temperature_em(goes_ts)   

and this should return a lovely timeseries with two extra columns for temperature and emission measure

Actual behavior

/Users/laurahayes/anaconda3/lib/python3.7/functools.py:824: UserWarning: Discarding nonzero nanoseconds in conversion
  return dispatch(args[0].__class__)(*args, **kw)
Files Downloaded: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  8.08file/s]
WARNING: SunpyUserWarning: [Errno 2] No such file or directory: '/Users/laurahayes/sunpy/data_manager/goes_chianti_temp_cor.csv' [sunpy.data.data_manager.cache]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-e8e77b0651c9> in <module>
----> 1 ts_new = goes.calculate_temperature_em(goes_ts)

~/sunpy_dev/sunpy-git/sunpy/instr/goes.py in calculate_temperature_em(goests, abundances, download, download_dir)
    268         satellite=goests.meta.metas[0]["TELESCOP"].split()[1],
    269         date=goests.to_dataframe().index[0],
--> 270         abundances=abundances, download=download, download_dir=download_dir)
    271 
    272     ts_new = timeseries.XRSTimeSeries(meta=copy.deepcopy(goests.meta),

~/anaconda3/lib/python3.7/site-packages/astropy/units/decorators.py in wrapper(*func_args, **func_kwargs)
    232             # Call the original function with any equivalencies in force.
    233             with add_enabled_equivalencies(self.equivalencies):
--> 234                 return_ = wrapped_function(*func_args, **func_kwargs)
    235             if wrapped_signature.return_annotation not in (inspect.Signature.empty, None):
    236                 return return_.to(wrapped_signature.return_annotation)

~/sunpy_dev/sunpy-git/sunpy/instr/goes.py in _goes_chianti_tem(longflux, shortflux, satellite, date, abundances, download, download_dir)
    418     temp = _goes_get_chianti_temp(fluxratio, satellite=satellite,
    419                                   abundances=abundances, download=download,
--> 420                                   download_dir=download_dir)
    421     em = _goes_get_chianti_em(longflux_corrected, temp, satellite=satellite,
    422                               abundances=abundances, download=download,

~/sunpy_dev/sunpy-git/sunpy/data/data_manager/manager.py in wrapper(*args, **kwargs)
     78                         if self._cache_has_file(urls):
     79                             raise ValueError(" Hash provided does not match the hash in database.")
---> 80                         file_path = self._cache.download(urls)
     81                         if hash_file(file_path) != sha_hash:
     82                             # the hash of the file downloaded does not match provided hash

~/sunpy_dev/sunpy-git/sunpy/data/data_manager/cache.py in download(self, urls, redownload)
     84                 return Path(details['file_path'])
     85 
---> 86         file_path, file_hash, url = self._download_and_hash(urls)
     87 
     88         self._storage.store({

~/sunpy_dev/sunpy-git/sunpy/data/data_manager/cache.py in _download_and_hash(self, urls)
    169                 warn(e, SunpyUserWarning)
    170         else:
--> 171             raise RuntimeError("Download failed")

RuntimeError: Download failed

Where is the bug?

I'm pretty confused about where the bug is lying - where is the hash key generated here?

@manager.require('file_temp_cor',
                 [urljoin(GOES_REMOTE_PATH, FILE_TEMP_COR)],
                 '3d8ddaaabf0faf75ba8d15e0c468896ce3d7622cc23076bf91437951e0ab3ad4')

The url exists and can be downloaded, the data manager is hurting my head 🤯

[vn-ki]

WARNING: SunpyUserWarning: [Errno 2] No such file or directory: '/Users/laurahayes/sunpy/data_manager/goes_chianti_temp_cor.csv' [sunpy.data.data_manager.cache]

Looking at the warning, I'd say the file is not downloaded. Can you confirm whether the file is actually downloaded or not?

If that doesn't give any insights, try removing the warn and raise the Exception produced.

https://github.com/sunpy/sunpy/blob/12225c4315d7dbe071bd0e3283adbf3f3e5f1709/sunpy/data/data_manager/cache.py#L168-L169

@manager.require('file_temp_cor',                                                                                                                       
                  [urljoin(GOES_REMOTE_PATH, FILE_TEMP_COR)],                                                                                            
                  '3d8ddaaabf0faf75ba8d15e0c468896ce3d7622cc23076bf91437951e0ab3ad4')                                                                    
 def foo():                                                                                                                                              
     pass                                                                                                                                                

The above code works fine for me. You can try this as a minimal example to reproduce the issue.

The url exists and can be downloaded, the data manager is hurting my head exploding_head

I'm sorry 😅. All the bad code in there should be attributed to me. Better error handling is due for downloading, I think.

[hayesla]

Hey @vn-ki !

No the file was not downloaded but I can't figure out why its not?

I'm sorry 😅. All the bad code in there should be attributed to me. Better error handling is due for downloading, I think.

The data manager is great! Its my own ignorance in how it works is the problem 😆

[dstansby]

WARNING: SunpyUserWarning: [Errno 2] No such file or directory: '/Users/laurahayes/sunpy/data_manager/goes_chianti_temp_cor.csv' [sunpy.data.data_manager.cache]

looks like the key - maybe /Users/laurahayes/sunpy/data_manager doesn't exist?

[vn-ki]

looks like the key - maybe /Users/laurahayes/sunpy/data_manager doesn't exist?

shouldnt that be automatically created?

I purged the directories in the morning to test it out and it worked without any problems.

No the file was not downloaded but I can't figure out why its not?

@hayesla I think you'll have to replace that warn with a raise (at least for now) to get a better insight into it.

[hayesla]

looks like the key - maybe /Users/laurahayes/sunpy/data_manager doesn't exist?

This exists - I've checked it

>>> os.path.exists('/Users/laurahayes/sunpy/data_manager/')
 True

Yep, I'll play around with it some more.

Can I ask a silly question though - how are the hash keys generated?

[vn-ki]

how are the hash keys generated?

sha256 iirc

[hayesla]

thanks!

Actually where else in core is the data manager used? Just want to test out some stuff

[wtbarnes]

It's not as far as I know. But we use it in aiapy: https://gitlab.com/LMSAL_HUB/aia_hub/aiapy/-/blob/master/aiapy/response/channel.py#L77

[vn-ki]

https://github.com/sunpy/sunpy/search?q=%22%40manager%5C.require%22&unscoped_q=%22%40manager%5C.require%22

This should be everything. Looks like we are not using enough of data manager 😆