Big FITS file tutorial

[ceb8]

• Check the box to confirm that you are familiar with the contributing guidelines and/or indicate (check the box) that you are familiar with our contributing workflow.
• Confirm that any contributed tutorials contain a complete Introduction which includes an Author list, Learning Goals, Keywords, Companion Content (if applicable), and a Summary.
• Check the box to confirm that you are familiar with the Astropy community code of conduct and you agree to follow the CoC.

This is an advanced tutorial that shows how to create a FITS file from scratch that is too large to fit in memory.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ceb8]

I'm not sure how to deal with the timing out issue. The point of this tutorial is to create an extra-large FITS file, so several of the operations to indeed take a bit of time, and can't really be modified not to.

[ceb8]

@embray Would you be willing to review this PR? I believe you were the author of the build-big-fits-file gallery example I built it off of and reference in the description (which I realized has now moved location, so I will update that).

[embray]

I'll try to take a look

[ceb8]

@embray thanks so much! No rush.

[adrn]

I think this looks great and we should get this in! One thing I would suggest is instead of your custom unit formatting function, just use astropy.units! Instead of print_file_size(), you could to:

import astropy.units as u
(os.path.getsize(path) * u.byte).to(u.kilobyte)

for example.

[embray]

Oh my goodness, when you wrote "no rush" I didn't mean to leave this lingering for 6 months either. I think I got swept up in summer travel and forgot.

[embray]

header = hdu.header

header["NAXIS2"] = array_dims[0]
header["NAXIS1"] = array_dims[1]

If I recall correctly, and I may not, I thought that when you instantiate an ImageHDU, say, it will automatically fill in the NAXISn headers (as well as the other basic data type headers, etc.). I might be wrong but it's worth checking. I seem to recall often see pyfits users wrangling these basic headers manually when there is no need to.

It's not so obvious but I think in the constructor it eventually sets self.data = data and then the magic happens here: https://github.com/astropy/astropy/blob/54a5a8e7d2217dda8968f10986198d97e01152d6/astropy/io/fits/hdu/image.py#L256
via the update_header() call towards the bottom.

Might be worth checking this and making a note of in the tutorial if I'm correct.

[embray]

with open(big_fits_fle, 'ab') as FITSFLE:  # 'ab' means open to append bytes
    FITSFLE.write(bytearray(header.tostring(), encoding="utf-8"))

Could maybe also use Header.tofile here https://github.com/astropy/astropy/blob/54a5a8e7d2217dda8968f10986198d97e01152d6/astropy/io/fits/header.py#L711

[embray]

mm = fits.util._get_array_mmap(data_array)
mm.madvise(MADV_SEQUENTIAL)

The idea of suggesting madvise is a nice one. It's a shame these extra steps need to be taken though. For now it's fine because there's no obvious "better" way but this might be a nice option to be able to set directly through the HDU object to use when writing to a file from mmap.

[embray]

small_hdu.header["EXTNAME"] = "MINI_IMG"

Piggie-backing in my previous comment about the NAXIS headers you can also set EXTNAME directly on the HDU like

hdu.name = 'MINI_IMG'

It doesn't really matter how you do it because it's effectively the same. For people who know FITS they'll know what they're doing here. I always just tried to hide FITS implementation details as much as possible. Up to you how you want to approach it.

[embray]

Anyways those are my only comments, this is great. I think there was some version of this in the main documentation too but I don't know how up to date it is. This is fun to read and easy to understand.

@adrn makes a good point too about using Astropy units for printing file sizes.

[ceb8]

I think this looks great and we should get this in! One thing I would suggest is instead of your custom unit formatting function, just use astropy.units! Instead of print_file_size(), you could to:

import astropy.units as u
(os.path.getsize(path) * u.byte).to(u.kilobyte)

for example.

I didn't know you could do this! I definitely will update it to do it this way.!

[ceb8]

Anyways those are my only comments, this is great. I think there was some version of this in the main documentation too but I don't know how up to date it is. This is fun to read and easy to understand.

@adrn makes a good point too about using Astropy units for printing file sizes.

Thanks so much for reviewing this! Very helpful!