M31 Example

[christinawlindberg]

This is an example workflow for the Beast. It needs the cutout M31 data to run, but that file is larger than 100 MB so let me know if there's a good way for it to run.

The only other thing that's different step-wise is when the datamodel is introduced. For this workflow, it isn't introduced until right before the physics model is created because the datamodel references some files (like the source density map) that isn't normally created until later, making it hard to explain to a new user.

[christinawlindberg]

@karllark

[karllark]

Very cool! Will take a close look at this in the next few days.

[christinawlindberg]

@cmurray-astro
This new example notebook should hopefully work for everyone. Please let me know if there's any text or explanation that is incorrect.

[cmurray-astro]

Some comments the workflow through step 2 below! Overall it's looking great -- we should probably talk through the comments to fix up a few things. In the meantime:

• Can add wget statements to the notebook for fetching the required data.
• Would it clean things up in the example directory to have a "data/" folder for storing the required data products or something like that? Would also help to have quick explanations of each downloaded file.
• Include a plot of the magnitude histograms within the notebook, rather than pointing to the pdf (same for all plots throughout)?
• The datamodel is missing a distance_prior_model which I think is a BEAST update made recently. Should fetch the most recent BEAST version and re-run.

[cmurray-astro]

In addition, rather than having the user manually update the datamodel throughout, maybe we can write a function for doing this as needed (similar to the create_datamodel function in Lea's workflow)?

[karllark]

F20 needs to be changed to F19. Newer version of dust_extinction has the F19 name for this model (paper was published faster than expected).

[kmcquinn]

I found this error in Step 7:

create the noise model with our ASTs

create_obsmodel.create_obsmodel(
use_sd=True, nsubs=datamodel.n_subgrid, nprocs=1
)

FileNotFoundError: [Errno 2] No such file or directory: 'M31-B09-EAST_chunk.gst.fake_bin2.fits'

[christinawlindberg]

I found this error in Step 7:

create the noise model with our ASTs

create_obsmodel.create_obsmodel(
use_sd=True, nsubs=datamodel.n_subgrid, nprocs=1
)

FileNotFoundError: [Errno 2] No such file or directory: 'M31-B09-EAST_chunk.gst.fake_bin2.fits'

@karllark @kmcquinn

This error is happening because the files in the datamodel.py weren't manually renamed. Look at step 4.5 and 5.5 to see what specific changes need to be made (sorry it's so clunky). We've since then created a function that will automatically rename the variable names in the datamodel.py file but it hasn't been uploaded to the pull request yet.

[karllark]

Is this PR ready for another review (and possible merging)?

[christinawlindberg]

Almost! There are some minor changes I had to make to update some function keywords, and then the last step where we merge all the files into a stats.fits file fails.

merge_files.merge_files(settings, use_sd=True, nsubs=settings.n_subgrid)

---------------------------------------------------------------------------
AstropyUserWarning                        Traceback (most recent call last)
<ipython-input-49-ae68a0817681> in <module>
----> 1 merge_files.merge_files(settings, use_sd=True, nsubs=settings.n_subgrid)

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/beast/tools/run/merge_files.py in merge_files(beast_settings_info, use_sd, nsubs)
     71         reorder_tags = ["bin{0}_sub{1}".format(x[0], x[1]) for x in unique_sd_sub]
     72         merge_beast_stats.merge_stats_files(
---> 73             stats_files, out_filebase, reorder_tag_list=reorder_tags
     74         )
     75 

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/beast/tools/merge_beast_stats.py in merge_stats_files(stats_files, out_stats_filebase, reorder_tag_list)
     31 
     32         # read in current catalog
---> 33         cur_cat = Table.read(cur_stat)
     34 
     35         if reorder_tag_list is None:

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     50     def __call__(self, *args, **kwargs):
     51         cls = self._cls
---> 52         out = registry.read(cls, *args, **kwargs)
     53 
     54         # For some readers (e.g., ascii.ecsv), the returned `out` class is not

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs)
    521 
    522         reader = get_reader(format, cls)
--> 523         data = reader(*args, **kwargs)
    524 
    525         if not isinstance(data, cls):

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    214         try:
    215             return read_table_fits(hdulist, hdu=hdu,
--> 216                                    astropy_native=astropy_native)
    217         finally:
    218             hdulist.close()

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    169                               " are present, reading in first available"
    170                               f" table (hdu={first(tables)})",
--> 171                               AstropyUserWarning)
    172                 hdu = first(tables)
    173 

AstropyUserWarning: hdu= was not specified but multiple tables are present, reading in first available table (hdu=1)

[karllark]

Is this an actual error? Looks like a warning. This is the result of a recent PR of mine where I added another extension to the stats file to provide the filter information (useful for the plot_indiv_fit and other scripts). This warning can be removed by changing the line cur_cat = Table.read(cur_stat) to cur_cat = Table.read(cur_stat, hdu=1) in beast/tools/merge_beast_stats.py.

[christinawlindberg]

It says that it's just a user warning but then the stats.fits file is never outputted.
I'll try to change the function first though.

[christinawlindberg]

I tried changing the function in my code and it still produces the same error.

AstropyUserWarning                        Traceback (most recent call last)
<ipython-input-67-ae68a0817681> in <module>
----> 1 merge_files.merge_files(settings, use_sd=True, nsubs=settings.n_subgrid)

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/beast/tools/run/merge_files.py in merge_files(beast_settings_info, use_sd, nsubs)
     71         reorder_tags = ["bin{0}_sub{1}".format(x[0], x[1]) for x in unique_sd_sub]
     72         merge_beast_stats.merge_stats_files(
---> 73             stats_files, out_filebase, reorder_tag_list=reorder_tags
     74         )
     75 

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/beast/tools/merge_beast_stats.py in merge_stats_files(stats_files, out_stats_filebase, reorder_tag_list)
     31 
     32         # read in current catalog
---> 33         cur_cat = Table.read(cur_stat, hdu=1)
     34 
     35         if reorder_tag_list is None:

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     50     def __call__(self, *args, **kwargs):
     51         cls = self._cls
---> 52         out = registry.read(cls, *args, **kwargs)
     53 
     54         # For some readers (e.g., ascii.ecsv), the returned `out` class is not

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs)
    521 
    522         reader = get_reader(format, cls)
--> 523         data = reader(*args, **kwargs)
    524 
    525         if not isinstance(data, cls):

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    214         try:
    215             return read_table_fits(hdulist, hdu=hdu,
--> 216                                    astropy_native=astropy_native)
    217         finally:
    218             hdulist.close()

~/anaconda3/envs/astroconda/lib/python3.6/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    169                               " are present, reading in first available"
    170                               f" table (hdu={first(tables)})",
--> 171                               AstropyUserWarning)
    172                 hdu = first(tables)
    173 

AstropyUserWarning: hdu= was not specified but multiple tables are present, reading in first available table (hdu=1)```

[karllark]

Sounds like more detailed investigation is needed. Probably need look more carefully at the merge_beast_stats.py file.

[christinawlindberg]

When I define the functions in the notebook I'm working in, then the method works. Maybe the change I made in the source files isn't working somehow or being read properly. Would it possible to update the main branch with this fix to see if that will fix my problem?

[karllark]

A PR is always welcome!