Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add end-to-end, SDP-lite regtest #1386

Open
schlafly opened this issue Aug 27, 2024 · 1 comment
Open

Add end-to-end, SDP-lite regtest #1386

schlafly opened this issue Aug 27, 2024 · 1 comment

Comments

@schlafly
Copy link
Collaborator

In order to check out the full SDP interface, we could make a new regtest / smoketest that runs all of the ELP / HLP steps.

This is annoying in that it would be slow. We would want to do something like the following:

  • take two overlapping uncal files (e.g., existing 0001 and 0002 imaging files) for WFI01
  • run the ELP on both through source catalogs (i.e., skipping tweakreg, as the SDP does), creating "2a" files that aren't archived
  • make an association file for the "2a" files
  • run tweakreg / the ELP on the "2a" files to produce the final cal files
  • make an association file for the cal files for a particular skycell
  • run the mosaic pipeline on the association

It's not obvious to me that I would want to do very much with the resulting files (e.g., compare_asdf, etc.); I might continue to leave that to the existing set of regtests. But it's at least important to know that all of those steps run and to have a place with a SDP-lite pipeline that the real SDP pipeline can emulate. This would make us more aware when the SDP commands need to change slightly to accommodate changes to the pipeline.

@schlafly
Copy link
Collaborator Author

While checking a new version of the release I did the following as a kind of SDP-lite set of invocations:

# make level "2a" files
strun roman_elp r0000101001001001001_01101_0001_WFI01_uncal.asdf --steps.tweakreg.skip=True --suffix=sourcedetection
strun roman_elp r0000101001001001001_01101_0002_WFI01_uncal.asdf --steps.tweakreg.skip=True --suffix=sourcedetection
# make association file linking these
asn_from_list *_WFI01_sourcedetection.asdf --product-name aoeu -o asn_sourcedetection.json
# make final cal files
strun romancal.step.TweakRegStep asn_sourcedetection.json --suffix=cal
# make association file for cal files
asn_from_list *_cal.asdf --product-name=r0000101001001001001_F158_visit_r274dp63x32y80 -o mosaic.json
# this file then needs to get hand edited to add the target skycell name to the "target" bit of the asn file
# i.e., add target="r274dp63x32y80" to the json file
# make final mosaic product
strun roman_mos mosaic.json

This makes the following files:

asn_sourcedetection.json  # association file we made for source detection files, could be ignored?
mosaic.json  # association file we made for cal files, could be ignored?
r0000101001001001001_01101_0001_WFI01_cal.asdf  # L2 file
r0000101001001001001_01101_0001_WFI01_cat.asdf  # corresponding cat file
r0000101001001001001_01101_0001_WFI01_segm.asdf  # corresponding segmentation file
r0000101001001001001_01101_0001_WFI01_sourcedetection.asdf  # "2a" file that we shouldn't save
r0000101001001001001_01101_0001_WFI01_uncal.asdf  # uncal input file
r0000101001001001001_01101_0002_WFI01_cal.asdf  # L2 file
r0000101001001001001_01101_0002_WFI01_cat.asdf  # corresponding cat file
r0000101001001001001_01101_0002_WFI01_segm.asdf  # corresponding segm file
r0000101001001001001_01101_0002_WFI01_sourcedetection.asdf  # corresponding sourcedetection file
r0000101001001001001_01101_0002_WFI01_uncal.asdf  # uncal input file
r0000101001001001001_F158_visit_r274dp63x32y80_cat.asdf  # L3 catalog
r0000101001001001001_F158_visit_r274dp63x32y80_i2d.asdf  # L3 image
r0000101001001001001_F158_visit_r274dp63x32y80_segm.asdf  # L3 segmentation map
skycell_wcs.asdf  # temporarry file that should be deleted

I think something of that form would exercise much of the needed SDP interface, but @ctbheller , let me know if there's something else I can add that would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant