Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2022/11/17 Meeting Notes #17

Open
Victoria-Samboco opened this issue Nov 17, 2022 · 3 comments
Open

2022/11/17 Meeting Notes #17

Victoria-Samboco opened this issue Nov 17, 2022 · 3 comments

Comments

@Victoria-Samboco
Copy link
Contributor

Today's Meeting was reserved to discuss about the pipeline. Here the gist link with the recipe, cab and the find_stimela.py file.
https://gist.github.com/Victoria-Samboco/3fc580598b69ad2c76469848da13a620

The pipeline works from calibration until the image_sun step, but need to be improved to be more robust, which one of the changes must be made is to use the MS to split the scans and not put automatically the list of scans, convert the sun-coordinates.txt to a yml file.... I ask please a detailed summary of what are the changes and how to do it.
@Victoria-Samboco
Copy link
Contributor Author

This command doesn't work
prefix: ="obsuhf/im-sun/" + STRIPEXT(BASENAME(recipe.mss))
it returns the following error

2022-11-17 17:25:09 STIMELA ERROR: error loading solarkat-pipeline-perscan.yml: while   
scanning for the next token                                                                 
found character '\t' that cannot start any token                                            
  in "/home/samboco/solarkat/SUN_IMAGING_STEPS/solarkat-pipeline-perscan.yml", line 286,    
column 5                                                                                 
error loading solarkat-pipeline-perscan.yml
ScannerError: while scanning for the next token
found character '\t' that cannot start any token
  in "/home/samboco/solarkat/SUN_IMAGING_STEPS/solarkat-pipeline-perscan.yml", line 286, 
column 5

I've tried to put it like this prefix: "obsuhf/im-sun/" + STRIPEXT{BASENAME{recipe.mss}}, but also doesn't work.
here is the recipe

image_sun:
      info: "Sun Imaging through scans"
      params:
        ms_list: = GLOB("/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/*.ms")        
      recipe:
        inputs:
          ms_list:
            dtype: List[MS]
        for_loop:
          var: mss
          over: ms_list
        steps:
          image:
            _use: lib.steps.wsclean.image
            params:
              ms: '{recipe.mss}'
              column: CORRECTED_DATA
              temp_dir: tmp
              niter: 0
              mgain: 0.9
              multiscale: false
              subtract-model: true
              #prefix: 'obsuhf/im-sun/scan{recipe.mss@index}'
    	      prefix: "obsuhf/im-sun/" + STRIPEXT{BASENAME{recipe.mss}}

with this command prefix: 'obsuhf/im-sun/scan{recipe.mss@index}' it works but the output doesn't come with the scan number, it starts from index zero scan0...fits, scan1...fits, .... which is not bad, but if we can have the image scan number matching with the scan number I think could be better.

@o-smirnov
Copy link
Collaborator

\t refers to a tab character. Could you please check the first case to make sure you didn't have a tab in there? I should look into making the parser less sensitive to these things...

@Victoria-Samboco
Copy link
Contributor Author

Victoria-Samboco commented Nov 22, 2022 •
edited
Loading

prefix: ="obsuhf/im-sun/" + STRIPEXT(BASENAME(recipe.mss))
I'm not sure if the = must be there because when so, stimela returns an error, so I removed it.
still not working, it run but then returns:

2022-11-22 08:43:27 STIMELA.boom.image_sun-1 INFO: for loop iteration 0: mss =
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-
scan11.ms
2022-11-22 08:43:27 STIMELA.boom.image_sun-1 INFO: processing step 'image'
2022-11-22 08:43:27 STIMELA.boom.image_sun-1 INFO:   (wsclean step for imaging)
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO: ### validated inputs
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO: cab image:
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   prefix = obsuhf/im-sun/ +                       
STRIPEXT(BASENAME(recipe.mss))
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   column = CORRECTED_DATA
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   nchan = 8
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   weight = briggs 0
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   size = 6076
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   multiscale = False
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   scale = 1.5asec
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   niter = 0
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   mgain = 0.9
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   padding = 1.3
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   subtract-model = True
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   temp_dir = tmp
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   source-list = obsuhf/im-sun/ +                  
STRIPEXT(BASENAME(recipe.mss))-sources.txt
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   dirty = []
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   restored = []
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   restored-timeint = []
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   residual = []
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   model = []
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   restored-mfs = obsuhf/im-sun/ +                 
STRIPEXT(BASENAME(recipe.mss))-MFS-image.fits
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   residual-mfs = obsuhf/im-sun/ +                 
STRIPEXT(BASENAME(recipe.mss))-MFS-residual.fits
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   model-mfs = obsuhf/im-sun/ +                    
STRIPEXT(BASENAME(recipe.mss))-MFS-model.fits
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   dirty-mfs = obsuhf/im-sun/ +                    
STRIPEXT(BASENAME(recipe.mss))-MFS-dirty.fits
2022-11-22 08:43:27 STIMELA.boom.image_sun-1.image INFO:   ms =                                            
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-can11.ms
# running /home/samboco/Git_clone/wsclean/build/wsclean -name obsuhf/im-sun/ +                              
STRIPEXT(BASENAME(recipe.mss)) -data-column CORRECTED_DATA -channels-out 8 -weight briggs 0 -size 6076 6076    
-scale 1.5asec -niter 0 -mgain 0.9 -padding 1.3 -subtract-model -temp-dir tmp                                  
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-can11.ms

# WSClean version 3.2 (2022-10-21)
# This software package is released under the GPL version 3.
# Author: André Offringa ([email protected]).
# 
# === IMAGING TABLE ===
#        # Pol Ch JG ²G FG FI In Freq(MHz)
# | Independent group:
# +-+-J- 0  I   0  0  0  0  0  0  544-612 (128)
# | Independent group:
# +-+-J- 1  I   1  1  1  1  0  0  612-680 (128)
# | Independent group:
# +-+-J- 2  I   2  2  2  2  0  0  680-748 (128)
# | Independent group:
# +-+-J- 3  I   3  3  3  3  0  0  748-816 (128)
# | Independent group:
# +-+-J- 4  I   4  4  4  4  0  0  816-884 (128)
# | Independent group:
# +-+-J- 5  I   5  5  5  5  0  0  884-952 (128)
# | Independent group:
# +-+-J- 6  I   6  6  6  6  0  0  952-1020 (128)
# | Independent group:
# +-+-J- 7  I   7  7  7  7  0  0  1020-1088 (128)
# Reordering                                                                                                
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-
scan11.ms into 8 x 1 parts.
# Reordering: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
#  == Constructing image ==
# Precalculating weights for Briggs'(0) weighting...
# Opening reordered part 0 spw 0 for                                                                        
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-
can11.ms
# Detected 503.6 GB of system memory, usage not limited.
# Opening reordered part 0 spw 0 for                                                                        
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-
can11.ms
# Determining min and max w & theoretical beam size... DONE (w=[0.00230244:7122.71] lambdas, maxuvw=15706.8 
lambda)
# Theoretic beam = 13.13''
# Minimal inversion size: 1805 x 1805, using optimal: 1920 x 1920
# The theoretically suggested number of w-layers (22) is less than the number of availables
# cores (64). Changing suggested number of w-layers to 64.
# Suggested number of w-layers: 64
# Will process 64/64 w-layers per pass.
# Gridding pass 0...
# Rows that were required: 234612/0
# Fourier transforms...
# Gridded visibility count: 2.07687e+07
# Writing changed model back to                                                                             
/home/samboco/solarkat/SUN_IMAGING_STEPS/scansuhf/perscan_ms/1583662427_sdp_l0.1024ch-J033230_280757-corr_self-
can11.ms:
#  0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
# Cleaning up temporary files...
# + + + + + + + + + + + + + + + + + + +
# + An exception occured:
# + >>> CFITSIO reported error when performing IO on file 'obsuhf/im-sun/ +                                 
STRIPEXT(BASENAME(recipe.mss))-residual-0000-tmp.fits':file open failed (parser) ()
# + + + + + + + + + + + + + + + + + + +
2022-11-22 08:43:54 STIMELA.boom.image_sun-1.image INFO: /home/samboco/Git_clone/wsclean/build/wsclean exited
with code 255 after 0:00:26
2022-11-22 08:43:54 STIMELA.boom.image_sun-1.image ERROR: /home/samboco/Git_clone/wsclean/build/wsclean    
returned non-zero exit status 255
2022-11-22 08:43:54 STIMELA.boom INFO: saving full profiling stats to ./obsuhf/logs/log-/stimela.stats

The 'obsuhf/im-sun/ + STRIPEXT(BASENAME(recipe.mss)) doesn't do what it's supposed to do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@o-smirnov @Victoria-Samboco