Skip to content

README_en.md

Latest commit

 

History

History
492 lines (360 loc) · 13.9 KB

README_en.md

File metadata and controls

492 lines (360 loc) · 13.9 KB
Raw

FAST Data Processor User Guide

FAST Data Processor is a preprocessing package for FAST PSRFITS observations. It reads a filelist.txt file, plots FITS files, combines polarization channels, crops frequency channels and TBIN rows, downsamples frequency channels, and merges processed FITS files into one final FITS file.

The command line interface intentionally exposes only three commands:

  • plot: generate plots only.
  • process: skip plotting, process FITS files, and merge the processed files.
  • pipeline: run plotting first, then run processing and merging.

Plotting runs in parallel by default. In process and pipeline, per-file processing runs in parallel by default, while the final FITS merge runs once in serial.

Installation

Installing in a Python virtual environment is recommended:

cd /Users/hyne/PycharmProjects/FAST/fast-data-processor
python -m venv .venv
source .venv/bin/activate
pip install -e .

After installation, the command line entry point is:

fast-process --help

You can also run from the source tree without installing:

python -m fast_data.cli --help

Input File List

filelist.txt should contain one FITS path per line:

PSRJ0002+6216_swiftcalibration-M18_0069.fits
PSRJ0002+6216_swiftcalibration-M18_0070.fits
PSRJ0002+6216_swiftcalibration-M18_0071.fits
PSRJ0002+6216_swiftcalibration-M18_0072.fits
PSRJ0002+6216_swiftcalibration-M18_0073.fits

Blank lines are ignored. Lines beginning with # are treated as comments and ignored.

Relative paths are resolved relative to the file-list location. For example:

data/a.fits
data/b.fits

If filelist.txt is located at /obs/run1/filelist.txt, these paths are resolved as:

/obs/run1/data/a.fits
/obs/run1/data/b.fits

Damaged FITS Files

If a FITS file is damaged or unreadable, for example with:

No SIMPLE card found

the software skips that file and continues processing the remaining inputs.

Skipped files are written to:

skipped_files.txt

inside the output directory. Each line contains the skipped file path and the error reason.

Default Frequency Cropping Rule

The processing stage in process and pipeline supports automatic frequency cropping. If --start-freq or --end-freq is not specified, the software first reads the actual frequency channel count, OBSNCHAN, from each FITS file and then resolves the default range as follows:

  • If OBSNCHAN = 4096, use the historical default range: start_freq = 408, end_freq = 3687.
  • If OBSNCHAN is not 4096, trim 10% of the channels from each band edge:
edge_channels = int(OBSNCHAN * 0.10)
start_freq = edge_channels
end_freq = OBSNCHAN - edge_channels - 1

For example, when OBSNCHAN = 2048:

edge_channels = 204
start_freq = 204
end_freq = 1843

If --start-freq or --end-freq is supplied explicitly, that boundary uses the user-provided value instead of the automatic value.

Default TBIN Row Cropping Rule

The processing stage in process and pipeline also supports automatic TBIN row cropping. If --start-time or --end-time is not specified, the software reads the number of SUBINT rows, NAXIS2, from each FITS file and resolves the default time range as follows:

  • If NAXIS2 = 128, use the historical default range: start_time = 0, end_time = 127.
  • If NAXIS2 is not 128, keep all rows: start_time = 0, end_time = NAXIS2 - 1.

If --start-time or --end-time is supplied explicitly, that boundary uses the user-provided value instead of the automatic value.

Merged Filename Rule

If --output is omitted, the final merged filename is generated from the first and last valid file names in filelist.txt.

For example, if the first file is:

PSRJ0002+6216_swiftcalibration-M18_0069.fits

and the last file is:

PSRJ0002+6216_swiftcalibration-M18_0073.fits

the merged filename is:

PSRJ0002+6216_swiftcalibration-M18_0069-0073.fits

The rule extracts the numeric sequence after the M..._ marker and preserves leading zeroes.

Polarization Handling

The default processing mode writes the tot polarization product.

The current implementation reads the first two polarization streams, scales them by their mean levels, and averages them into one total-intensity channel. The processed DATA column has one polarization dimension:

DATA shape = (NAXIS2, NSBLK, 1, NCHAN, 1)

The processed FITS header is written as:

NPOL = 1
POL_TYPE = I

The four 3280E columns often shown by FITS info tools are not four polarization channels. They are frequency-dependent metadata columns:

DAT_FREQ
DAT_WTS
DAT_OFFS
DAT_SCL

The actual data column is DATA. For example:

DAT_FREQ   3280E
DAT_WTS    3280E
DAT_OFFS   3280E
DAT_SCL    3280E
DATA       3358720B
TDIM17     (1,3280,1,1024)

Here:

3358720 = 1024 * 3280 * 1

which means 1024 samples, 3280 frequency channels, and 1 polarization channel.

Command 1: plot

plot generates spectrogram images only. It does not write processed FITS files.

Basic Usage

Plot one FITS file:

fast-process plot \
  --input data.fits \
  --output-dir plots/

Plot several FITS files:

fast-process plot \
  --input data1.fits data2.fits data3.fits \
  --output-dir plots/

Plot all files in filelist.txt:

fast-process plot \
  --filelist filelist.txt \
  --output-dir plots/ \
  --ncpus 8

You can also pass a single .txt file to --input; it will be treated as a file list:

fast-process plot \
  --input filelist.txt \
  --output-dir plots/

plot Parameters

Option Required Default Description
--input Either this or --filelist None One or more FITS files. If a single .txt file is supplied, it is treated as a file list.
--filelist Either this or --input None Text file containing one FITS path per line.
--output-dir Yes None Output directory for PNG images.
--start-freq No 400 Start frequency channel for plotting.
--end-freq No None End frequency channel for plotting. If None, the last channel in the file is used.
--start-time No 0 Start time bin for plotting.
--end-time No None End time bin for plotting. If None, the last SUBINT row is used.
--downsample No 64 Time downsample factor for plotting only. It does not modify FITS data.
--config No None Optional YAML config path.
--ncpus No None Number of worker processes. If omitted, CPU count minus one is used.

plot Outputs

The output directory contains:

<input_stem>_all_pols.png
plotted_files.txt

For non-four-polarization inputs, the command may produce:

<input_stem>_pol1.png
<input_stem>_pol2.png
...

plotted_files.txt records successfully plotted input files.

If damaged files are found, the command also writes:

skipped_files.txt

Command 2: process

process skips plotting. It writes processed per-file FITS files and merges them into one final FITS file.

Processing steps:

  1. Read filelist.txt.
  2. Skip damaged or unreadable FITS files.
  3. For each valid file, run in parallel:
    • combine polarization streams into total intensity I
    • crop frequency channels
    • crop TBIN rows
    • downsample frequency channels
    • write an intermediate processed FITS file
  4. Merge all intermediate processed FITS files in serial.
  5. Write the final merged FITS file.

Basic Usage

fast-process process \
  --filelist filelist.txt \
  --output-dir processed/ \
  --ncpus 8

Set the final merged FITS path explicitly:

fast-process process \
  --filelist filelist.txt \
  --output-dir processed/ \
  --output processed/merged.fits \
  --ncpus 8

Set crop and downsample options:

fast-process process \
  --filelist filelist.txt \
  --output-dir processed/ \
  --start-freq 408 \
  --end-freq 3687 \
  --start-time 0 \
  --end-time 127 \
  --fdsamp 4 \
  --ncpus 8

If --start-freq / --end-freq are omitted, the processing stage automatically resolves the frequency crop range from each file's OBSNCHAN:

fast-process process \
  --filelist filelist.txt \
  --output-dir processed/ \
  --fdsamp 4 \
  --ncpus 8

process Parameters

Option Required Default Description
--filelist Yes None Text file containing one FITS path per line.
--output-dir Yes None Output directory for intermediate processed FITS files and the final merged FITS file.
--output No None Final merged FITS path. If omitted, the name is generated from the first and last file names in the file list.
--start-freq No None First frequency channel to keep. If omitted, it is resolved from OBSNCHAN: 408 for 4096-channel files, otherwise trim 10% from the lower band edge.
--end-freq No None Last frequency channel to keep. If omitted, it is resolved from OBSNCHAN: 3687 for 4096-channel files, otherwise trim 10% from the upper band edge.
--start-time No None First TBIN row to keep. If omitted, 0 is used.
--end-time No None Last TBIN row to keep. If omitted, it is resolved from NAXIS2: 127 for 128-row files, otherwise the last row.
--fdsamp No 1 Frequency downsample factor. The selected channel count must be divisible by this value.
--overwrite / --no-overwrite No True Whether to overwrite an existing final merged FITS file.
--config No None Optional YAML config path.
--ncpus No None Number of worker processes. If omitted, CPU count minus one is used.

process Outputs

The output directory contains intermediate processed FITS files:

<input_stem>_tot_<start_freq>_<end_freq>_<start_time>_<end_time>_Fdsamp<fdsamp>.fits

Example:

PSRJ0002+6216_swiftcalibration-M18_0069_tot_408_3687_0_127_Fdsamp1.fits

If --output is omitted, the final merged FITS may look like:

PSRJ0002+6216_swiftcalibration-M18_0069-0073.fits

The command also writes a temporary merge list:

reduce_filelist_<timestamp>.txt

If any files are skipped, it writes:

skipped_files.txt

Command 3: pipeline

pipeline runs plot first and then runs process. Use it when you want visualization and preprocessing in one command.

Basic Usage

fast-process pipeline \
  --filelist filelist.txt \
  --output-dir results/ \
  --ncpus 8

Set processing and plotting options:

fast-process pipeline \
  --filelist filelist.txt \
  --output-dir results/ \
  --start-freq 408 \
  --end-freq 3687 \
  --start-time 0 \
  --end-time 127 \
  --fdsamp 4 \
  --plot-start-freq 400 \
  --plot-downsample 64 \
  --ncpus 8

If processing-stage --start-freq / --end-freq are omitted, pipeline uses the same automatic frequency cropping rule:

fast-process pipeline \
  --filelist filelist.txt \
  --output-dir results/ \
  --plot-start-freq 400 \
  --plot-downsample 64 \
  --ncpus 8

pipeline Parameters

Option Required Default Description
--filelist Yes None Text file containing one FITS path per line.
--output-dir Yes None Pipeline output directory. Plots, intermediate FITS files, and final merged FITS files are written here.
--output No None Final merged FITS path. If omitted, the name is generated from the first and last file names in the file list.
--start-freq No None Processing-stage start frequency channel. If omitted, it is resolved from OBSNCHAN: 408 for 4096-channel files, otherwise trim 10% from the lower band edge.
--end-freq No None Processing-stage end frequency channel. If omitted, it is resolved from OBSNCHAN: 3687 for 4096-channel files, otherwise trim 10% from the upper band edge.
--start-time No None Processing-stage start TBIN row. If omitted, 0 is used.
--end-time No None Processing-stage end TBIN row. If omitted, it is resolved from NAXIS2: 127 for 128-row files, otherwise the last row.
--fdsamp No 1 Processing-stage frequency downsample factor.
--plot-start-freq No 400 Plotting-stage start frequency channel.
--plot-end-freq No None Plotting-stage end frequency channel. If None, the last channel is used.
--plot-start-time No 0 Plotting-stage start time bin.
--plot-end-time No None Plotting-stage end time bin. If None, the last SUBINT row is used.
--plot-downsample No 64 Plotting-stage time downsample factor.
--overwrite / --no-overwrite No True Whether to overwrite an existing final merged FITS file.
--config No None Optional YAML config path.
--ncpus No None Number of worker processes. If omitted, CPU count minus one is used.

pipeline Outputs

The output directory contains all outputs from plot and process:

<input_stem>_all_pols.png
plotted_files.txt
<input_stem>_tot_<start_freq>_<end_freq>_<start_time>_<end_time>_Fdsamp<fdsamp>.fits
<auto_or_user_named_merged>.fits
reduce_filelist_<timestamp>.txt
skipped_files.txt

skipped_files.txt is written only when one or more inputs are skipped.

Help

Global help:

fast-process --help

Command help:

fast-process plot --help
fast-process process --help
fast-process pipeline --help

Each command help page shows default values for optional parameters.

Recommended Workflow

First, plot the data for quick inspection:

fast-process plot \
  --filelist filelist.txt \
  --output-dir plots/ \
  --ncpus 8

Then process after confirming the desired frequency and time ranges:

fast-process process \
  --filelist filelist.txt \
  --output-dir processed/ \
  --fdsamp 1 \
  --ncpus 8

Or run plotting and processing together:

fast-process pipeline \
  --filelist filelist.txt \
  --output-dir results/ \
  --ncpus 8