PyCBC Live O3 development

PyCBC Live Development

To Do Table for O3/ER14

Issue	description	Assigned to	Status
Reduce memory usage on master node		Alex/Tito	Done
Verify FAR + pvalue combination	To see if satisfies expected distribution as shown here: pvalue combining tests	Tom, Alex
Verify triggers from simulated data	To check triple, double coincs and single trigger
Verify triggers from O2 replay	To check triple, double coincs and single trigger
Verify stability of multidetector analysis	Check lag, duty factor and memory usage over time	Tito, Khun Sang (?)	Stable over O(hours), need longer test
GraceDB upload	To check the structure of the coinc xml file and check uploads	Bhooshan	Uploading works, need to fix a few BAYESTAR-related bugs
Follow-up early trigger https://pycbc.slack.com/archives/lvc-low-latency/p1550510201048400	Error while assigning p-value to trigger at the beginning of the run	Alex, Bhooshan
Phase-time-amp files for all HLV combinations		Tito	Under test

Additional things

Task	Description	Assigned to	Relevance + Status
Unit testing			As soon as possible
Data fetching and processing at main process	Data fetching and preprocessing to head node of MPI and then distribute to filtering nodes	Bhooshan	Not necessary for O3 but good to investigate
Autogating	Applying autogating to strain	Liting Xiao	Done, needs testing
SG Chisq	Use SG chisq and relevant statistics	Ian	Done, needs testing
Save more metadata in output	add channel names of strain, status and DQ, status and DQ flags, PyCBC version, values of thresholds, maybe the entire command line to output hdf5 file	Tito	Basically done in #2538, should also save latency/duty factor.
Data-fetching after template generation	Start reading frames at a time after the template waveforms have been generated. This should reduce the amount of lookback frames that need to be present. Sometimes this results in a file not found error if the bank takes a long time to generate.		Done
Trigger followup	Upload plots of trigger SNR time series and background		Good to have for rapid responses in O3
Run on Python 3			Good to have towards end of 2019
p_astro	PyCBC Live should become compatible with the code for generating p_astro in low latency		Sometimes in O3
Better handling of single-detector triggers	Signals loud in one detector but not others should generate triggers with an appropriate FAR		Sometimes in O3
iDQ integration	Can low-latency iDQ info be used to improve VT?		Hopefully in O3

Multi IFO Plan Outline

• Calculate all possible double coinc combinations
• Pick "best" one
• For all remaining IFOS calculate a p-value for the on-source time with the same template
• combine double coinc FAR with pvalues from other ifos

Methods and options for improvements

• handling of singles

  • improvement: handle case where one detector is non-vetoed by not the only "active" one.

• Preparation of strain

  • improvement: add in autogating

• Calculation of the pvalue

  • improvement: Better check of DQ in the background time

• How to pick the "best" double coinc?

  • Simple implementation: best far, largest SNR
    • Downside: Larger trials factor
  • Possibly better: Choose based on best instrument combination, past noise, sensitivity

• How to combine FAR with pvalues from additional detectors?

  • Simple implementation: Fisher's method, using fiducial "foreground time" (.01 * IFAR).
  • Possibly better: Take trials factor if additional ifos will reduce significance (might expect low snr)
  • Possibly better: Do a weighted combinations of p-values based on additional information (detector sensitivity, background).

Write methods paper

1. Where?
2. By when? (Before first publication on O3 data would be nice!)
3. Paper with 5 detector network and for BNS and NSBH (q < 5) (may be)

(Don't need answers here yet, but this needs to be on the radar)

Older page for reference: PyCBC-Live-O3-development-Old