Injection Auditing, T1 Candidate Dumping, Updated Trigger Logic, Offline T2 Script, and Bug Fixes

[vishnubk]

This PR introduces new optional features, updates the trigger logic of T2 for injections, adds an offline T2 replay script, and some bug fixes.

---

New Features

1. Injection Auditing

Enabled with --audit_injections.

When active, the system tracks each injected FRB as it passes (or fails) through the major T2 gates. It maintains:

• an append-only timeline CSV (audit_log_<date>.csv)
• a rolling snapshot (injections.csv)
• optional per-injection JSON records (state/<inj_id>.json) when --audit_dump_json is set

Each injection is evaluated through the T2 decision path using a simple state model:

• -1 = N/A
• 0 = failed
• 1 = passed

Gates tracked:

• G0: injected
• G1: parsed
• G2: T1-detected
• G3: beam-kept
• G4: clustered
• G5: filters-passed
• G6: cooldown-ok
• G7: triggered

This provides a clear diagnostic trail for injection testing and is useful for validating different parts of the T2 pipeline.

2. T1 Candidate Dumping

Enabled with --dump_t1.

Raw T1 candidates can now be written to disk for offline debugging and replay. Appends a single file for each day t1_candidates_<date>.csv.
--t1_dump_dir controls the output location (default is /operations/T2/cluster_output/).

3. Offline T2 Script (offline_t2.py)

Added offline_t2.py, a standalone tool for evaluating any version of T2 entirely offline:

• no voltage triggers
• no Slack messages
• no interaction with live services

It reads previously dumped T1 candidate files and runs them through the T2 decision logic, producing a record of what T2 would have triggered.
This is very useful for:

• comparing two versions of T2
• validating changes to gating or clustering logic

---

Improvements

Updated Injection Script

The injection service should now use new_injection_script.py, replacing the older dsa_injection.py.
The new script:

• draws injection SNRs from a Euclidean-distribution.
• injects into both NS and EW beams
• uses a unified scale factor for now (future work will split these per arm)

Trigger Handling During Injections

Suppresses voltage triggers for injected events. This logic has also been implemented for flush triggers.

---

Bug Fixes

Empty Gulp Handling

A gulp with zero candidates now appends 0 to nbeams_queue.
This prevents long cooldown stalls after RFI storms and improves overall T2 up-time.

Single-arm SNR logic has been corrected.

Previously, the single-arm SNR check inside filter_clustered was applied at a global level, which would produce incorrect behavior in some edge cases.

The updated implementation applies the SNR thresholds per candidate using boolean masks, and the two-arm and one-arm logic is now evaluated consistently on a row-by-row basis.

---

Fixed pytests not building

Additional small support files were added to allow all existing and new pytest tests to pass given the new module layout and options.

---

Future Work

• Consolidate logging across modules. There are still redundant print and logger.info statements that should be replaced with just logger.info / logger.debug calls. Before doing this, need to revisit how DsaSyslogger attaches handlers to avoid duplication.

• Extend the injection script so NS and EW arms use different scale factors, and tie those scale factors more directly to the requested SNR.

[caseyjlaw]

Great set of changes. Very happy to have your eyes on this and fixing the tests is much appreciated!

[caseyjlaw]

@vishnubk I've made one more commit to get it to merge. Be sure to merge main back into your development branch.
I'll deploy this now on cal23.