Skip to content

Run neutrino filter from XML files. #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
schuetha wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Run neutrino filter from XML files. #292

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7557a59
Add run xml scripts
schuetha Oct 28, 2025
9eb0f2c
Revert "Add run xml scripts"
schuetha Oct 28, 2025
98cef31
Add run xml scripts
schuetha Oct 28, 2025
c9dc1cf
Add the config.sh file and add use of sndGeometryGetter function
schuetha Nov 11, 2025
666e643
Adjust tehe code to be able to run in parallel and the .list file mak…
schuetha Nov 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions analysis/scripts/run_neutrino_filter_xml.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#!/bin/bash

xml=$1
output=$2
mode=$3
cutset=$4

# Iterate through every <run> automatically
echo "Reading the XML file: $xml"
for run in $(xmllint --xpath 'count(/runlist/runs/run)' "${xml}"); do :; done # pre-check count

mode_list=(STAGE1 RECO STAGE2)

if [[ ! ${mode_list[@]} =~ $mode ]]
then
echo Mode $mode not available. It must be one of "${mode_list[*]}"
echo Exitting.
exit
fi

if [ -z ${SNDSW_ROOT+x} ]
then
echo "Setting up SNDSW"
export ALIBUILD_WORK_DIR=/afs/cern.ch/user/s/schuetha/work/public/data_work_flow/sw
Copy link

@siilieva siilieva Oct 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no hard-coding of specific user dirs please

Copy link
Author

@schuetha schuetha Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@siilieva What I can think of is to let the user input their ALIBUILD_WORK_DIR into the shell script. Do you have any suggestions?

Copy link

@cvilelahep cvilelahep Nov 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scripts @eduard322 prepared for running MC productions have a separate config.sh file where this sort of user-related environment is set up. Maybe we could use a similar strategy here? @schuetha, you could commit an example config.sh.example script with your own hard-coded paths and with some comments with instructions on what to update.
See @eduard322's script here:
https://github.com/eduard322/snd_dag/blob/main/config.sh

Copy link

@eduard322 eduard322 Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, since you are discussing here the unified way of launching htcondor events, we can prepare a unified repo (or folder) for the most frequently simulated events with scripts that can be used almost out-of-the box. What do you think? I have updated now the above mentioned repo, so that now it has instructions, argparser for choosing the simulation parameter and can generate automatically README file in the simulation folder with the parameters that had been used for this dataset. It is based on HTCondor DAG, not the perfect solution, but now works more or less stable.

Copy link

@cvilelahep cvilelahep Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good idea. I started preparing some scripts for running the event selection code based on your DAG scripts.
I'm doing this in a fork of your repo. Still work in progress: https://github.com/cvilelahep/snd_dag/blob/main/run_submit_data_proc.py

Copy link

@siilieva siilieva Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would you agree it is best to have that unified repo under the SNDLHC organization? A thumb up is enough.
I'd propose to add the MC coordinators as admins of it, so they can approve PRs and handle it fully.
We only need a general structure to control its growth a bit.
Like, we can have as a start subdirs for different job cases e.g. simulation, analysis, electronic det data management(e.g. track reconstruction, data reprocessing) and emulsion data management (alignment, reconstruction(track, vertices) etc).
Also we need to make sure we adopt the info from the nusim_automation, I believe it was the basis for other cases developed by individuals.
We can discuss the setup on SoftPhys meeting tomorrow. I will add a separate slot.
Perhaps you realized reading so far, I am on board for this idea :D

Copy link

@eduard322 eduard322 Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added significant changes into the pipeline: https://github.com/eduard322/snd_dag. Now the simulation parameters can be set via flags or via yaml file in simulation_configs folder. Also I added the script that automtically creates gdml file via shipLHC/makeGeoFile.py. README now describes the whole process of the simulation and how to launch. To launch any kind of simulation, user needs just to specify the sw folder and the output folder. Feedback and suggestions are welcome!

source /cvmfs/sndlhc.cern.ch/SNDLHC-2025/Jan30/setUp.sh # recommended latest version
eval `alienv -a slc9_x86-64 load --no-refresh sndsw/master-local1`
# eval `alienv -a slc9_x86-64 load --no-refresh sndsw/latest`
echo "Finished setting up SNDSW env"
export EOSSHIP=root://eosuser.cern.ch
fi

n=$(xmllint --xpath 'count(/runlist/runs/run)' "${xml}")
for i in $(seq 1 $n)
do
run_number=$(xmllint --xpath "string(/runlist/runs/run[$i]/run_number)" "${xml}")
start_year=$(xmllint --xpath "string(/runlist/runs/run[$i]/start)" "${xml}")
year=${start_year:0:4}
end=$(xmllint --xpath "string(/runlist/runs/run[$i]/end)" "${xml}")
n_events=$(xmllint --xpath "string(/runlist/runs/run[$i]/n_events)" "${xml}")
n_files=$(xmllint --xpath "string(/runlist/runs/run[$i]/n_files)" "${xml}")
n_files=$((n_files-1)) # to make the seq 0 based
path=$(xmllint --xpath "string(/runlist/runs/run[$i]/path)" "${xml}")

if [[ "$year" == "2022" || "$year" == "2023" ]]; then
geo_file="/eos/experiment/sndlhc/legacy_geofiles/2023/geofile_sndlhc_TI18_V4_2023.root"
Copy link

@siilieva siilieva Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why reading the legacy geo files and not the updated ones?

Copy link

@cvilelahep cvilelahep Nov 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was ran some checks on this. The legacy geo file give slightly different DS track reconstruction results compared to using the geo files on the physics directories.

In particular, event number 23622206 in run 5159 is selected with the legacy geofile but it fails the cut requiring that the DS track intercepts the z=300 cm plane within 5 cm of the SciFi edges if the physics directory geofile is used for the track reconstruction.

@siilieva, do we expect legacy geofiles to result in different DS reconstruction compared to the ones in the physics dir?

Copy link

@cvilelahep cvilelahep Nov 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a more general solution to this problem, maybe we could use the geometry file path getter in tools?
https://github.com/SND-LHC/sndsw/blob/master/analysis/tools/sndGeometryGetter.h#L12C21-L12C31

Copy link

@siilieva siilieva Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5159

short answer: yes if using the /eos/experiment/sndlhc/legacy_geofiles/2023/geofile_sndlhc_TI18_V4_2023.root and not the /eos/experiment/sndlhc/legacy_geofiles/2023/geofile_sndlhc_TI18_V3_2023.root
The legacy file geofile_sndlhc_TI18_V4_2023.root was done in Feb 2024 for the 2023 ion run, for which we had issues with low-efficiency in tracking. So the MuFilter spatial alignment was redone for this target run specifically as a measure to solve it(Ivo's report 13th Feb 2024). In the end legacy 2023 V4 has little differences wrt legacy 2023 V3. Little means below the DS res of 0.33cm, but that can be huge for an extrapolation to z=300cm. We ended up separating the geos for 2022 and 2023 to have this fully sorted.
As reported on the June 2025 CM slide 15 here the new geofiles, ones stored in /physics, showed good behaviour when redong all det alignments over the 2022-2023 runs.
Also, trying the 2dED run 5159, partition 23, event 622 206 using the
latest 2022 geo, left plot: /eos/experiment/sndlhc/convertedData/physics/2022/geofile_sndlhc_TI18_V4_2022.root
and the
legacy geo, right plot: /eos/experiment/sndlhc/legacy_geofiles/2023/geofile_sndlhc_TI18_V4_2023.root
it seems the track passes the criterion for a 5-cm distance at z=300cm with very little though so it is surely a close-mis when using the proper track extrapolation and not the 2dED fast solution
Screenshot from 2025-11-11 10-53-31

else
echo "May need other Geofile year than 2023 Legacy. Exitting."
fi

if [ "$mode" == "STAGE1" ]
then
for j in $(seq 0 "$n_files")
do
if [ -f "${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}.root" ]
then
echo "File ${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}.root exists. Skipping."
continue
fi
js=$(printf "%03d" $((j)))
neutrinoFilterGoldenSample ${path}/sndsw_raw-0${js}.root filtered_MC_00${run_number}_${j}.root ${cutset}
mkdir -p ${output}/${run_number}/${mode}/
xrdcp -f ./filtered_MC_00${run_number}_${j}.root ${output}/${run_number}/${mode}/
rm -f ./filtered_MC_00${run_number}_${j}.root
done

elif [ "$mode" == "RECO" ]
then
for j in $(seq 0 "$n_files")
do
if [ -f "${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}__muonReco.root" ]
then
echo "File ${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}.root exists. Skipping."
continue
fi
python3 $SNDSW_ROOT/shipLHC/run_muonRecoSND.py -f ${output}/${run_number}/STAGE1/filtered_MC_00${run_number}_${j}.root -g ${geo_file} -c passing_mu_DS -sc 1 -s ./ -hf linearSlopeIntercept -o
mkdir -p ${output}/${run_number}/${mode}/
xrdcp -f ./filtered_MC_00${run_number}_${j}__muonReco.root ${output}/${run_number}/${mode}/
rm -f ./filtered_MC_00${run_number}_${j}__muonReco.root
done

elif [ "$mode" == "STAGE2" ]
then
for j in $(seq 0 "$n_files")
do
if [ -f "${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}_stage2_noscifi2.root" ]
then
echo "File "${output}/${run_number}/${mode}/filtered_MC_00${run_number}_${j}_stage2_noscifi2.root" exists. Skipping."
continue
fi
python3 ${SNDSW_ROOT}/analysis/neutrinoFilterGoldenSample_stage2.py -f ${output}/${run_number}/STAGE1/${run_number}/filtered_MC_00${run_number}_${j}.root -t ${output}/${run_number}/RECO/filtered_MC_00${run_number}_${j}__muonReco.root -o filtered_MC_00${run_number}_${j}_stage2_noscifi2.root -g ${geo_file};
mkdir -p ${output}/${run_number}/${mode}/
xrdcp -f ./filtered_MC_00${run_number}_${j}_stage2_noscifi2.root ${output}/${run_number}/${mode}/
rm -f ./filtered_MC_00${run_number}_${j}_stage2_noscifi2.root
done

else
echo "Mode $mode not recognized. Exitting."
exit
fi
echo "Finished processing run number $run_number"
done
55 changes: 55 additions & 0 deletions analysis/scripts/run_neutrino_xml_all_stage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/bin/bash

xml=$1
output=$2
cutset=$3

# Iterate through every <run> automatically
echo "Reading the XML file: $xml"
for run in $(xmllint --xpath 'count(/runlist/runs/run)' "${xml}"); do :; done # pre-check count

if [ -z ${SNDSW_ROOT+x} ]
then
echo "Setting up SNDSW"
export ALIBUILD_WORK_DIR=/afs/cern.ch/user/s/schuetha/work/public/data_work_flow/sw
Copy link

@siilieva siilieva Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no private dirs please

source /cvmfs/sndlhc.cern.ch/SNDLHC-2025/Jan30/setUp.sh # recommended latest version
eval `alienv -a slc9_x86-64 load --no-refresh sndsw/master-local1`
echo "Finished setting up SNDSW env"
export EOSSHIP=root://eosuser.cern.ch
fi

n=$(xmllint --xpath 'count(/runlist/runs/run)' "${xml}")
for i in $(seq 1 $n)
do
run_number=$(xmllint --xpath "string(/runlist/runs/run[$i]/run_number)" "${xml}")
start_year=$(xmllint --xpath "string(/runlist/runs/run[$i]/start)" "${xml}")
year=${start_year:0:4}
end=$(xmllint --xpath "string(/runlist/runs/run[$i]/end)" "${xml}")
n_events=$(xmllint --xpath "string(/runlist/runs/run[$i]/n_events)" "${xml}")
n_files=$(xmllint --xpath "string(/runlist/runs/run[$i]/n_files)" "${xml}")
n_files=$((n_files-1)) # to make the seq 0 based
path=$(xmllint --xpath "string(/runlist/runs/run[$i]/path)" "${xml}")

if [[ "$year" == "2022" || "$year" == "2023" ]]; then
geo_file="/eos/experiment/sndlhc/legacy_geofiles/2023/geofile_sndlhc_TI18_V4_2023.root"
else
echo "May need other Geofile year than 2023 Legacy. Exitting."
fi

for j in $(seq 0 "$n_files")
do
js=$(printf "%03d" $((j)))
neutrinoFilterGoldenSample ${path}/sndsw_raw-0${js}.root filtered_MC_00${run_number}_${j}.root ${cutset}
python3 $SNDSW_ROOT/shipLHC/run_muonRecoSND.py -f filtered_MC_00${run_number}_${j}.root -g ${geo_file} -c passing_mu_DS -sc 1 -s ./ -hf linearSlopeIntercept -o
python3 ${SNDSW_ROOT}/analysis/neutrinoFilterGoldenSample_stage2.py -f ${run_number}/filtered_MC_00${run_number}_${j}.root -t filtered_MC_00${run_number}_${j}__muonReco.root -o filtered_MC_00${run_number}_${j}_stage2_noscifi2.root -g ${geo_file};
done

mkdir -p ${output}/${run_number}/
xrdcp -f ./filtered_MC_00${run_number}_*.root ${output}/${run_number}/
xrdcp -f ./filtered_MC_00${run_number}_*__muonReco.root ${output}/${run_number}/
xrdcp -f ./filtered_MC_00${run_number}_*_stage2_noscifi2.root ${output}/${run_number}/
rm -f ./filtered_MC_00${run_number}_*.root
rm -f ./filtered_MC_00${run_number}_*__muonReco.root
rm -f ./filtered_MC_00${run_number}_*_stage2_noscifi2.root
echo "Finished processing run number $run_number"
done