Higgs Signal Classification using XGBoost and SHAP

---

Highlights

• Dataset: ~11 Million HIGGS Events
• Best Model: Tuned XGBoost
• Test AUC: 0.8548
• Explainability: SHAP
• Learning Curve Analysis up to 5M samples

---

Table of Contents

• Overview
• Research Questions
• Dataset
• Methodology
• Results
• SHAP Analysis
• Learning Curve
• Key Findings
• Repository Structure
• References

---

Overview

This project investigates the classification of Higgs boson signal events from background noise using machine learning on the HIGGS dataset. This study reproduces and extends the work of Baldi et al. (2014) by comparing Logistic Regression and modern XGBoost models across different feature groups, then performing hyperparameter optimization, analyzing feature importance using SHAP, and evaluating the effect of training dataset size through learning curve analysis.

---

Research Questions

1. How does modern XGBoost perform compared to Baldi et al. (2014) BDT, NN, and DNN -> How much classical ML improved?
2. Which physics features actually drive the classification?
3. How much information is contained in low-level detector features versus high-level physics engineered features?
4. How much training data is required before performance begins to saturate?

---

Dataset

Source: UCI Machine Learning Repository - HIGGS Dataset

Dataset Characteristics

• Total Events - (~ 11,000,000)
• Features - 28
• Target - Binary Classification (Signal vs Background)
• Low-Level Features - 21
• High-Level Features - 7

Signal vs. Background Count

The first 21 features correspond to detector-level measurements, while the remaining 7 are physics-derived variables constructed from combinations of low-level measurements.

Features correlations (low-level features and high level features)

---

Methodology

Baseline Models The following models were trained on three feature subsets:

• 21 Low - Level features
• 7 High - Level features
• All 28 Features

Models:

• Logistic Regression
• XGBoost

Hyperparameter Tuning RandomizedSearchCV was used to optimize:

• max_depth
• learning_rate
• n_estimators
• subsample
• colsample_bytree
• min_child_weight
• reg_alpha
• reg_lambda

Best Parameters:

max_depth        = 8
learning_rate    = 0.1
subsample        = 0.9
colsample_bytree = 0.7
min_child_weight = 3
gamma            = 0
reg_alpha        = 0.1
reg_lambda       = 1
n_estimators     = 6194

Explainability

SHAP (SHapley Additive exPlanations) was used to:

• Rank feature importance
• Interpret model predictions
• Analyze feature effects

Learning Curve An XGBoost model with the best parameter was trained on progressively larger subsets:

| Step | 10K | 50K | 100K | 250K | 500K | 1M | 2.5M | 5M |

---

Results

Validation Performance

Model	21 features	7 features	28 features
Logistic Regression	0.5944	0.6451	0.6831
XGBoost (default)	0.7265	0.7899	0.8240
XGBoost (tuned)	0.8007	0.7962	0.8547

Final Test Performance

Model	Test AUC Score
XGBoost (tuned) 21 features	0.8005
XGBoost (tuned) 7 features	0.7955
XGBoost (tuned) 28 features	0.8548

Comparision with Baldi et al. (2014)

To directly compare modern gradient boosting with the original study by Baldi et al. (2014), the models were evaluated using the same three feature partitions: 21 low-level features, 7 high-level features, and all 28 features.

Model	Low-Level (21)	High-Level (7)	All 28
Logistic Regression	0.5944	0.6451	0.6831
BDT (Baldi et al., 2014)	0.7300	0.7800	0.8100
Shallow Neural Network (Baldi et al., 2014)	0.7330	0.7770	0.8160
XGBoost (Default)	0.7265	0.7899	0.8240
XGBoost (Tuned)	0.8005	0.7955	0.8548
Deep Neural Network (Baldi et al., 2014)	0.8800	0.8000	0.8850

Note: Results reported for BDT, Shallow Neural Network, and Deep Neural Network are taken from Baldi et al. (2014). The XGBoost models in this project were trained using the same feature partitions to enable a direct comparison.

Key Takeaways

• Tuned XGBoost outperforms the 2014 Boosted Decision Tree (BDT) across all three feature subsets.
• Using all 28 features, the tuned XGBoost model achieved a Test AUC of 0.8548, representing a substantial improvement over the original BDT baseline (0.8100).
• The performance of tuned XGBoost approaches that of the Deep Neural Network reported by Baldi et al. (2014), while maintaining the advantages of a classical gradient boosting model, including faster training and greater interpretability through SHAP.

SHAP Feature Importance

Top 5 Most Important Features (28 features)

Rank	Feature	MeanAbsSHAP
1	m_bb	0.605438
2	m_wbb	0.301321
3	jet 1 pt	0.293371
4	m_jjj	0.282922
5	m_jlv	0.275762

These results align with known physics expectations regarding Higgs boson decay signatures, indicating that invariant mass metrics provide the strongest geometric cuts for classification.

SHAP Summary Plot - (28 Features)

SHAP Feature Importance Ranking - (28 Features)

SHAP Dependence plot

Most important feature overall: m_bb

SHAP Summary Plot - (7 Features)

SHAP Feature Importance Ranking - (7 Features)

Learning Curve - Results

Training Size	Validation AUC	Best Iterations	Training Time (sec)	AUC Gain
10K	0.782333	55	1.823531	NaN
50K	0.805421	108	1.475551	0.023087
100K	0.813080	190	2.207243	0.007659
250K	0.821936	413	4.575684	0.008856
500K	0.828882	577	7.442126	0.006945
1M	0.834993	1034	16.674511	0.006111
2.5M	0.843019	2261	64.083215	0.008026
5M	0.848819	3526	194.734479	0.005801

Learning Curve - Tuned XGBoost

Training Time vs Dataset Size

Early stopping Iteration vs Dataset Size

Key Findings

1. High-level features are highly informative The physics - engineered features provide substantial discriminative power and outperform low-level detector measurements when used alone.

2. Combining feature sets produces the best performance. The highest performance was achieved using all 28 features

Final Test AUC: 0.8548

3. Hyperparameter tuning significantly improves performance

Feature	Default AUC	Tuned AUC
21 Features	0.7265	0.8005
7 Features	0.7899	0.7955
28 Features	0.8240	0.8548

4. Validation and test performance are consistent

Feature Set	Validation AUC	Test AUC
21 Features	0.8007	0.8005
7 Features	0.7962	0.7955
28 Features	0.8547	0.8548

---

Technologies Used

• Python
• Pandas
• NumPy
• Matplotlib
• Scikit-Learn
• XGBoost
• SHAP

Repository Structure

.
├── notebooks/
│   ├── 01_data_analysis.ipynb
│   ├── 02_eda_plots.ipynb
│   ├── 03_ml_pipeline.ipynb
|   ├── 04_ml_models.ipynb
│   ├── 05_shap_analysis.ipynb
│   └── 06_learning_curve.ipynb
│
├── data/
│   ├── raw
|   ├── processed
│   └── output
│
├── plots/
│   ├── high_level_features/
│   │   ├── m_bb.png
│   │   ├── m_jjj.png
│   │   ├── m_jj.png
│   │   └── ...
│   │
│   ├── low_level_features/
│   │   ├── jet 1 b-tag.png
│   │   ├── jet 1 eta.png
│   │   ├── jet 1 phi.png
│   │   └── ...
|   |
│   ├── signal vs background count.png
|   ├── feature_correlation matrix.png
│   ├── SHAP Summary Plot - Tuned XGBoost (28 features).png
│   └── ...
│
├── models/
│   ├── logistic_regression_21.pkl
│   ├── logistic_regression_7.pkl
│   ├── logistic_regression_28.pkl
│   ├── xgb_21_default.pkl
│   ├── xgb_7_default.pkl
│   ├── xgb_28_default.pkl
│   ├── final_xgb_21_tuned.pkl
│   ├── final_xgb_7_tuned.pkl
│   └── final_xgb_28_tuned.pkl
│
├── requirements.txt
└── README.md

---

Conclusion

Modern XGBoost substantially outperforms the original BDT and Shallow NN, and approaches the Deep Neural Network reported by Baldi et al.. while providing interpretable feature attributions through SHAP. The learning curve further demonstrates that predictive performance continues to improve with additional training data, although gains diminish at larger scales.

---

Installation

1. Clone the repository

git clone https://github.com/Master-Zero1/higgs-signal-classifier.git
cd higgs-signal-classifier

2. Create a virtual environment (optional but recommended)

Windows

python -m venv .venv
.venv\Scripts\activate

Linux / macOS

python -m venv .venv
source .venv/bin/activate

3. Install the required packages

pip install -r requirements.txt

---

Dataset

The HIGGS dataset is available from the UCI Machine Learning Repository.

Download the dataset from:

https://archive.ics.uci.edu/dataset/280/higgs

Place the downloaded CSV file inside:

data/raw/

The repository does not include the dataset because of its large size (~2.6 GB compressed, ~7.5 GB extracted).

---

Running the Project

Run the notebooks in the following order:

1. 01_data_analysis.ipynb

   • Load the dataset
   • Rename features
   • Create train, validation and test splits

2. 02_eda_plots.ipynb

   • Exploratory data analysis
   • Distribution plots
   • Correlation matrix

3. 03_ml_pipeline.ipynb

   • Feature preprocessing
   • Train/validation/test preparation

4. 04_ml_models.ipynb

   • Logistic Regression baseline
   • Default XGBoost
   • Hyperparameter tuning
   • Final model evaluation

5. 05_shap_analysis.ipynb

   • SHAP summary plots
   • Feature importance
   • Dependence plots

6. 06_learning_curve.ipynb

   • Learning curve analysis
   • Training time analysis
   • Early stopping analysis

---

Model Generation

The trained model files (.pkl) are not included in this repository because they exceed GitHub's file size limits.

To generate the models, run the notebooks in the following order:

1. 01_data_analysis.ipynb
2. 02_eda_plots.ipynb
3. 03_ml_pipeline.ipynb
4. 04_ml_models.ipynb

The 04_ml_models.ipynb notebook trains and evaluates:

• Logistic Regression (21 low-level features)
• Logistic Regression (7 high-level features)
• Logistic Regression (28 features)
• Default XGBoost (21 low-level features)
• Default XGBoost (7 high-level features)
• Default XGBoost (28 features)
• Tuned XGBoost (21 low-level features)
• Tuned XGBoost (7 high-level features)
• Tuned XGBoost (28 features)

If desired, the trained models can be saved locally using joblib:

import joblib

joblib.dump(final_xgb_28, "models/final_xgb_28_tuned.pkl")

The generated model files will be saved to your local machine and are not required to reproduce the experiments.

Reproducing the Results

To reproduce the results reported in this repository:

• Download the HIGGS dataset.
• Place it in data/raw/.
• Install the dependencies listed in requirements.txt.
• Run the notebooks sequentially in the order listed above.

All reported figures, tables, SHAP visualizations, learning curves, and evaluation metrics can be reproduced using the provided notebooks.

---

References

Baldi, P., Sadowski, P., & Whiteson, D. (2014). Searching for Exotic Particles in High-Energy Physics with Deep Learning.

Nature Communications, 5, 4308.

UCI Machine Learning Repository - HIGGS Dataset