A fraud risk operations analytics platform built on the synthetic PaySim dataset, combining BigQuery SQL monitoring, Python feature engineering, threshold-optimized machine learning, and an interactive Dash dashboard for analyst review prioritization.
Live dashboard: https://fintech-fraud-risk-ops.onrender.com
Hosted on Render Free tier. The first request after a period of inactivity may take 30–60 seconds due to cold start.
This project simulates a fraud risk operations analytics workflow on the synthetic PaySim mobile-money transaction dataset. It is built as a portfolio demonstration of end-to-end data and ML engineering skills, from raw data ingestion and SQL-based monitoring through leakage-audited machine learning to an operational Dash dashboard.
The platform focuses on analyst-facing decision support rather than automated classification. The central problem is: given a limited analyst review budget, which transactions should be flagged for investigation? The project evaluates rule-based baselines, SQL seed scores, and ML risk scores, then provides an interactive threshold explorer to examine the operational tradeoffs at different review volumes.
All metrics, costs, and savings figures are derived from the synthetic PaySim test split under simplified assumptions. They are not real financial outcomes and the model is not deployed in any production system.
Fraud teams cannot review every transaction. A payment platform processing tens of thousands of transactions per day needs a prioritization system that catches the highest-risk cases while keeping the analyst workload within budget.
This project frames that problem concretely:
- Review budget constraint: Flag at most ~5% of transactions for analyst review.
- Recall objective: Catch as many fraud cases as possible within that constraint.
- Precision context: Understand the false-positive cost so analysts know the expected signal-to-noise ratio.
- Missed fraud exposure: Quantify what slips through below the review threshold.
The project evaluates a rule baseline, a SQL seed rule, and a conservative SGD logistic model, then selects an operating threshold that maximises recall within the review budget.
All values are from the synthetic PaySim test split (1,248,736 transactions) under simplified cost assumptions:
| Metric | Value |
|---|---|
| Test transactions | 1,248,736 |
| Test fraud cases | 4,250 (0.34%) |
| Operating threshold | 0.610 |
| Review queue | 61,384 transactions (4.92%) |
| Recall | 87.25% |
| Precision | 6.04% |
| Fraud cases caught | 3,708 |
| Fraud cases missed | 542 |
| Fraud value caught | ~$6.60B (synthetic, simplified) |
| Fraud value missed | ~$79.5M (synthetic, simplified) |
| Average Precision (AP) | 0.345 |
| ROC-AUC | 0.983 |
Important: Raw model outputs are fraud risk scores used for ranking and analyst review prioritization. They are not calibrated probability estimates. All monetary values are from the synthetic PaySim dataset under simplified cost assumptions ($5/review, $1/FP friction). They do not represent real financial outcomes.
Synthetic PaySim CSV (6.3M rows, 8,213 fraud)
│
├─► BigQuery raw table upload
│ └─► SQL monitoring views (11 SQL files)
│ · Executive KPIs
│ · Transaction type risk
│ · Rule baseline (isFlaggedFraud)
│ · Fraud loss by step
│ · High-value risk
│ · Balance drain patterns
│ · Investigation seed queue
│
├─► Python data quality audit (pandas)
│
├─► Feature engineering pipeline
│ · Time features (step, day, hour)
│ · Amount and balance transformations
│ · Transaction type indicators
│ · Leakage audit → conservative feature set (30 features)
│
├─► Rule baseline evaluation
│ · Existing rule (isFlaggedFraud): recall ~0.19%
│ · SQL seed score (score ≥90): recall ~45.2%
│
├─► ML model training + MLflow tracking
│ · Initial baseline (LightGBM): data leakage detected
│ · Conservative SGD logistic: AP = 0.345, AUC = 0.983
│
├─► Threshold and cost simulation
│ · 103 threshold grid
│ · Operating point: 0.610 (max recall ≤ 5% review)
│
├─► Calibration and explainability audits
│ · Sigmoid calibration explored; raw scores used for ranking
│ · Coefficient + permutation + SHAP importance
│
└─► Dashboard asset CSVs (12 files)
└─► Dash dashboard (app.py)
· 7 tabs · 3 callbacks · 1440×900 responsive layout
See docs/architecture.md for a detailed breakdown.
| Layer | Tools |
|---|---|
| Data storage | Google BigQuery |
| SQL analytics | BigQuery SQL (11 monitoring views) |
| Data processing | Python, pandas, NumPy, PyArrow |
| ML / modeling | scikit-learn (SGD, calibration, metrics), LightGBM |
| Experiment tracking | MLflow |
| Explainability | SHAP, permutation importance |
| Dashboard | Dash 4, Plotly 6 |
| Version control | Git, GitHub |
| Environment | conda, Python 3.11 |
fintech-fraud-risk-ops/
├── app.py # Dash dashboard (loads reports/dashboard_assets/ only)
├── requirements.txt
├── environment.yml
├── src/ # Pipeline scripts (Steps 01–16)
│ ├── 01_bigquery_upload.py
│ ├── 02_data_quality_audit.py
│ ├── 03_bigquery_sql_views.py
│ ├── 04_class_imbalance_analysis.py
│ ├── 05_rule_baseline_evaluation.py
│ ├── 06_feature_engineering.py
│ ├── 07_baseline_ml_models.py
│ ├── 08_ml_leakage_audit.py
│ ├── 09_mlflow_conservative_models.py
│ ├── 10_threshold_cost_simulation.py
│ ├── 11_model_explainability.py
│ ├── 12_calibration_audit.py
│ ├── 13_dashboard_data_assets.py
│ ├── 14_dashboard_validation.py
│ ├── 15d_full_metrics_recalculation_audit.py
│ └── 16_capture_dashboard_screenshots.py
├── sql/ # BigQuery SQL monitoring views (11 files)
├── docs/ # Architecture and methodology documentation
├── reports/
│ ├── dashboard_assets/ # 12 committed CSVs powering the dashboard
│ ├── screenshots/ # 9 dashboard screenshots
│ ├── sql_outputs/ # Validation and audit reports
│ └── figures/ # Analysis charts
├── data/
│ ├── raw/ # Ignored (raw CSV/ZIP not committed)
│ ├── interim/ # Ignored (parquet not committed)
│ └── processed/ # Ignored (parquet not committed)
└── models/ # Ignored (joblib artifacts not committed)
What is not committed: raw CSV/ZIP data, interim/processed parquet files, trained model artifacts (.joblib), MLflow runs, environment secrets (.env), and any credential files.
The project uses Google BigQuery as the analytical SQL layer for raw data loading and fraud monitoring views.
SQL views (sql/):
00_bigquery_validation.sql— raw table validation01_executive_fraud_kpis.sql— aggregate KPIs by step02_transaction_type_fraud_summary.sql— TRANSFER/CASH_OUT/CASH_IN breakdown03_rule_performance_isflagged.sql— existing rule recall and precision04_fraud_loss_by_step.sql— fraud amount timeline05_high_value_transaction_risk.sql— high-value transaction risk profile06_balance_drain_patterns.sql— origin balance depletion signals07_origin_account_risk_summary.sql— account-level origin risk08_destination_account_risk_summary.sql— account-level destination risk09_fraud_monitoring_step_view.sql— step-level monitoring view10_investigation_seed_queue.sql— SQL rule seed score for investigation
Dashboard note: The deployed dashboard reads 12 committed CSV files from reports/dashboard_assets/ rather than querying BigQuery directly. This keeps the portfolio deployment lightweight and credential-free. The CSVs are regenerated from the full pipeline whenever model updates require it.
Baseline evaluation:
The existing isFlaggedFraud rule catches only ~0.19% of fraud at 100% precision. The selected SQL seed operating point was score ≥90, reaching about 45.2% recall before ML-based prioritization.
Initial ML baseline: Initial LightGBM and logistic regression models showed near-perfect AUC, which prompted a leakage audit.
Leakage audit (Step 8):
Post-transaction balance fields (newbalanceOrig, newbalanceDest, balance delta features) create data leakage — they encode fraud outcomes rather than predict them. Models trained with these features are not operationally valid.
Conservative model (Steps 9–10):
A pre-transaction conservative feature set (30 features) was constructed excluding all post-transaction balance fields. An SGD logistic regression model (loss=log_loss, class_weight=balanced) was selected based on ranking performance (AP=0.345, AUC=0.983) and calibration behavior.
Threshold selection: A 103-point threshold grid was evaluated. The operating threshold of 0.610 gives the highest recall within a ≤5% review rate budget:
| Threshold | Review Rate | Recall | Precision |
|---|---|---|---|
| 0.610 | 4.92% | 87.25% | 6.04% |
| 0.500 | 6.13% | 91.11% | 5.06% |
| 0.130 | 13.54% | 99.95% | 2.51% |
Leakage controls:
- Post-transaction fields excluded:
newbalanceOrig,newbalanceDest, balance error terms, balance delta ratios,sql_rule_seed_score,sql_rule_seed_tier. - Feature set verified by cross-referencing Step 8 leakage audit report.
- Model re-trained on 30 pre-transaction features only.
Calibration decision:
- Sigmoid calibration improves ECE (0.072 → 0.003) and Brier score.
- However, it reduces Average Precision (0.345 → 0.283), which hurts ranking performance.
- Dashboard uses raw risk scores for ranking/thresholding. Calibrated estimates are documented separately in Step 12 calibration audit.
The Dash dashboard (app.py) has 7 tabs:
| Tab | Contents |
|---|---|
| Executive Overview | 10-metric KPI grid, Operating Point Comparison chart, Fraud Capture Summary |
| Threshold Explorer | Interactive threshold selector with live KPI update, Threshold Tradeoff Curve |
| Review Queue | Risk Score Distribution (dual-axis), Review Queue Volume chart, Detailed Queue Metrics table |
| Risk Segments | Score Band Segments chart, Transaction Type Risk Profile, Breakdown table |
| Explainability | Coefficient Importance (top 15), Permutation Importance (top 15) |
| Investigation Queue | 500-row analyst case sample with type/tier filters, formatted USD amounts, risk scores |
| Caveats & Docs | Model caveats, score interpretation, synthetic data caveat, Data Dictionary |
- Anaconda or Miniconda
- Python 3.11
git clone https://github.com/PrajwalShekar22/fintech-fraud-risk-ops.git
cd fintech-fraud-risk-ops
conda env create -f environment.yml
conda activate fintech-fraud-risk-ops
python app.pyOpen in browser: http://127.0.0.1:8050
The dashboard runs entirely from committed CSV files in reports/dashboard_assets/. No raw data, model artifacts, or BigQuery credentials are required to view the dashboard.
The full pipeline (Steps 1–13) requires:
- The synthetic PaySim dataset (available on Kaggle:
ntnu-cybersecurity/paysim1) - A Google Cloud project with BigQuery enabled
- A service account key file (not committed)
- A configured
GOOGLE_APPLICATION_CREDENTIALSenvironment variable
The project includes multi-layer validation checks:
- Dashboard validation (
src/14_dashboard_validation.py): 105-check script covering score wording, forbidden references, queue quality, syntax, attribution, and git safety. - Full metrics recalculation audit (
src/15d_full_metrics_recalculation_audit.py): Reloads saved model and data, recalculates all threshold metrics from scratch, compares against all dashboard assets. Confirmed review_count = 61,384 as the authoritative value. - Screenshot QA (
src/16_capture_dashboard_screenshots.py): Playwright-based automated screenshot capture with visual QA. - Token and credential scans: Grep-based checks across all source files before each push.
- Unsafe tracked file checks:
git ls-filesverified before every push.
- Synthetic dataset: All data is from the PaySim synthetic mobile-money simulator. Results are not applicable to real bank or payment system performance.
- Not production-validated: This model has not been validated against real transaction data, regulatory requirements, or production system constraints.
- Simplified cost model: Review cost ($5/transaction) and FP friction ($1/FP) are illustrative. Actual analyst costs and fraud recovery rates vary significantly.
- Score interpretation: Raw model outputs are risk-ranking scores, not calibrated probability estimates. The dashboard explicitly labels them as fraud risk scores (not calibrated probability estimates).
- BigQuery as analytical layer: BigQuery is used for SQL-based fraud monitoring and seed scoring. The dashboard deployment uses static CSV assets, not live BigQuery queries.
- No fairness/regulatory validation: No analysis of demographic, geographic, or other protected-class disparate impact has been performed.
- dbt pipeline: Refactor SQL views into a dbt project for lineage tracking, tests, and documentation.
- CI/CD: Add GitHub Actions for automated dashboard validation on every push.
- Cloud deployment: Deploy dashboard to Render or Cloud Run with automated CSV refresh.
- Drift monitoring: Add score distribution and feature drift monitoring for production-like tracking.
- Scheduled refresh: Automate dashboard asset CSV regeneration on a schedule.
- Analyst workflow: Add role-based investigation queue management with case status tracking.
- Real data adaptation: Adapt pipeline for real-world transaction data where legally and ethically available.
Prajwal Gorkhar Chandrashekar
Portfolio project using the synthetic PaySim dataset. All metrics, cost figures, and outcomes are simulated under simplified assumptions and do not represent real financial data, real bank operations, or production system performance.