LLM-Nexus: Smart LLM Router

A machine learning system that classifies incoming LLM prompts into complexity tiers and routes each one to the cheapest model capable of handling it well — targeting 40–50% inference cost reduction vs always using GPT-4.

The Problem

Running every prompt through an expensive frontier model (GPT-4, Claude Opus) wastes money. Simple factual questions don't need the same model as a complex reasoning task. The goal: train a router that predicts the minimum tier a prompt actually needs, using real human preference data to define "good enough."

Solution Overview

Incoming Prompt
      │
      ▼
┌─────────────┐     Tier 0 (Easy)   → GPT-3.5 / Claude Haiku   ($0.50/1M)
│ LLM Router  │──►  Tier 1 (Medium) → Claude Sonnet             ($3.00/1M)
└─────────────┘     Tier 2 (Hard)   → GPT-4 / Claude Opus       ($15.00/1M)

Labels are derived from real human preference votes in the LMSYS Chatbot Arena — not synthetic heuristics.

Notebooks

Notebook 1 — Data Pipeline & Feature Engineering

• Loads the LMSYS Chatbot Arena dataset (~55k head-to-head model comparisons)
• Maps 30+ models to 3 tiers based on parameter count and capability
• Derives routing labels from human votes: if a small model won or tied, the prompt is "easy"
• Engineers 12 prompt complexity features across three categories:

Category	Features
Lexical	Token count, unique token ratio, avg word length, Flesch reading ease
Syntactic	Sentence count, max dependency depth, question count, code block flag
Semantic	Reasoning marker score, simplicity marker score, instruction step count, math flag

Outputs: lmsys_labeled_dataset.csv, router_features_labels.csv, router_prompts_labels.csv

---

Notebook 2 — XGBoost Baseline

Trains an XGBoost classifier on the 12 hand-crafted features. Five hyperparameter configurations are tracked with MLflow.

Config	n_estimators	max_depth	lr	Val F1
1	100	4	0.10	0.288
2	200	6	0.10	0.326
3	200	6	0.05	0.304
4	300	8	0.05	0.329
5	300	8	0.01	0.292

Test Results (best config):

Metric	Score
Accuracy	0.4030
Weighted F1	0.3237
Hard tier F1	0.05 ⚠️

The baseline almost never routes to Tier 3 (Hard recall = 3%) — a critical failure mode in production.

---

Notebook 3 — DistilBERT + LoRA Fine-Tuning

Fine-tunes distilbert-base-uncased on raw prompt text using LoRA adapters (only 1.31% of parameters are trainable). Two runs were conducted:

	Run 1	Run 2
MAX_LEN	128	256
Class weights	Yes	No
Best epoch	9	7
Test Accuracy	0.3458	0.4122
Test Macro F1	0.3383	0.2695
Hard tier F1	0.28	0.04

LoRA config: r=16, alpha=32, dropout=0.1, targets=q_lin + v_lin

Run 1 achieves 5× better Hard-tier detection than XGBoost (F1: 0.28 vs 0.05).

---

Notebook 4 — Ablation Study & Cost-Quality Tradeoff

Aggregates all results and surfaces which design decisions actually mattered.

Ablation findings:

Change	Macro F1 Δ	Accuracy Δ
XGBoost → DistilBERT Run1 (add class weights)	+0.058	−0.057
Run1 → Run2 (longer context, remove weights)	−0.069	+0.066

Key insight: Class weighting is more impactful than context length. Removing class weights causes the model to collapse to the majority class (Medium), ignoring Hard prompts entirely.

Cost-quality tradeoff:

Router	Avg Cost / 1M tokens	Cost Savings	Macro F1
No Router (always GPT-4)	$15.00	0%	—
XGBoost	~$4.26	~72%	0.28
DistilBERT Run1	~$5.85	~61%	0.34
DistilBERT Run2	~$3.07	~80%	0.27

Recommendation: DistilBERT Run1 — best macro F1, best Hard-tier coverage, meaningful cost savings.

Results Summary

Model	Accuracy	Macro F1	Hard F1	GPU Needed
XGBoost (baseline)	0.4030	0.28	0.05	No
DistilBERT Run1 ✅	0.3458	0.3383	0.28	Yes
DistilBERT Run2	0.4122	0.2695	0.04	Yes

Stack

• Data: LMSYS Chatbot Arena (Kaggle), 51,734 labeled samples
• Feature engineering: spaCy, textstat, sentence-transformers
• Baseline: XGBoost + MLflow experiment tracking
• Neural model: HuggingFace Transformers (DistilBERT), PEFT/LoRA, PyTorch
• Imbalance handling: SMOTE (imbalanced-learn), class weighting
• Training infra: Kaggle GPU (Tesla T4, 15.6 GB VRAM)
• Evaluation: scikit-learn, matplotlib, seaborn

Getting Started

All notebooks are self-contained and designed to run on Kaggle with GPU enabled.

1. Notebook 1 — Add the LMSYS Chatbot Arena dataset via + Add Data → Search 'lmsys chatbot arena'
2. Notebook 2 — Add the output dataset from Notebook 1 as input
3. Notebook 3 — Add the output dataset from Notebook 1 as input; enable GPU accelerator
4. Notebook 4 — Add outputs from Notebooks 2 and 3 as input

Notebook 1 ──► router_features_labels.csv ──► Notebook 2
           └──► router_prompts_labels.csv  ──► Notebook 3
                                                    │
                      Notebook 2 results ───────────┤
                      Notebook 3 results ───────────┴──► Notebook 4