This study provides empirical evidence for task-aware model selection, showing that architectural design choices — not just parameter count — determine performance across NLP tasks. Our results contradicted several baseline assumptions: ALBERT, despite having only 12M parameters, outperformed larger models on QA, while RoBERTa's 125M parameters proved advantageous for NLI. These findings suggest a practical framework for model selection based on task characteristics rather than model size alone.
Comparative study of 5 transformer architectures (BERT, DistilBERT, RoBERTa, XLNet, ALBERT) fine-tuned across 3 NLP benchmark tasks.
| Model | NER (ACC) | NLI (ACC) | QA (EM) |
|---|---|---|---|
| BERT | 98.57% | 82.42% | 78.70 |
| DistilBERT | 98.58% | 80.68% | 72.89 |
| RoBERTa | 99.07% | 87.40% | 78.34 |
| XLNet | 99.11% | 86.02% | 80.52 |
| ALBERT | 98.46% | 83.50% | 83.28 |
- XLNet achieved best NER accuracy (99.11%) and competitive NLI performance
- ALBERT outperformed all models on QA (83.28 EM) despite having only 12M parameters
- Higher parameter count favors NLI tasks; embedding factorization benefits QA tasks
| Task | Dataset | Metric |
|---|---|---|
| NER | CoNLL-2003 | ACC, F1 |
| NLI | GLUE MNLI | ACC |
| QA | SQuAD | EM, F1 |
notebooks/ ├── ner.ipynb ├── nli.ipynb └── qa.ipynb
Python, PyTorch, HuggingFace Transformers, HuggingFace Datasets