ECE1508 — Automatic Speech Recognition

ECE1508 · University of Toronto Jiarong Edwin Chen · Trung-Lam Nguyen · Anubhav Sharma

---

End-to-end ASR system that converts speech to text using a family of DeepSpeech2-inspired architectures trained with CTC loss on the LJSpeech dataset. Four temporal network variants are compared head-to-head — unidirectional GRU, bidirectional GRU, LSTM, and Conformer — each trained across three random seeds, evaluated by Character Error Rate (CER) and Word Error Rate (WER), and decoded with greedy CTC, beam search, and beam search + KenLM 3-gram language model rescoring.

---

Repository Structure

ECE1508/
├── Code/
│   ├── main.ipynb                  # Training pipeline (run this first)
│   ├── analyse.ipynb               # Cross-architecture analysis & plots
│   ├── builders.py                 # Model factory functions (one per architecture)
│   ├── config.py                   # Global constants (audio params, paths, tokenizer)
│   ├── deep_speech_2.py            # GRU model (uni- and bidirectional via flag)
│   ├── deep_speech_2_lstm.py       # LSTM model
│   ├── conformer.py                # Conformer (CNN + encoder) model
│   ├── decoder.py                  # Greedy & beam CTC decoders (+ KenLM)
│   ├── data_loader.py              # Dataset download helpers
│   ├── ljspeech.py                 # LJSpeech dataset class
│   ├── utils/                      # Utility package
│   │   ├── __init__.py             # Re-exports all public symbols
│   │   ├── analysis.py             # Result/Stats dataclasses, load_results, compute_stats, visualise_stats
│   │   ├── checkpointing.py        # save_model, load_model, save_history, load_h5_struct
│   │   ├── data.py                 # build_dataloaders, collate functions
│   │   ├── lm.py                   # KenLM decoder helpers
│   │   ├── multi_seed.py           # run_multi_seed_experiment
│   │   ├── training.py             # train, test, ctc_greedy_decode
│   │   └── visualization.py        # Plotting helpers (loss, CER, WER curves)
│   └── requirements.txt
├── models/
│   └── seed_runs/                  # Per-architecture, per-seed checkpoints
│       └── <arch_name>/            # One directory per trained architecture, e.g. gru_bidirectional_3
│           ├── seed_1508/  seed_2603/  seed_9102/
│           │   ├── model.pth           # Last-epoch checkpoint
│           │   ├── model_best.pth      # Best val-WER checkpoint
│           │   └── history.h5          # Per-epoch train/val loss, CER, WER, time
│           ├── seed_results.csv        # Per-seed test metrics
│           └── seed_summary.json       # Mean ± std across seeds
├── Proposal/                       # Project proposal (LaTeX)
└── Report/                         # Final report (LaTeX)

---

Setup

cd Code
pip install -r requirements.txt

kenlm is installed automatically by the first cell in main.ipynb via a pre-built wheel. No manual build needed.

---

Notebook 1 — Code/main.ipynb · Training Pipeline

This notebook runs the full pipeline: data download → dataset loading → multi-seed training → test evaluation → LM decoding comparison. Outputs (checkpoints, history, CSV/JSON summaries) are written to models/seed_runs/.

Step-by-step

1. Architecture flags — set before running anything else

Set these flags in the configuration cell to control which architectures are trained. All default to False except RUN_TRANSFORMER:

Flag	Default	Architecture
RUN_GRU	False	Unidirectional GRU, depth 3, hidden 512, no look-ahead
RUN_GRU_LOOKAHEAD	False	Unidirectional GRU, depth 3, with look-ahead conv (context=40)
RUN_GRU_BIDIRECTIONAL	False	Bidirectional GRU, depth 3, hidden 512
RUN_LSTM	False	Unidirectional LSTM, depth 3, hidden 512
RUN_LSTM_LOOKAHEAD	False	Bidirectional LSTM, depth 3, with look-ahead conv (context=40)
RUN_TRANSFORMER	True	Conformer (CNN + encoder)

Other config values come from config.py: DEFAULT_SEEDS = [1508, 2603, 9102], TRAIN_NUM_EPOCHS = 20.

2. Download cell

Auto-downloads LJSpeech-1.1 into ../data/, plus the KenLM 3-gram ARPA file and LibriSpeech phoneme lexicon into ../data/lm/. Safe to re-run — skips files that already exist.

3. Dataset + DataLoader cell

Loads LJSpeech with mel-spectrogram caching, plots the first sample for a sanity-check, then splits 80% train / 10% val / 10% test (fixed seed 1508). Builds train_loader, val_loader, test_loader. Infers section_in_channels = 1 and section_in_feat_dim = 80 (mel bins) for use by all model builders.

4. Import builders

Imports all six factory functions from builders.py: build_gru_model, build_gru_look_ahead_model, build_gru_bidirectional_model, build_lstm_model, build_lstm_look_ahead_model, build_transformer_model.

5. Architecture section cells

Each section cell calls _run_architecture(arch_name, builder_fn), which runs run_multi_seed_experiment() across three seeds and saves results to ../models/seed_runs/<arch_name>/.

Section 1: GRU — three sub-variants:

if RUN_GRU:               _run_architecture("GRU_NO_LOOKAHEAD_3", build_gru_model)
if RUN_GRU_LOOKAHEAD:     _run_architecture("GRU_LOOKAHEAD",       build_gru_look_ahead_model)
if RUN_GRU_BIDIRECTIONAL: _run_architecture("GRU_BIDIRECTIONAL",   build_gru_bidirectional_model)

Section 2: LSTM — two sub-variants:

if RUN_LSTM:              _run_architecture("LSTM_NO_LOOKAHEAD_3", build_lstm_model)
if RUN_LSTM_LOOKAHEAD:    _run_architecture("LSTM_LOOKAHEAD_3",    build_lstm_look_ahead_model)

Section 3: Conformer:

if RUN_TRANSFORMER:       _run_architecture("TRANSFORMER",         build_transformer_model)

6. Select active run

SELECT_ACTIVE_ARCH = "TRANSFORMER"

Calls set_active_run(arch_name) to set the global model, history, and active_loss_fn used by the downstream eval cells. Valid values: "GRU_NO_LOOKAHEAD_3", "GRU_LOOKAHEAD", "GRU_BIDIRECTIONAL", "LSTM_NO_LOOKAHEAD_3", "LSTM_LOOKAHEAD_3", "TRANSFORMER".

7. Test evaluation + history plots

Runs utils.test() on the test set and prints test_loss, test_CER, test_WER. Plots training loss, CER, and WER curves (train vs. val) for the active run.

8. LM decoder setup + sweep

Builds two decoders — beam search (no LM) and beam search + KenLM 3-gram LM. RUN_LM_SWEEP = True runs a grid search over lm_weight × word_score on the validation set and rebuilds the LM decoder with the best-found params.

9. Qualitative decode comparison

Decodes one test sample three ways and prints side-by-side:

Ground truth         : PRINTING IN THE ONLY SENSE...
Greedy               : PRITING IN THE ONLE SENCE...
Beam (no LM)         : PRINTING IN THE ONLY SENSE...
Beam + LibriSpeech LM: PRINTING IN THE ONLY SENSE WITH...

---

Notebook 2 — Code/analyse.ipynb · Cross-Architecture Analysis

Loads all completed seed runs from disk, computes per-metric mean ± std across seeds, and plots overlaid training curves for each architecture.

Step-by-step

1. Copy seed runs into the notebook's working directory

# Cell already present in the notebook — just run it:
!cp -R ../models/seed_runs/ ./

2. Run remaining cells in order

Cells call utils.load_results(), utils.compute_stats(), and utils.visualise_stats() to produce shaded-band plots for train/val loss, CER, and WER across all architectures in seed_runs/.

---

Using Pre-trained Checkpoints

Model checkpoints (.pth) and training histories (.h5) are hosted on Google Drive due to file size.

Download: seed_runs (Google Drive)

After downloading, place the seed_runs/ folder at models/seed_runs/ so the directory structure matches the layout in the Repository Structure section above.

To use the checkpoints without retraining:

• To analyse: place seed_runs/ as above, then run analyse.ipynb directly (step 1 copies the checkpoints into the notebook's working directory).
• To evaluate a specific seed: skip all training cells in main.ipynb, load the checkpoint manually (example using gru_bidirectional_3):

  from builders import build_gru_bidirectional_model
  import utils
  
  model = build_gru_bidirectional_model(in_channels, in_feat_dim)
  optimizer = torch.optim.AdamW(model.parameters(), lr=LEARNING_RATE)
  utils.load_model(model, device, "../models/seed_runs/<arch_name>/seed_1508/model_best.pth", optimizer)
  history = utils.load_h5_struct("../models/seed_runs/<arch_name>/seed_1508/history.h5")

---

Default Training Configuration

Setting	Value
Dataset	LJSpeech (13,100 utterances)
Split	80% train / 10% val / 10% test
Seeds	1508, 2603, 9102
Epochs	20 per seed
Batch size	32
Optimizer	AdamW
Scheduler	OneCycleLR (peak LR = 3e-4, 5% warmup, cosine anneal)
Precision	AMP (fp16 on CUDA, fp32 on CPU)
Audio features	80-bin log-mel spectrogram (16 kHz, n_fft=512, hop=256)
CTC blank	[PAD] token from facebook/wav2vec2-base tokenizer
LM	KenLM 3-gram (LibriSpeech, pruned 1e-7)