FedSpeech: Federated Learning for Speech Command Recognition

A comprehensive implementation of Federated Learning (FL) algorithms for speech command recognition using the Google Speech Commands dataset. This project compares multiple FL approaches including FedAvg, pFedAvg, pFedAvg V2, and FedProx under heterogeneous data distributions.

Overview

This project implements and compares several federated learning algorithms for training speech command recognition models across distributed clients with non-IID (non-identically distributed) data. The system simulates 10 edge devices (clients) with heterogeneous data distributions to study the effectiveness of personalized federated learning approaches.

Key Features

• Multiple FL Algorithms: FedAvg, Personalized FedAvg (pFedAvg), pFedAvg V2, and FedProx
• Speech Command Recognition: Classifies directional commands (go, left, right, stop)
• Non-IID Data Simulation: Realistic heterogeneous data distribution across clients
• CNN-based Model: Convolutional neural network for MFCC feature classification
• Comprehensive Evaluation: Per-client accuracy, loss tracking, confusion matrices, and statistical tests

Model Architecture

Input (13 x 32 x 1) - MFCC Features | Conv2D (32 filters, 3x3) + ReLU + MaxPool | Conv2D (64 filters, 3x3) + ReLU + MaxPool | Flatten -> Dense (128) + ReLU + Dropout (0.5) | Dense (4) + Softmax -> Output

Client Distribution

Client Group	Clients	Data Characteristics
Balanced	0-2	All commands with varying noise levels
Left/Right Heavy	3-4	More left/right, fewer go/stop commands
Go/Stop Heavy	5-7	More go/stop, fewer left/right commands
Different Speakers	8-9	Different speaker distribution

Getting Started

Prerequisites

pip install tensorflow numpy librosa matplotlib seaborn pandas scikit-learn networkx scipy kagglehub

Dataset

The project uses the Google Speech Commands Dataset:

import kagglehub
kagglehub.dataset_download('neehakurelli/google-speech-commands')

Or download manually and place in ./data/speech_commands/

Running the Experiment

1. Open the notebook in Jupyter or VS Code
2. Run all cells sequentially
3. Configure experiment parameters:

results = run_experiment(
    DATA_DIR,
    n_clients=10,
    standalone_epochs=25,
    fl_rounds=20,
    local_epochs=5,
    lambda_reg=0.1,      # pFedAvg regularization
    lambda_reg_v2=0.1,   # pFedAvg V2 regularization
    mu=0.01              # FedProx proximal term
)

Algorithms Implemented

1. Standalone (Baseline)

Each client trains independently without any collaboration.

2. FedAvg (Federated Averaging)

Standard federated learning with global model averaging.

3. pFedAvg (Personalized FedAvg)

Graph-based personalization using neighbor aggregation with edge weights based on data similarity.

4. pFedAvg V2 (Global Interpolation)

Personalization through global-local interpolation controlled by lambda parameter.

5. FedProx

Proximal regularization for heterogeneous settings to prevent client drift.

Outputs

The notebook generates:

• Accuracy Comparison: Per-client test accuracy bar charts
• Training Progress: Accuracy and loss curves over communication rounds
• Confusion Matrices: Per-client classification performance
• F1 Score Heatmaps: Per-command performance across clients
• Statistical Tests: Paired t-tests for algorithm comparison
• CSV Export: Detailed results in fl_detailed_results_comparison.csv

Configuration

Audio Processing Parameters

Parameter	Value	Description
SAMPLE_RATE	16000 Hz	Audio sample rate
DURATION	1.0 s	Audio clip duration
N_MFCC	13	Number of MFCC coefficients
N_FFT	2048	FFT window size
HOP_LENGTH	512	STFT hop length

Training Parameters

Parameter	Default	Description
n_clients	10	Number of simulated clients
fl_rounds	20	Communication rounds
local_epochs	5	Local training epochs per round
standalone_epochs	25	Standalone baseline epochs

Project Structure

fl-improvement.ipynb # Main notebook with all implementations README.md # This file Project_report.pdf # Detailed project report Response_letter.pdf # Response letter Results/ # Output results directory .gitignore

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

---

Note: This project was developed for educational and research purposes in the field of Federated Learning and Edge AI.