classic_gmm_vr.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c29f761d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import librosa\n",
    "\n",
    "from collections import namedtuple\n",
    "from sklearn.mixture import GaussianMixture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fb66178",
   "metadata": {},
   "source": [
    "# 0. Very brief introduction\n",
    "\n",
    "In this notebook a classic GMM voice recognition method is presented."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c46858f",
   "metadata": {},
   "source": [
    "# 1. Definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e833d061",
   "metadata": {},
   "source": [
    "## 1.1 Convenient datatypes\n",
    "\n",
    "For convenience we use 3 datatypes defined below, as all the objects have to be labeled in order to make learning and testing convenient.\n",
    "They can be essentially replaced by just a single named tuple like\n",
    "\n",
    "```python\n",
    "LabeledObject = namedtuple('LabeledObject', ['label', 'object'])\n",
    "```\n",
    "\n",
    "but I found more verbose notation to be more intuitive when reading code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "610e8d73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# (Label, Corresponding model).\n",
    "LabeledModel = namedtuple('LabeledModel', ['label', 'model'])\n",
    "\n",
    "# (Label, Path to files of that label).\n",
    "LabeledPaths = namedtuple('LabeledPaths', ['label', 'paths'])\n",
    "\n",
    "# (Label, scoring resut for given label).\n",
    "LabeledResult = namedtuple('LabeledResult', ['label', 'score'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5014e1",
   "metadata": {},
   "source": [
    "## 1.2 Paths file parsing\n",
    "\n",
    "This function parses given file of form\n",
    "\n",
    "```\n",
    "label1-metadata1/subfolder1/sample-01.wav\n",
    "label1-metadata2/subfolder2/sample-02.wav\n",
    "label1-metadata3/subfolder3/sample-03.wav\n",
    "\n",
    "label2-metadata1/subfolder1/sample-01.wav\n",
    "label2-metadata2/subfolder2/sample-02.wav\n",
    "label2-metadata3/subfolder3/sample-03.wav\n",
    "\n",
    "...\n",
    "```\n",
    "\n",
    "into a list of LabeledPaths.\n",
    "Blank lines act as separators between differently-labeled models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e879f8fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data_paths(paths_file: str, prefix: str='',\n",
    "                   name_stop_symb: str='-') -> list[LabeledPaths]:\n",
    "    \"\"\"Construct a list of LabeledPaths (label, filepaths) from a given file\n",
    "    containing filepaths separated by whitespaces.\n",
    "    \"\"\"\n",
    "    \n",
    "    # To be a list of LabeledPaths.\n",
    "    training_set = []\n",
    "    with open(paths_file) as file:\n",
    "        \n",
    "        # After first read get label and first path.\n",
    "        while line := file.readline().rstrip():\n",
    "                \n",
    "            # Set part of the path before `name_stop_symb' as label.\n",
    "            label = line.split(name_stop_symb)[0]\n",
    "            ex = LabeledPaths(label, [os.path.join(prefix, line)])\n",
    "\n",
    "            # Continue to read paths until there's no left.\n",
    "            while line := file.readline().rstrip():\n",
    "                ex.paths.append(os.path.join(prefix, line))\n",
    "\n",
    "            training_set.append(ex)\n",
    "    \n",
    "    return training_set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39245470",
   "metadata": {},
   "source": [
    "## 1.3 Feature extraction\n",
    "\n",
    "For feature extraction we use very convenient Librosa library providing one-liners for extraction of MFCCs and their any-order differences.\n",
    "Before extraction of features for each audio we first normalize it to bring all the clips to the same level.\n",
    "\n",
    "`get_features_path` extracts features from files given, concatenates and returns them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e6e05e9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features(utterance: np.ndarray, sr: int, n_mfccs: int=20):\n",
    "    \"\"\"Get features from a given librosa-returned array. Returns n_mfccs * 3\n",
    "    features: MFCC, their 1st and 2nd differences concatenated horizontally.\n",
    "    \"\"\"\n",
    "    \n",
    "    mfccs  = librosa.feature.mfcc(utterance, sr, n_mfcc=n_mfccs)\n",
    "    mfccs1 = librosa.feature.delta(mfccs)\n",
    "    mfccs2 = librosa.feature.delta(mfccs, order=2)\n",
    "    \n",
    "    # Librosa returns features in shape (n_mfcc, ...). We want mfccs to\n",
    "    # be \"columns\" in our dataset => we transpose them.\n",
    "    features = np.hstack((mfccs.T, mfccs1.T, mfccs2.T))\n",
    "    return features\n",
    "\n",
    "def get_features_path(filepaths: list[str], n_mfccs: int=20) -> np.ndarray:\n",
    "    \"\"\"Gets features from given files. Uses get_features for feature-extraction.\n",
    "    \"\"\"\n",
    "    \n",
    "    features = np.empty((0, n_mfccs * 3))\n",
    "    for path in filepaths:\n",
    "        utt, sr = librosa.load(path)\n",
    "        utt = librosa.util.normalize(utt)\n",
    "        features_utt = get_features(utt, sr, n_mfccs=n_mfccs)\n",
    "        \n",
    "        features = np.vstack((features, features_utt))\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c3802fe",
   "metadata": {},
   "source": [
    "## 1.4 Model construction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e5e085e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_models(examples: list[LabeledPaths]) -> list[LabeledModel]:\n",
    "    \"\"\"Create and train models for a given set of LabeledPaths (label, filepaths)\n",
    "    created via get_data_paths function. Returns labeled models of type\n",
    "    (label, model). Labels are preserved from corresponding LabeledPaths.\n",
    "    \"\"\"\n",
    "    \n",
    "    # To be a list of labeled models.\n",
    "    labeled_models = []\n",
    "    for example in examples:\n",
    "        \n",
    "        # Get features from files.\n",
    "        features = get_features_path(example.paths)\n",
    "        \n",
    "        # Create and train GMM.\n",
    "        gmm = GaussianMixture(n_components=16,\n",
    "                              covariance_type='diag',\n",
    "                              n_init=3,\n",
    "                              max_iter=200)\n",
    "        gmm.fit(features)\n",
    "        \n",
    "        # Add generated model to the list.\n",
    "        lm = LabeledModel(example.label, gmm)\n",
    "        labeled_models.append(lm)\n",
    "\n",
    "    return labeled_models\n",
    "\n",
    "def test_models(test_examples:  list[LabeledPaths],\n",
    "                labeled_models: list[LabeledModel]) -> float:\n",
    "    \"\"\"Test models and return accuracy. Takes list of LabeledPaths (label, filepaths)\n",
    "    and tests labeled models (label, model) against these examples.\n",
    "    \"\"\"\n",
    "    \n",
    "    # To be a list of results: Trues and Falses.\n",
    "    results = []\n",
    "    for test_example in test_examples:\n",
    "        \n",
    "        # Extract features for given speakers' filepaths.\n",
    "        features = get_features_path(test_example.paths)\n",
    "        \n",
    "        # Get results for given speaker on all models in labeled_models.\n",
    "        labeled_results = []\n",
    "        for labeled_model in labeled_models:\n",
    "            res = LabeledResult(labeled_model.label,\n",
    "                                labeled_model.model.score(features))\n",
    "            labeled_results.append(res)\n",
    "        \n",
    "        # Get index of a highest score and make corresponding label a prediction.\n",
    "        scores_only = np.array([lr.score for lr in labeled_results], dtype=float)\n",
    "        index = np.argmax(scores_only)\n",
    "        label_pred = labeled_results[index].label\n",
    "        \n",
    "        # Check if prediction if correct and store result in results.\n",
    "        results.append(label_pred == test_example.label)\n",
    "    \n",
    "    return sum(results) / len(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c9b5335",
   "metadata": {},
   "source": [
    "# 2. Training\n",
    "\n",
    "Dataset is taken from [here](https://appliedmachinelearning.blog/2017/11/14/spoken-speaker-identification-based-on-gaussian-mixture-models-python-implementation/) contains 10 utterances from 34 speakers.\n",
    "Training and testing sets both use 5 utterances for each speaker.\n",
    "\n",
    "My folder setup looks like this:\n",
    "\n",
    "```\n",
    ".\n",
    "├── classic_gmm_vr.ipynb\n",
    "└── dataset_vr\n",
    "    ├── label1-metadata1/wav/sample-01.wav\n",
    "    ├── label1-metadata1/wav/sample-02.wav\n",
    "    │   ...\n",
    "    ├── label1-metadata1/wav/sample-10.wav\n",
    "    │\n",
    "    │   .\n",
    "    │   .\n",
    "    │   .\n",
    "    │\n",
    "    ├── label34-metadata1/wav/sample-01.wav\n",
    "    ├── label34-metadata1/wav/sample-02.wav\n",
    "    │   ...\n",
    "    └── label34-metadata1/wav/sample-10.wav \n",
    "```\n",
    "\n",
    "which I didn't change from original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7018bd1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dir = './dataset_vr'\n",
    "train_paths_file = './train_data_paths.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1b4edb63",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "training_set = get_data_paths(train_paths_file, prefix=train_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1511aa21",
   "metadata": {},
   "outputs": [],
   "source": [
    "models = get_models(training_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f6ba1b",
   "metadata": {},
   "source": [
    "# 3. Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d09f0589",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_paths_file = './test_data_paths.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fc36572e",
   "metadata": {},
   "outputs": [],
   "source": [
    "testing_set = get_data_paths(test_paths_file, prefix=train_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c17585c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_models(testing_set, models)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2563f799",
   "metadata": {},
   "source": [
    "# 4. Very brief result discussion\n",
    "\n",
    "All the data was apparently recorded under identical conditions so it's no surprize we've got accuracy of 1.0."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}