audio/audio_classification/music_genre_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c186f132",
   "metadata": {},
   "source": [
    "# Recognize Music Genre using Embeddings\n",
    "\n",
    "\n",
    "## Scenario Introduction\n",
    "\n",
    "A **music genre classification system** automatically identifies a piece of music by matching a short snippet against a database of known music. Compared to the traditional methods using frequency domain analysis, the use of embedding vectors generated by 1D convolutional neural networks improves recall and can, in some cases, improve query speed.\n",
    "\n",
    "A music genre classification system generally transforms audio data to embeddings and compares similarity based on distances between embeddings. Therefore, an encoder converting audio to embedding and a database for vector storage and retrieval are main components.\n",
    "\n",
    "## Tutorial Overview\n",
    "\n",
    "Normally an audio embedding pipeline generates a set of embeddings given an audio path, which composes a unique fingerprint representing the input music. Each embedding corresponds to features extracted for a snippet of the input audio. By comparing embeddings of audio snippets, the system can determine the similarity between audios. The image below explains the music fingerprinting by audio embeddings.\n",
    "\n",
    "<img src=\"./music_embedding.png\" width = \"60%\" height = \"60%\" align=center />\n",
    "\n",
    "\n",
    "A block diagram for a basic music genre classification system is shown in images below. The first image illustrates how the system transforms a music dataset to vectors with [Towhee](https://github.com/towhee-io/towhee) and then inserts all vectors into [Milvus](https://github.com/milvus-io/milvus). The second image shows the querying process of an unknown music snippet.\n",
    "\n",
    "<img src=\"./music_recog_system.png\"  width = \"60%\" height = \"60%\" align=center />\n",
    "\n",
    "Building a music genre classification system typically involves the following steps:\n",
    "\n",
    "1. Model and pipeline selection\n",
    "2. Computing embeddings for the existing music dataset\n",
    "3. Insert all generated vectors into a vector database\n",
    "4. Identify an unknown music snippet by similarity search of vectors\n",
    "\n",
    "In the upcoming sections, we will first walk you through some of the prep work for this tutorial. After that, we will elaborate on each of the four steps mentioned above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f243b6f6",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "First, we need to install Python packages, download example data, and prepare Milvus.\n",
    "\n",
    "### Install packages\n",
    "Make sure you have installed required python packages with proper versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1cbf2191",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q towhee towhee.models gradio scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc91053",
   "metadata": {},
   "source": [
    "### Download dataset\n",
    "\n",
    "This tutorial uses a subset of [GTZAN](http://marsyas.info/downloads/datasets.html). You can download it via [Github](https://github.com/towhee-io/examples/releases/download/data/gtzan300.zip).\n",
    "\n",
    "The data is organized as follows:\n",
    "\n",
    "- train: candidate music, 10 classes, 30 audio files per class (300 in total)\n",
    "- test: query music clips, same 10 classes as train data, 3 audio files per class (30 in total)\n",
    "- gtzan300.csv: a csv file containing an id, path, and label for each video in train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d787aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "! curl -L https://github.com/towhee-io/examples/releases/download/data/gtzan300.zip -O\n",
    "! unzip -q -o gtzan300.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ce6ce98",
   "metadata": {},
   "source": [
    "Let's take a quick look and prepare a id-label dictionary for future use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9d8209aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>path</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>./train/hiphop/hiphop.00096.wav</td>\n",
       "      <td>hiphop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>./train/hiphop/hiphop.00024.wav</td>\n",
       "      <td>hiphop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>./train/hiphop/hiphop.00015.wav</td>\n",
       "      <td>hiphop</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                             path   label\n",
       "0   0  ./train/hiphop/hiphop.00096.wav  hiphop\n",
       "1   1  ./train/hiphop/hiphop.00024.wav  hiphop\n",
       "2   2  ./train/hiphop/hiphop.00015.wav  hiphop"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./gtzan300.csv')\n",
    "id_label = df.set_index('id')['label'].to_dict()\n",
    "\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c8f1f13",
   "metadata": {},
   "source": [
    "### Start Milvus\n",
    "\n",
    "Before getting started with the system, we also need to prepare Milvus in advance. Milvus is an open-source vector database built to power embedding similarity search and AI applications. More info about Milvus is available [here](https://github.com/milvus-io/milvus).\n",
    "\n",
    "Please make sure that you have started a [Milvus service](https://milvus.io/docs/install_standalone-docker.md). This notebook uses [milvus 2.2.10](https://milvus.io/docs/v2.2.x/install_standalone-docker.md) and [pymilvus 2.2.11](https://milvus.io/docs/release_notes.md#2210)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c395b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m pip install -q pymilvus==2.2.11"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "320dbb43",
   "metadata": {},
   "source": [
    "Here we prepare a function to work with a Milvus collection with the following parameters:\n",
    "\n",
    "- [L2 Distance](https://milvus.io/docs/metric.md#Euclidean-distance-L2)\n",
    "- [IVF-Flat Index](https://milvus.io/docs/index.md#IVF_FLAT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "93066395",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection, utility\n",
    "\n",
    "connections.connect(host='localhost', port='19530')\n",
    "\n",
    "def create_milvus_collection(collection_name, dim):    \n",
    "    if utility.has_collection(collection_name):\n",
    "        utility.drop_collection(collection_name)\n",
    "    \n",
    "    fields = [\n",
    "    FieldSchema(name='id', dtype=DataType.VARCHAR, descrition='ids', max_length=500, is_primary=True, auto_id=False),\n",
    "    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, descrition='embedding vectors', dim=dim)\n",
    "    ]\n",
    "    schema = CollectionSchema(fields=fields, description='audio classification')\n",
    "    collection = Collection(name=collection_name, schema=schema)\n",
    "\n",
    "    # create IVF_FLAT index for collection.\n",
    "    index_params = {\n",
    "        'metric_type':'L2',\n",
    "        'index_type':\"IVF_FLAT\",\n",
    "        'params':{\"nlist\": 400}\n",
    "    }\n",
    "    collection.create_index(field_name=\"embedding\", index_params=index_params)\n",
    "    return collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f82f7b6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = create_milvus_collection('vggish', 128)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9fdb1a",
   "metadata": {},
   "source": [
    "## Build System\n",
    "\n",
    "Now we are ready to build a music genre classification system. We will select models, generate & save embeddings, and then perform a query example.\n",
    "\n",
    "### 1. Model and pipeline selection\n",
    "\n",
    "The first step in building a music genre classification system is selecting an appropriate embedding model and one of its associated pipelines. Within Towhee, all pipelines can be found on the [Towhee hub](https://towhee.io/tasks/operator). Clicking on any of the categories will list available operators based on the specified task; selecting `audio-embedding` will reveal all audio embedding operators that Towhee offers. We also provide an another option with summary of popular audio embedding pipelines [here](https://docs.towhee.io/pipelines/audio-embedding). \n",
    "\n",
    "Resource requirements, accuracy, inference latency are key trade-offs when selecting a proper pipeline. Towhee provides a multitude of pipelines to meet various application demands. For demonstration purposes, we will use [vggish](https://towhee.io/audio-embedding/vggish)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4992e71",
   "metadata": {},
   "source": [
    "### 2. Generating embeddings for the existing music dataset\n",
    "\n",
    "With optimal operators selected, generating audio embeddings over our music dataset is the next step. Each audio path will go though the pipeline and then output a set of vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "38dc6eb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/chizzy/opt/anaconda3/envs/torch19/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from towhee import pipe, ops\n",
    "from towhee.datacollection import DataCollection\n",
    "from csv import reader\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "# Please note the first time run will take time to download model and other files.\n",
    "\n",
    "emb_pipe = (\n",
    "    pipe.input('id', 'path', 'label')\n",
    "        .map('path', 'frames', ops.audio_decode.ffmpeg())\n",
    "        .map('frames', 'frames', lambda x: [i[0] for i in x])\n",
    "        .flat_map('frames', 'vec', ops.audio_embedding.vggish())\n",
    "        .map('vec', 'vec', ops.towhee.np_normalize())\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc4e677",
   "metadata": {},
   "source": [
    "- `ops.audio_decode.ffmpeg()`: an embeded Towhee operator reading audio as frames [learn more](https://towhee.io/audio-decode/ffmpeg)\n",
    "- `ops.audio_embedding.vggish()`: an embeded Towhee operator applying pretrained VGGish to audio frames, which can be used to generate video embedding [learn more](https://towhee.io/audio-embedding/vggish)\n",
    "- `ops.towhee.np_normalize()`: normalize the embedding results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a6f7faa",
   "metadata": {},
   "source": [
    "### 3. Insert all generated embedding vectors into a vector database\n",
    "\n",
    "While brute-force computation of distances between queries and all audio vectors is perfectly fine for small datasets, scaling to billions of music dataset items requires a production-grade vector database that utilizes a search index to greatly speed up the query process. Here, we'll insert vectors computed in the previous section into a Milvus collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "134de9c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "objc[17868]: Class AVFFrameReceiver is implemented in both /Users/chizzy/opt/anaconda3/envs/torch19/lib/python3.8/site-packages/av/.dylibs/libavdevice.59.7.100.dylib (0x11047c118) and /usr/local/Cellar/ffmpeg/4.4_1/lib/libavdevice.58.13.100.dylib (0x11f968a28). One of the two will be used. Which one is undefined.\n",
      "objc[17868]: Class AVFAudioReceiver is implemented in both /Users/chizzy/opt/anaconda3/envs/torch19/lib/python3.8/site-packages/av/.dylibs/libavdevice.59.7.100.dylib (0x11047c168) and /usr/local/Cellar/ffmpeg/4.4_1/lib/libavdevice.58.13.100.dylib (0x11f968a78). One of the two will be used. Which one is undefined.\n"
     ]
    }
   ],
   "source": [
    "insert_pipe = (\n",
    "    emb_pipe.map(('id', 'vec'), 'insert_status',ops.ann_insert.milvus_client(host='127.0.0.1', port='19530', collection_name='vggish'))\n",
    "        .output()\n",
    ")\n",
    "\n",
    "with open('gtzan300.csv') as f:\n",
    "    freader = reader(f)\n",
    "    next(freader)\n",
    "    for row in freader:\n",
    "        res = insert_pipe(*row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6a3b935e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total inserted data:9300\n"
     ]
    }
   ],
   "source": [
    "print(f'Total inserted data:{collection.num_entities}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32dc8e0e",
   "metadata": {},
   "source": [
    "### 4. Identify an unknown music snippet by similarity search of vectors\n",
    "\n",
    "We can use the same pipeline to generate a set of vectors for a query audio. Then searching across the collection will find the closest embeddings for each vector in the set. \n",
    "\n",
    "Our stratrgy is to find the most likely genre for each frame, acoording to the prediction for all frames, determine the most likely genre of the query audio file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82a45874",
   "metadata": {},
   "source": [
    "The following example recognizes music genres for each audio under `./test/pop/*`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a2f782d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">path</th> <th style=\"text-align: center; font-size: 130%; border: none;\">predict</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">./test/pop/pop.00053.wav</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">pop</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">path</th> <th style=\"text-align: center; font-size: 130%; border: none;\">predict</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">./test/pop/pop.00091.wav</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">pop</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table style=\"border-collapse: collapse;\"><tr><th style=\"text-align: center; font-size: 130%; border: none;\">path</th> <th style=\"text-align: center; font-size: 130%; border: none;\">predict</th></tr> <tr><td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">./test/pop/pop.00054.wav</td> <td style=\"text-align: center; vertical-align: center; border-right: solid 1px #D3D3D3; border-left: solid 1px #D3D3D3; \">pop</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import glob\n",
    "from statistics import mode\n",
    "collection.load()\n",
    "\n",
    "search_pipe = (\n",
    "    pipe.input('path')\n",
    "        .map('path', 'frames', ops.audio_decode.ffmpeg())\n",
    "        .map('frames', 'frames', lambda x: [i[0] for i in x])\n",
    "        .flat_map('frames', 'vec', ops.audio_embedding.vggish())\n",
    "        .map('vec', 'vec', ops.towhee.np_normalize())\n",
    "        .map('vec', 'results', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='vggish', limit=10, metric_type='L2'))\n",
    "        .map('results', 'results', lambda x: [id_label[int(i[0])] for i in x])\n",
    "        .window_all('results', 'predict', lambda x: mode([mode(i) for i in x]))\n",
    ")\n",
    "\n",
    "query_pipe = search_pipe.output('path', 'predict')\n",
    "\n",
    "path = glob.glob('./test/pop/*')\n",
    "for i in path:\n",
    "    res = query_pipe(i)\n",
    "    DataCollection(res).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96df7dd4",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "We have just built a music genre classification system. But how's its performance? We can evaluate the search engine against the ground truths. Here we use the metric `accuracy` to measure performance with the example test data of 30 audio files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d69b2f42",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_pipe = (\n",
    "    search_pipe.map('path', 'ground_truth', lambda x: x.split('/')[-2])\n",
    "        .output('path', 'predict', 'ground_truth')\n",
    ")\n",
    "\n",
    "\n",
    "predicts = []\n",
    "facts = []\n",
    "path = glob.glob('./test/*/*')\n",
    "for i in path:\n",
    "    res = eval_pipe(i).get()\n",
    "    predicts.append(res[1])\n",
    "    facts.append(res[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f98be4d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7666666666666667"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(facts, predicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe2af89",
   "metadata": {},
   "source": [
    "From test above, we can tell the accuracy of this basic music genre classification system is 77%. To make your own solution in production, you can build some more complicated system to improve performance. For example, Towhee provides more options of models and APIs to optimize execution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de19002",
   "metadata": {},
   "source": [
    "## Release a Showcase\n",
    "\n",
    "We've just built a music genre classification system and tested its performance. Now it's time to add some interface and release a showcase. Towhee provides `towhee.api()` to wrap the data processing pipeline as a function with `.as_function()`. So we can build a quick demo with this `demo_function` with [Gradio](https://gradio.app/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "be5721ff",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local URL:  http://127.0.0.1:7860\n",
      "Running on public URL: https://ec6e6de541b21311cd.gradio.live\n",
      "\n",
      "This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"https://ec6e6de541b21311cd.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gradio\n",
    "\n",
    "def music_classify(path):\n",
    "    query_pipe = (\n",
    "        pipe.input('path')\n",
    "            .map('path', 'frames', ops.audio_decode.ffmpeg())\n",
    "            .map('frames', 'frames', lambda x: [i[0] for i in x])\n",
    "            .flat_map('frames', 'vec', ops.audio_embedding.vggish())\n",
    "            .map('vec', 'vec', ops.towhee.np_normalize())\n",
    "            .map('vec', 'results', ops.ann_search.milvus_client(host='127.0.0.1', port='19530', collection_name='vggish', limit=10, metric_type='L2'))\n",
    "            .map('results', 'results', lambda x: [id_label[int(i[0])] for i in x])\n",
    "            .window_all('results', 'predict', lambda x: mode([mode(i) for i in x]))\n",
    "            .output('predict')\n",
    "    )\n",
    "\n",
    "    res = query_pipe(path).get()[0]\n",
    "    return res\n",
    "\n",
    "collection.load()\n",
    "interface = gradio.Interface(music_classify, \n",
    "                             inputs=gradio.Audio(source='upload', type='filepath'),\n",
    "                             outputs=gradio.Textbox(label=\"Music Genre\")\n",
    "                            )\n",
    "\n",
    "interface.launch(inline=True, share=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeb1e145",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "e5c40972ba1b0a4aef81aa5d9f627baea15a8165c0b1725d61ea660ca15c78ae"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}