speech-emotion-recognition-model.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["#  <center> Voice Classification Using ML and Emotion Recognization <center>Compsoft Technology"]},{"cell_type":"markdown","metadata":{},"source":["#  <center> Team : Isaac, Jahnavi ,Irfan , Hemanth <center>"]},{"cell_type":"markdown","metadata":{},"source":["### We Are going to build a speech emotion detection classifier.\n","### But first we need to learn about what is speech recognition (SER) and why are we building this project? Well, few of the reasons are-\n","\n","#### First, lets define SER i.e. Speech Emotion Recognition.\n","* Speech Emotion Recognition, abbreviated as SER, is the act of attempting to recognize human emotion and affective states from speech. This is capitalizing on the fact that voice often reflects underlying emotion through tone and pitch. This is also the phenomenon that animals like dogs and horses employ to be able to understand human emotion.\n","\n","#### Why we need it?\n","\n","1. Emotion recognition is the part of speech recognition which is gaining more popularity and need for it increases enormously. Although there are methods to recognize emotion using machine learning techniques, this project attempts to use deep learning to recognize the emotions from data.\n","\n","2. SER(Speech Emotion Recognition) is used in call center for classifying calls according to emotions and can be used as the performance parameter for conversational analysis thus identifying the unsatisfied customer, customer satisfaction and so on.. for helping companies improving their services\n","\n","3. It can also be used in-car board system based on information of the mental state of the driver can be provided to the system to initiate his/her safety preventing accidents to happen\n","\n","#### Datasets used in this project\n","\n","* Crowd-sourced Emotional Mutimodal Actors Dataset (Crema-D)\n","* Ryerson Audio-Visual Database of Emotional Speech and Song (Ravdess)\n","* Surrey Audio-Visual Expressed Emotion (Savee)\n","* Toronto emotional speech set (Tess)"]},{"cell_type":"markdown","metadata":{},"source":["# Importing Libraries"]},{"cell_type":"code","execution_count":1,"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2022-09-25T07:35:38.034768Z","iopub.status.busy":"2022-09-25T07:35:38.034448Z","iopub.status.idle":"2022-09-25T07:35:39.300773Z","shell.execute_reply":"2022-09-25T07:35:39.296933Z","shell.execute_reply.started":"2022-09-25T07:35:38.034736Z"},"trusted":true},"outputs":[{"ename":"OSError","evalue":"sndfile library not found","output_type":"error","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)","\u001b[0;32m<ipython-input-1-13f767c30986>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# librosa is a Python library for analyzing audio and music. It can be used to extract the data from the audio files we will see it later.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mlibrosa\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mlibrosa\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mseaborn\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.7/site-packages/librosa/__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;31m# And all the librosa sub-modules\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0m_cache\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcache\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mbeat\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdecompose\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.7/site-packages/librosa/core/__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m    124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mtime_frequency\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m  \u001b[0;31m# pylint: disable=wildcard-import\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 126\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0maudio\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m  \u001b[0;31m# pylint: disable=wildcard-import\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    127\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mspectrum\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m  \u001b[0;31m# pylint: disable=wildcard-import\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mpitch\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m  \u001b[0;31m# pylint: disable=wildcard-import\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.7/site-packages/librosa/core/audio.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mwarnings\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0msoundfile\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0maudioread\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/opt/conda/lib/python3.7/site-packages/soundfile.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m    140\u001b[0m     \u001b[0m_libname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_find_library\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'sndfile'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_libname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mOSError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'sndfile library not found'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    143\u001b[0m     \u001b[0m_snd\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_ffi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_libname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mOSError\u001b[0m: sndfile library not found"]}],"source":["import pandas as pd\n","import numpy as np\n","\n","import os\n","import sys\n","\n","# librosa is a Python library for analyzing audio and music. It can be used to extract the data from the audio files we will see it later.\n","import librosa\n","import librosa.display\n","import seaborn as sns\n","import matplotlib.pyplot as plt\n","\n","from sklearn.preprocessing import StandardScaler, OneHotEncoder\n","from sklearn.metrics import confusion_matrix, classification_report\n","from sklearn.model_selection import train_test_split\n","\n","# to play the audio files\n","from IPython.display import Audio\n","\n","import keras\n","from keras.callbacks import ReduceLROnPlateau\n","from keras.models import Sequential\n","from keras.layers import Dense, Conv1D, MaxPooling1D, Flatten, Dropout, BatchNormalization\n","from keras.utils import np_utils, to_categorical\n","from keras.callbacks import ModelCheckpoint\n","\n","import warnings\n","if not sys.warnoptions:\n","    warnings.simplefilter(\"ignore\")\n","warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "]},{"cell_type":"markdown","metadata":{},"source":["## Data Preparation\n","* As we are working with four different datasets, so i will be creating a dataframe storing all emotions of the data in dataframe with their paths.\n","* We will use this dataframe to extract features for our model training."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.302166Z","iopub.status.idle":"2022-09-25T07:35:39.302572Z"},"trusted":true},"outputs":[],"source":["# Paths for data.\n","Ravdess = \"/kaggle/input/ravdess-emotional-speech-audio/audio_speech_actors_01-24/\"\n","Crema = \"/kaggle/input/cremad/AudioWAV/\"\n","Tess = \"/kaggle/input/toronto-emotional-speech-set-tess/tess toronto emotional speech set data/TESS Toronto emotional speech set data/\"\n","Savee = \"/kaggle/input/surrey-audiovisual-expressed-emotion-savee/ALL/\""]},{"cell_type":"markdown","metadata":{},"source":["##  <center> 1. Ravdess Dataframe <center>\n","Here is the filename identifiers as per the official RAVDESS website:\n","\n","* Modality (01 = full-AV, 02 = video-only, 03 = audio-only).\n","* Vocal channel (01 = speech, 02 = song).\n","* Emotion (01 = neutral, 02 = calm, 03 = happy, 04 = sad, 05 = angry, 06 = fearful, 07 = disgust, 08 = surprised).\n","* Emotional intensity (01 = normal, 02 = strong). NOTE: There is no strong intensity for the 'neutral' emotion.\n","* Statement (01 = \"Kids are talking by the door\", 02 = \"Dogs are sitting by the door\").\n","* Repetition (01 = 1st repetition, 02 = 2nd repetition).\n","* Actor (01 to 24. Odd numbered actors are male, even numbered actors are female).\n","\n","So, here's an example of an audio filename. 02-01-06-01-02-01-12.mp4\n","This means the meta data for the audio file is:\n","\n","* Video-only (02)\n","* Speech (01)\n","* Fearful (06)\n","* Normal intensity (01)\n","* Statement \"dogs\" (02)\n","* 1st Repetition (01)\n","* 12th Actor (12) - Female (as the actor ID number is even)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.304049Z","iopub.status.idle":"2022-09-25T07:35:39.304887Z"},"trusted":true},"outputs":[],"source":["ravdess_directory_list = os.listdir(Ravdess)\n","\n","file_emotion = []\n","file_path = []\n","for dir in ravdess_directory_list:\n","    # as their are 20 different actors in our previous directory we need to extract files for each actor.\n","    actor = os.listdir(Ravdess + dir)\n","    for file in actor:\n","        part = file.split('.')[0]\n","        part = part.split('-')\n","        # third part in each file represents the emotion associated to that file.\n","        file_emotion.append(int(part[2]))\n","        file_path.append(Ravdess + dir + '/' + file)\n","        \n","# dataframe for emotion of files\n","emotion_df = pd.DataFrame(file_emotion, columns=['Emotions'])\n","\n","# dataframe for path of files.\n","path_df = pd.DataFrame(file_path, columns=['Path'])\n","Ravdess_df = pd.concat([emotion_df, path_df], axis=1)\n","\n","# changing integers to actual emotions.\n","Ravdess_df.Emotions.replace({1:'neutral', 2:'calm', 3:'happy', 4:'sad', 5:'angry', 6:'fear', 7:'disgust', 8:'surprise'}, inplace=True)\n","Ravdess_df.head()"]},{"cell_type":"markdown","metadata":{},"source":["## <center>2. Crema DataFrame</center>"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.306251Z","iopub.status.idle":"2022-09-25T07:35:39.306946Z"},"trusted":true},"outputs":[],"source":["crema_directory_list = os.listdir(Crema)\n","\n","file_emotion = []\n","file_path = []\n","\n","for file in crema_directory_list:\n","    # storing file paths\n","    file_path.append(Crema + file)\n","    # storing file emotions\n","    part=file.split('_')\n","    if part[2] == 'SAD':\n","        file_emotion.append('sad')\n","    elif part[2] == 'ANG':\n","        file_emotion.append('angry')\n","    elif part[2] == 'DIS':\n","        file_emotion.append('disgust')\n","    elif part[2] == 'FEA':\n","        file_emotion.append('fear')\n","    elif part[2] == 'HAP':\n","        file_emotion.append('happy')\n","    elif part[2] == 'NEU':\n","        file_emotion.append('neutral')\n","    else:\n","        file_emotion.append('Unknown')\n","        \n","# dataframe for emotion of files\n","emotion_df = pd.DataFrame(file_emotion, columns=['Emotions'])\n","\n","# dataframe for path of files.\n","path_df = pd.DataFrame(file_path, columns=['Path'])\n","Crema_df = pd.concat([emotion_df, path_df], axis=1)\n","Crema_df.head()"]},{"cell_type":"markdown","metadata":{},"source":["##  <center> 3. TESS dataset <center>"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.309542Z","iopub.status.idle":"2022-09-25T07:35:39.310315Z"},"trusted":true},"outputs":[],"source":["tess_directory_list = os.listdir(Tess)\n","\n","file_emotion = []\n","file_path = []\n","\n","for dir in tess_directory_list:\n","    directories = os.listdir(Tess + dir)\n","    for file in directories:\n","        part = file.split('.')[0]\n","        part = part.split('_')[2]\n","        if part=='ps':\n","            file_emotion.append('surprise')\n","        else:\n","            file_emotion.append(part)\n","        file_path.append(Tess + dir + '/' + file)\n","        \n","# dataframe for emotion of files\n","emotion_df = pd.DataFrame(file_emotion, columns=['Emotions'])\n","\n","# dataframe for path of files.\n","path_df = pd.DataFrame(file_path, columns=['Path'])\n","Tess_df = pd.concat([emotion_df, path_df], axis=1)\n","Tess_df.head()"]},{"cell_type":"markdown","metadata":{},"source":["##  <center> 4. CREMA-D dataset <center>\n","The audio files in this dataset are named in such a way that the prefix letters describes the emotion classes as follows:\n","\n","* 'a' = 'anger'\n","* 'd' = 'disgust'\n","* 'f' = 'fear'\n","* 'h' = 'happiness'\n","* 'n' = 'neutral'\n","* 'sa' = 'sadness'\n","* 'su' = 'surprise'"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.311410Z","iopub.status.idle":"2022-09-25T07:35:39.312156Z"},"trusted":true},"outputs":[],"source":["savee_directory_list = os.listdir(Savee)\n","\n","file_emotion = []\n","file_path = []\n","\n","for file in savee_directory_list:\n","    file_path.append(Savee + file)\n","    part = file.split('_')[1]\n","    ele = part[:-6]\n","    if ele=='a':\n","        file_emotion.append('angry')\n","    elif ele=='d':\n","        file_emotion.append('disgust')\n","    elif ele=='f':\n","        file_emotion.append('fear')\n","    elif ele=='h':\n","        file_emotion.append('happy')\n","    elif ele=='n':\n","        file_emotion.append('neutral')\n","    elif ele=='sa':\n","        file_emotion.append('sad')\n","    else:\n","        file_emotion.append('surprise')\n","        \n","# dataframe for emotion of files\n","emotion_df = pd.DataFrame(file_emotion, columns=['Emotions'])\n","\n","# dataframe for path of files.\n","path_df = pd.DataFrame(file_path, columns=['Path'])\n","Savee_df = pd.concat([emotion_df, path_df], axis=1)\n","Savee_df.head()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.313295Z","iopub.status.idle":"2022-09-25T07:35:39.314035Z"},"trusted":true},"outputs":[],"source":["# creating Dataframe using all the 4 dataframes we created so far.\n","data_path = pd.concat([Ravdess_df, Crema_df, Tess_df, Savee_df], axis = 0)\n","data_path.to_csv(\"data_path.csv\",index=False)\n","data_path.head()"]},{"cell_type":"markdown","metadata":{},"source":["## Data Visualisation and Exploration"]},{"cell_type":"markdown","metadata":{},"source":["First let's plot the count of each emotions in our dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.315152Z","iopub.status.idle":"2022-09-25T07:35:39.315925Z"},"trusted":true},"outputs":[],"source":["plt.title('Count of Emotions', size=16)\n","sns.countplot(data_path.Emotions)\n","plt.ylabel('Count', size=12)\n","plt.xlabel('Emotions', size=12)\n","sns.despine(top=True, right=True, left=False, bottom=False)\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["We can also plot waveplots and spectograms for audio signals\n","\n","* Waveplots - Waveplots let us know the loudness of the audio at a given time.\n","* Spectograms - A spectrogram is a visual representation of the spectrum of frequencies of sound or other signals as they vary with time. It’s a representation of frequencies changing with respect to time for given audio/music signals."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.317118Z","iopub.status.idle":"2022-09-25T07:35:39.317954Z"},"trusted":true},"outputs":[],"source":["def create_waveplot(data, sr, e):\n","    plt.figure(figsize=(10, 3))\n","    plt.title('Waveplot for audio with {} emotion'.format(e), size=15)\n","    librosa.display.waveplot(data, sr=sr)\n","    plt.show()\n","\n","def create_spectrogram(data, sr, e):\n","    # stft function converts the data into short term fourier transform\n","    X = librosa.stft(data)\n","    Xdb = librosa.amplitude_to_db(abs(X))\n","    plt.figure(figsize=(12, 3))\n","    plt.title('Spectrogram for audio with {} emotion'.format(e), size=15)\n","    librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='hz')   \n","    #librosa.display.specshow(Xdb, sr=sr, x_axis='time', y_axis='log')\n","    plt.colorbar()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.319074Z","iopub.status.idle":"2022-09-25T07:35:39.319788Z"},"trusted":true},"outputs":[],"source":["emotion='fear'\n","path = np.array(data_path.Path[data_path.Emotions==emotion])[1]\n","data, sampling_rate = librosa.load(path)\n","create_waveplot(data, sampling_rate, emotion)\n","create_spectrogram(data, sampling_rate, emotion)\n","Audio(path)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.320875Z","iopub.status.idle":"2022-09-25T07:35:39.321625Z"},"trusted":true},"outputs":[],"source":["emotion='angry'\n","path = np.array(data_path.Path[data_path.Emotions==emotion])[1]\n","data, sampling_rate = librosa.load(path)\n","create_waveplot(data, sampling_rate, emotion)\n","create_spectrogram(data, sampling_rate, emotion)\n","Audio(path)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.322709Z","iopub.status.idle":"2022-09-25T07:35:39.323463Z"},"trusted":true},"outputs":[],"source":["emotion='sad'\n","path = np.array(data_path.Path[data_path.Emotions==emotion])[1]\n","data, sampling_rate = librosa.load(path)\n","create_waveplot(data, sampling_rate, emotion)\n","create_spectrogram(data, sampling_rate, emotion)\n","Audio(path)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.324559Z","iopub.status.idle":"2022-09-25T07:35:39.325315Z"},"trusted":true},"outputs":[],"source":["emotion='happy'\n","path = np.array(data_path.Path[data_path.Emotions==emotion])[1]\n","data, sampling_rate = librosa.load(path)\n","create_waveplot(data, sampling_rate, emotion)\n","create_spectrogram(data, sampling_rate, emotion)\n","Audio(path)"]},{"cell_type":"markdown","metadata":{},"source":["## Data Augmentation\n","\n","- Data augmentation is the process by which we create new synthetic data samples by adding small perturbations on our initial training set.\n","- To generate syntactic data for audio, we can apply noise injection, shifting time, changing pitch and speed.\n","- The objective is to make our model invariant to those perturbations and enhace its ability to generalize.\n","- In order to this to work adding the perturbations must conserve the same label as the original training sample.\n","- In images data augmention can be performed by shifting the image, zooming, rotating ...\n","\n","First, let's check which augmentation techniques works better for our dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.326452Z","iopub.status.idle":"2022-09-25T07:35:39.327203Z"},"trusted":true},"outputs":[],"source":["def noise(data):\n","    noise_amp = 0.035*np.random.uniform()*np.amax(data)\n","    data = data + noise_amp*np.random.normal(size=data.shape[0])\n","    return data\n","\n","def stretch(data, rate=0.8):\n","    return librosa.effects.time_stretch(data, rate)\n","\n","def shift(data):\n","    shift_range = int(np.random.uniform(low=-5, high = 5)*1000)\n","    return np.roll(data, shift_range)\n","\n","def pitch(data, sampling_rate, pitch_factor=0.7):\n","    return librosa.effects.pitch_shift(data, sampling_rate, pitch_factor)\n","\n","# taking any example and checking for techniques.\n","path = np.array(data_path.Path)[1]\n","data, sample_rate = librosa.load(path)"]},{"cell_type":"markdown","metadata":{},"source":["#### 1. Simple Audio"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.328336Z","iopub.status.idle":"2022-09-25T07:35:39.329084Z"},"trusted":true},"outputs":[],"source":["plt.figure(figsize=(14,4))\n","librosa.display.waveplot(y=data, sr=sample_rate)\n","Audio(path)"]},{"cell_type":"markdown","metadata":{},"source":["#### 2. Noise Injection"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.330193Z","iopub.status.idle":"2022-09-25T07:35:39.330953Z"},"trusted":true},"outputs":[],"source":["x = noise(data)\n","plt.figure(figsize=(14,4))\n","librosa.display.waveplot(y=x, sr=sample_rate)\n","Audio(x, rate=sample_rate)"]},{"cell_type":"markdown","metadata":{},"source":["We can see noise injection is a very good augmentation technique because of which we can assure our training model is not overfitted"]},{"cell_type":"markdown","metadata":{},"source":["#### 3. Stretching"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.332068Z","iopub.status.idle":"2022-09-25T07:35:39.332791Z"},"trusted":true},"outputs":[],"source":["x = stretch(data)\n","plt.figure(figsize=(14,4))\n","librosa.display.waveplot(y=x, sr=sample_rate)\n","Audio(x, rate=sample_rate)"]},{"cell_type":"markdown","metadata":{},"source":["#### 4. Shifting"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.333965Z","iopub.status.idle":"2022-09-25T07:35:39.334759Z"},"trusted":true},"outputs":[],"source":["x = shift(data)\n","plt.figure(figsize=(14,4))\n","librosa.display.waveplot(y=x, sr=sample_rate)\n","Audio(x, rate=sample_rate)"]},{"cell_type":"markdown","metadata":{},"source":["#### 5. Pitch"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.335880Z","iopub.status.idle":"2022-09-25T07:35:39.336649Z"},"trusted":true},"outputs":[],"source":["x = pitch(data, sample_rate)\n","plt.figure(figsize=(14,4))\n","librosa.display.waveplot(y=x, sr=sample_rate)\n","Audio(x, rate=sample_rate)"]},{"cell_type":"markdown","metadata":{},"source":["- From the above types of augmentation techniques i am using noise, stretching(ie. changing speed) and some pitching."]},{"cell_type":"markdown","metadata":{},"source":["## Feature Extraction\n","- Extraction of features is a very important part in analyzing and finding relations between different things. As we already know that the data provided of audio cannot be understood by the models directly so we need to convert them into an understandable format for which feature extraction is used.\n","\n","\n","The audio signal is a three-dimensional signal in which three axes represent time, amplitude and frequency.\n","\n","![image.png](https://miro.medium.com/max/633/1*7sKM9aECRmuoqTadCYVw9A.jpeg)\n","\n","I am no expert on audio signals and feature extraction on audio files so i need to search and found a very good blog written by [Askash Mallik](https://medium.com/heuristics/audio-signal-feature-extraction-and-clustering-935319d2225) on feature extraction.\n","\n","As stated there with the help of the sample rate and the sample data, one can perform several transformations on it to extract valuable features out of it.\n","1. Zero Crossing Rate : The rate of sign-changes of the signal during the duration of a particular frame.\n","2. Energy : The sum of squares of the signal values, normalized by the respective frame length.\n","3. Entropy of Energy : The entropy of sub-frames’ normalized energies. It can be interpreted as a measure of abrupt changes.\n","4. Spectral Centroid : The center of gravity of the spectrum.\n","5. Spectral Spread : The second central moment of the spectrum.\n","6. Spectral Entropy :  Entropy of the normalized spectral energies for a set of sub-frames.\n","7. Spectral Flux : The squared difference between the normalized magnitudes of the spectra of the two successive frames.\n","8. Spectral Rolloff : The frequency below which 90% of the magnitude distribution of the spectrum is concentrated.\n","9.  MFCCs Mel Frequency Cepstral Coefficients form a cepstral representation where the frequency bands are not linear but distributed according to the mel-scale.\n","10. Chroma Vector : A 12-element representation of the spectral energy where the bins represent the 12 equal-tempered pitch classes of western-type music (semitone spacing).\n","11. Chroma Deviation : The standard deviation of the 12 chroma coefficients.\n","\n","\n","In this project i am not going deep in feature selection process to check which features are good for our dataset rather i am only extracting 5 features:\n","- Zero Crossing Rate\n","- Chroma_stft\n","- MFCC\n","- RMS(root mean square) value\n","- MelSpectogram to train our model."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.337763Z","iopub.status.idle":"2022-09-25T07:35:39.338521Z"},"trusted":true},"outputs":[],"source":["def extract_features(data):\n","    # ZCR\n","    result = np.array([])\n","    zcr = np.mean(librosa.feature.zero_crossing_rate(y=data).T, axis=0)\n","    result=np.hstack((result, zcr)) # stacking horizontally\n","\n","    # Chroma_stft\n","    stft = np.abs(librosa.stft(data))\n","    chroma_stft = np.mean(librosa.feature.chroma_stft(S=stft, sr=sample_rate).T, axis=0)\n","    result = np.hstack((result, chroma_stft)) # stacking horizontally\n","\n","    # MFCC\n","    mfcc = np.mean(librosa.feature.mfcc(y=data, sr=sample_rate).T, axis=0)\n","    result = np.hstack((result, mfcc)) # stacking horizontally\n","\n","    # Root Mean Square Value\n","    rms = np.mean(librosa.feature.rms(y=data).T, axis=0)\n","    result = np.hstack((result, rms)) # stacking horizontally\n","\n","    # MelSpectogram\n","    mel = np.mean(librosa.feature.melspectrogram(y=data, sr=sample_rate).T, axis=0)\n","    result = np.hstack((result, mel)) # stacking horizontally\n","    \n","    return result\n","\n","def get_features(path):\n","    # duration and offset are used to take care of the no audio in start and the ending of each audio files as seen above.\n","    data, sample_rate = librosa.load(path, duration=2.5, offset=0.6)\n","    \n","    # without augmentation\n","    res1 = extract_features(data)\n","    result = np.array(res1)\n","    \n","    # data with noise\n","    noise_data = noise(data)\n","    res2 = extract_features(noise_data)\n","    result = np.vstack((result, res2)) # stacking vertically\n","    \n","    # data with stretching and pitching\n","    new_data = stretch(data)\n","    data_stretch_pitch = pitch(new_data, sample_rate)\n","    res3 = extract_features(data_stretch_pitch)\n","    result = np.vstack((result, res3)) # stacking vertically\n","    \n","    return result"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.339615Z","iopub.status.idle":"2022-09-25T07:35:39.340404Z"},"trusted":true},"outputs":[],"source":["X, Y = [], []\n","for path, emotion in zip(data_path.Path, data_path.Emotions):\n","    feature = get_features(path)\n","    for ele in feature:\n","        X.append(ele)\n","        # appending emotion 3 times as we have made 3 augmentation techniques on each audio file.\n","        Y.append(emotion)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.341533Z","iopub.status.idle":"2022-09-25T07:35:39.342277Z"},"trusted":true},"outputs":[],"source":["len(X), len(Y), data_path.Path.shape"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.343388Z","iopub.status.idle":"2022-09-25T07:35:39.344146Z"},"trusted":true},"outputs":[],"source":["Features = pd.DataFrame(X)\n","Features['labels'] = Y\n","Features.to_csv('features.csv', index=False)\n","Features.head()"]},{"cell_type":"markdown","metadata":{},"source":["* We have applied data augmentation and extracted the features for each audio files and saved them."]},{"cell_type":"markdown","metadata":{},"source":["## Data Preparation\n","\n","- As of now we have extracted the data, now we need to normalize and split our data for training and testing."]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.345259Z","iopub.status.idle":"2022-09-25T07:35:39.346014Z"},"trusted":true},"outputs":[],"source":["X = Features.iloc[: ,:-1].values\n","Y = Features['labels'].values"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.347113Z","iopub.status.idle":"2022-09-25T07:35:39.347835Z"},"trusted":true},"outputs":[],"source":["# As this is a multiclass classification problem onehotencoding our Y.\n","encoder = OneHotEncoder()\n","Y = encoder.fit_transform(np.array(Y).reshape(-1,1)).toarray()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.348959Z","iopub.status.idle":"2022-09-25T07:35:39.349735Z"},"trusted":true},"outputs":[],"source":["# splitting data\n","x_train, x_test, y_train, y_test = train_test_split(X, Y, random_state=0, shuffle=True)\n","x_train.shape, y_train.shape, x_test.shape, y_test.shape"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.350846Z","iopub.status.idle":"2022-09-25T07:35:39.351679Z"},"trusted":true},"outputs":[],"source":["# scaling our data with sklearn's Standard scaler\n","scaler = StandardScaler()\n","x_train = scaler.fit_transform(x_train)\n","x_test = scaler.transform(x_test)\n","x_train.shape, y_train.shape, x_test.shape, y_test.shape"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.352770Z","iopub.status.idle":"2022-09-25T07:35:39.353525Z"},"trusted":true},"outputs":[],"source":["# making our data compatible to model.\n","x_train = np.expand_dims(x_train, axis=2)\n","x_test = np.expand_dims(x_test, axis=2)\n","x_train.shape, y_train.shape, x_test.shape, y_test.shape"]},{"cell_type":"markdown","metadata":{},"source":["## Modelling"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.354637Z","iopub.status.idle":"2022-09-25T07:35:39.355389Z"},"trusted":true},"outputs":[],"source":["model=Sequential()\n","model.add(Conv1D(256, kernel_size=5, strides=1, padding='same', activation='relu', input_shape=(x_train.shape[1], 1)))\n","model.add(MaxPooling1D(pool_size=5, strides = 2, padding = 'same'))\n","\n","model.add(Conv1D(256, kernel_size=5, strides=1, padding='same', activation='relu'))\n","model.add(MaxPooling1D(pool_size=5, strides = 2, padding = 'same'))\n","\n","model.add(Conv1D(128, kernel_size=5, strides=1, padding='same', activation='relu'))\n","model.add(MaxPooling1D(pool_size=5, strides = 2, padding = 'same'))\n","model.add(Dropout(0.2))\n","\n","model.add(Conv1D(64, kernel_size=5, strides=1, padding='same', activation='relu'))\n","model.add(MaxPooling1D(pool_size=5, strides = 2, padding = 'same'))\n","\n","model.add(Flatten())\n","model.add(Dense(units=32, activation='relu'))\n","model.add(Dropout(0.3))\n","\n","model.add(Dense(units=8, activation='softmax'))\n","model.compile(optimizer = 'adam' , loss = 'categorical_crossentropy' , metrics = ['accuracy'])\n","\n","model.summary()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.356482Z","iopub.status.idle":"2022-09-25T07:35:39.357235Z"},"trusted":true},"outputs":[],"source":["rlrp = ReduceLROnPlateau(monitor='loss', factor=0.4, verbose=0, patience=2, min_lr=0.0000001)\n","history=model.fit(x_train, y_train, batch_size=64, epochs=50, validation_data=(x_test, y_test), callbacks=[rlrp])"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.358331Z","iopub.status.idle":"2022-09-25T07:35:39.359079Z"},"trusted":true},"outputs":[],"source":["print(\"Accuracy of our model on test data : \" , model.evaluate(x_test,y_test)[1]*100 , \"%\")\n","\n","epochs = [i for i in range(50)]\n","fig , ax = plt.subplots(1,2)\n","train_acc = history.history['accuracy']\n","train_loss = history.history['loss']\n","test_acc = history.history['val_accuracy']\n","test_loss = history.history['val_loss']\n","\n","fig.set_size_inches(20,6)\n","ax[0].plot(epochs , train_loss , label = 'Training Loss')\n","ax[0].plot(epochs , test_loss , label = 'Testing Loss')\n","ax[0].set_title('Training & Testing Loss')\n","ax[0].legend()\n","ax[0].set_xlabel(\"Epochs\")\n","\n","ax[1].plot(epochs , train_acc , label = 'Training Accuracy')\n","ax[1].plot(epochs , test_acc , label = 'Testing Accuracy')\n","ax[1].set_title('Training & Testing Accuracy')\n","ax[1].legend()\n","ax[1].set_xlabel(\"Epochs\")\n","plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.360188Z","iopub.status.idle":"2022-09-25T07:35:39.360915Z"},"trusted":true},"outputs":[],"source":["# predicting on test data.\n","pred_test = model.predict(x_test)\n","y_pred = encoder.inverse_transform(pred_test)\n","\n","y_test = encoder.inverse_transform(y_test)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.362231Z","iopub.status.idle":"2022-09-25T07:35:39.363035Z"},"trusted":true},"outputs":[],"source":["df = pd.DataFrame(columns=['Predicted Labels', 'Actual Labels'])\n","df['Predicted Labels'] = y_pred.flatten()\n","df['Actual Labels'] = y_test.flatten()\n","\n","df.head(10)"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.364190Z","iopub.status.idle":"2022-09-25T07:35:39.364920Z"},"trusted":true},"outputs":[],"source":["cm = confusion_matrix(y_test, y_pred)\n","plt.figure(figsize = (12, 10))\n","cm = pd.DataFrame(cm , index = [i for i in encoder.categories_] , columns = [i for i in encoder.categories_])\n","sns.heatmap(cm, linecolor='white', cmap='Blues', linewidth=1, annot=True, fmt='')\n","plt.title('Confusion Matrix', size=20)\n","plt.xlabel('Predicted Labels', size=14)\n","plt.ylabel('Actual Labels', size=14)\n","plt.show()"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.status.busy":"2022-09-25T07:35:39.366072Z","iopub.status.idle":"2022-09-25T07:35:39.366821Z"},"trusted":true},"outputs":[],"source":["print(classification_report(y_test, y_pred))"]},{"cell_type":"markdown","metadata":{},"source":["- We can see our model is more accurate in predicting surprise, angry emotions and it makes sense also because audio files of these emotions differ to other audio files in a lot of ways like pitch, speed etc..\n","- We overall achieved 61% accuracy on our test data and its decent but we can improve it more by applying more augmentation techniques and using other feature extraction methods."]},{"cell_type":"markdown","metadata":{},"source":["### This Concludes Our InternShip Project for Voice Classification using ML that analyses the sentiment behind the tone of the voice and predicts the sentiment invovled from the Dataset"]}],"metadata":{"kernelspec":{"display_name":"Python 3.10.4 64-bit","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.4"},"vscode":{"interpreter":{"hash":"58837b1b657ea91009af8409fc244ae3b5ccf93ea980d6fb6b80adc5f697f4cc"}}},"nbformat":4,"nbformat_minor":4}