zishansami102 · THEFASHIONGEEK · Dec 3, 2019 · Dec 3, 2019 · Dec 3, 2019 · Dec 3, 2019
diff --git a/1.pdf b/1.pdf
diff --git a/annotation_training.pkl → Annotations/annotation_training.pkl b/annotation_training.pkl → Annotations/annotation_training.pkl
diff --git a/annotation_validation.pkl → Annotations/annotation_validation.pkl b/annotation_validation.pkl → Annotations/annotation_validation.pkl
diff --git a/Images/Dan_model.PNG b/Images/Dan_model.PNG
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 THEFASHIONGEEK
+Copyright (c) 2019 AKULA HEMANTH KUMAR
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@ As is known, the first impression made is highly important in many contexts, suc
 
 The model used is called `Descriptor Aggregation Network` called DAN in short.
 
-![Model Archi](modelImg.png)
+![Model Archi](Images/Dan_model.PNG)
 
 What distinguishes DAN from the traditional CNN is: the fully connected layers are discarded, and replaced by both average- and max-pooling following the last convolutional layers (Pool5). Meanwhile, each pooling operation is followed by the standard L2-normalization. After that, the obtained two 512-d feature vectors are concatenated as the final image representation. Thus, in DAN, the deep descriptors of the last convolutional layers are aggregated as a single visual feature. Finally, a regression (fc+sigmoid) layer is added for end-to-end training.
 
@@ -28,7 +28,7 @@ These instructions will get you a copy of the project up and running on your loc
 ### Prerequisites
 
 * [Python3](https://www.python.org/downloads/release/python-373/) - Python version 3.7.3
-* [Numpy](http://www.numpy.org/) - Multidimensioanl Mathematical Computing
+* [Numpy](http://www.numpy.org/) - Multidimensional Mathematical Computing
 * [Tensorflow 1.14.0](https://www.tensorflow.org/) - Deep Learning python module
 * [Pandas](https://pandas.pydata.org/) - Loading csv files
 * [Cha-Learn Dataset](http://chalearnlap.cvc.uab.es/dataset/24/description/) - Dataset for this problem
@@ -37,6 +37,7 @@ These instructions will get you a copy of the project up and running on your loc
 * [OpenCV 3.4.1](https://breakthrough.github.io/Installing-OpenCV/)  library used for Image Processing
 * [ffmpeg](https://ffmpeg.zeranoe.com/builds/ ) software suite of libraries and programs for handling video, audio, and other multimedia 
 files and streams
+* [python_speech_features](https://pypi.org/project/python_speech_features/) This library provides common speech features for ASR including MFCCs and filterbank energies
 
 ### Installing
 
@@ -46,10 +47,15 @@ Clone the repository
 git clone https://github.com/THEFASHIONGEEK/First-Impression.git
 ```
 
-Downlad the training dataset and extract it into a new /data directory with all 75 training zip files and 25 validation zip files as it is, we will extract them through the script.
+Download the training dataset and extract it into a new /data directory with all 75 training zip files and 25 validation zip files as it is, we will extract them through the script.
 
 [Download](http://www.vlfeat.org/matconvnet/models/vgg-face.mat) Pretrained Vgg-face model and move it to the root directory
 
+Run the requirements.txt 
+
+```
+pip install -r requirements.txt
+```
 Run the Video_to_Image.py file to scrape the images from the videos and save it to a new ImageData directory
 
 ```
@@ -66,6 +72,11 @@ If succesfully completed then run the Write_Into_TFRecords.py file to form a dat
 ```
 python Write_Into_TFRecords.py
 ```
+Run the feat_extraction_from_wav.py file to form a data pipeline by saving the all the train audio into train_audio_full.tfrecords file , all the validation images into val_audio_full.tfrecords to load it later during training
+
+```
+python feat_extraction_from_wav.py
+```
 
 Start the training by running the following command
 
@@ -78,3 +89,5 @@ python train.py
 
 * [paper](https://cs.nju.edu.cn/wujx/paper/eccvw16_APA.pdf) - Implemented paper
 * [TfRecord Data Pipeline](http://machinelearninguru.com/deep_learning/data_preparation/tfrecord/tfrecord.html#read) - Used to make data pipeline
+* [VGG16 in TensorFlow](https://www.cs.toronto.edu/~frossard/post/vgg16/) - Used in DAN 
+* [Author's code](https://github.com/tzzcl/ChaLearn-APA-Code#2-extract-audio-feature-from-video)
diff --git a/Video_to_Image.py b/Video_to_Image.py
@@ -1,107 +1,127 @@
-'''
+"""
 Extract all the 6 training zipped files and 2 validation zipped files into data folder and then run this script
-'''
-import cv2
-import numpy as np
+"""
 import os
 import zipfile
 
-## Runnin a loop throught all the zipped training file to extract all video and then extract 100 frames from each.
-for i in range(1,76):
-    if i<10:
-        zipfilename = 'training80_0'+str(i)+'.zip'
+import numpy as np
+
+import cv2
+
+# Runnin a loop throught all the zipped training file to extract all video and then extract 100 frames from each.
+
+### Training data ###
+for i in range(1, 76):
+    if i < 10:
+        zipfilename = "training80_0" + str(i) + ".zip"
     else:
-        zipfilename = 'training80_'+str(i)+'.zip'
+        zipfilename = "training80_" + str(i) + ".zip"
     ## Accessing the zipfile i
-    archive = zipfile.ZipFile('data/'+zipfilename, 'r')
-    zipfilename = zipfilename.split('.zip')[0]
+    archive = zipfile.ZipFile("data/" + zipfilename, "r")
+    zipfilename = zipfilename.split(".zip")[0]
 
     ##Extracting all videos in it and saving it all to the new folder with same name as zipped one
-    archive.extractall('unzippedData/'+zipfilename)
-    
+    archive.extractall("unzippedData/" + zipfilename)
+
     ## Running a loop over all the videos in the zipped file and extracting 100 frames from each
     for file_name in archive.namelist():
-        cap = cv2.VideoCapture('unzippedData/'+zipfilename+'/'+file_name)
+        cap = cv2.VideoCapture("unzippedData/" + zipfilename + "/" + file_name)
 
-        file_name=(file_name.split('.mp4'))[0]
+        file_name = (file_name.split(".mp4"))[0]
         ## Creating folder to save all the 100 frames from the video
         try:
-            if not os.path.exists('ImageData/trainingData/'+file_name):
-                os.makedirs('ImageData/trainingData/'+file_name)
+            if not os.path.exists("ImageData/trainingData/" + file_name):
+                os.makedirs("ImageData/trainingData/" + file_name)
         except OSError:
-            print ('Error: Creating directory of data')
+            print("Error: Creating directory of data")
 
         ## Setting the frame limit to 100
         cap.set(cv2.CAP_PROP_FRAME_COUNT, 101)
-        length=101
-        count=0
+        length = 101
+        count = 0
         ## Running a loop to each frame and saving it in the created folder
-        while(cap.isOpened()):
-            count+=1
-            if length==count:
+        while cap.isOpened():
+            count += 1
+            if length == count:
                 break
             ret, frame = cap.read()
             if frame is None:
                 continue
 
             ## Resizing it to 256*256 to save the disk space and fit into the model
-            frame = cv2.resize(frame,(256, 256), interpolation = cv2.INTER_CUBIC)
+            frame = cv2.resize(frame, (256, 256), interpolation=cv2.INTER_CUBIC)
             # Saves image of the current frame in jpg file
-            name = 'ImageData/trainingData/'+str(file_name)+'/frame' + str(count) + '.jpg'
+            name = (
+                "ImageData/trainingData/"
+                + str(file_name)
+                + "/frame"
+                + str(count)
+                + ".jpg"
+            )
             cv2.imwrite(name, frame)
-            
-            if cv2.waitKey(1) & 0xFF == ord('q'):
+
+            if cv2.waitKey(1) & 0xFF == ord("q"):
                 break
 
         ## Print the file which is done
-        print (zipfilename, ':', file_name)
-#
-for i in range(1,26):
-    if i<10:
-        zipfilename = 'validation80_0'+str(i)+'.zip'
+        print(zipfilename, ":", file_name)
+### Training data ###
+
+### Validation data ###
+for i in range(1, 26):
+    if i < 10:
+        zipfilename = "validation80_0" + str(i) + ".zip"
     else:
-        zipfilename = 'validation80_'+str(i)+'.zip'
+        zipfilename = "validation80_" + str(i) + ".zip"
     ## Accessing the zipfile i
-    archive = zipfile.ZipFile('data/'+zipfilename, 'r')
-    zipfilename = zipfilename.split('.zip')[0]
+    archive = zipfile.ZipFile("data/" + zipfilename, "r")
+    zipfilename = zipfilename.split(".zip")[0]
 
     ##Extracting all videos in it and saving it all to the new folder with same name as zipped one
-    archive.extractall('unzippedData/'+zipfilename)
-    
+    archive.extractall("unzippedData/" + zipfilename)
+
     ## Running a loop over all the videos in the zipped file and extracting 100 frames from each
     for file_name in archive.namelist():
-        cap = cv2.VideoCapture('unzippedData/'+zipfilename+'/'+file_name)
+        cap = cv2.VideoCapture("unzippedData/" + zipfilename + "/" + file_name)
 
-        file_name=(file_name.split('.mp4'))[0]
+        file_name = (file_name.split(".mp4"))[0]
         ## Creating folder to save all the 100 frames from the video
         try:
-            if not os.path.exists('ImageData/validationData/'+file_name):
-                os.makedirs('ImageData/validationData/'+file_name)
+            if not os.path.exists("ImageData/validationData/" + file_name):
+                os.makedirs("ImageData/validationData/" + file_name)
         except OSError:
-            print ('Error: Creating directory of data')
+            print("Error: Creating directory of data")
 
         ## Setting the frame limit to 100
         cap.set(cv2.CAP_PROP_FRAME_COUNT, 101)
-        length=101
-        count=0
+        length = 101
+        count = 0
         ## Running a loop to each frame and saving it in the created folder
-        while(cap.isOpened()):
-            count+=1
-            if length==count:
+        while cap.isOpened():
+            count += 1
+            if length == count:
                 break
             ret, frame = cap.read()
             if frame is None:
                 continue
 
             ## Resizing it to 256*256 to save the disk space and fit into the model
-            frame = cv2.resize(frame,(256, 256), interpolation = cv2.INTER_CUBIC)
+            frame = cv2.resize(frame, (256, 256), interpolation=cv2.INTER_CUBIC)
             # Saves image of the current frame in jpg file
-            name = 'ImageData/validationData/'+str(file_name)+'/frame' + str(count) + '.jpg'
+            name = (
+                "ImageData/validationData/"
+                + str(file_name)
+                + "/frame"
+                + str(count)
+                + ".jpg"
+            )
             cv2.imwrite(name, frame)
-            
-            if cv2.waitKey(1) & 0xFF == ord('q'):
+
+            if cv2.waitKey(1) & 0xFF == ord("q"):
                 break
 
         ## Print the file which is done
-        print (zipfilename, ':', file_name) 
+        print(zipfilename, ":", file_name)
+
+### Validation data ###