Merge pull request #174 from philipperemy/weight-norm

Weight norm

Author: philipperemy

README.md

@@ -227,21 +227,19 @@ The task consists of feeding a large array of decimal numbers to the network, al
 
 #### Implementation results
 
-The model takes time to learn this task. It's symbolized by a very long plateau (could take ~8 epochs on some runs).
-
 ```
-200000/200000 [==============================] - 293s 1ms/step - loss: 0.1731 - val_loss: 0.1662
-200000/200000 [==============================] - 289s 1ms/step - loss: 0.1675 - val_loss: 0.1665
-200000/200000 [==============================] - 287s 1ms/step - loss: 0.1670 - val_loss: 0.1665
-200000/200000 [==============================] - 288s 1ms/step - loss: 0.1668 - val_loss: 0.1669
-200000/200000 [==============================] - 285s 1ms/step - loss: 0.1085 - val_loss: 0.0019
-200000/200000 [==============================] - 285s 1ms/step - loss: 0.0011 - val_loss: 4.1667e-04
-200000/200000 [==============================] - 282s 1ms/step - loss: 6.0470e-04 - val_loss: 6.7708e-04
-200000/200000 [==============================] - 282s 1ms/step - loss: 4.3099e-04 - val_loss: 7.3898e-04
-200000/200000 [==============================] - 282s 1ms/step - loss: 3.9102e-04 - val_loss: 1.8727e-04
-200000/200000 [==============================] - 280s 1ms/step - loss: 3.1040e-04 - val_loss: 0.0010
-200000/200000 [==============================] - 281s 1ms/step - loss: 3.1166e-04 - val_loss: 2.2333e-04
-200000/200000 [==============================] - 281s 1ms/step - loss: 2.8046e-04 - val_loss: 1.5194e-04
+782/782 [==============================] - 154s 197ms/step - loss: 0.8437 - val_loss: 0.1883
+782/782 [==============================] - 154s 196ms/step - loss: 0.0702 - val_loss: 0.0111
+782/782 [==============================] - 153s 195ms/step - loss: 0.0053 - val_loss: 0.0038
+782/782 [==============================] - 154s 196ms/step - loss: 0.0035 - val_loss: 0.0027
+782/782 [==============================] - 153s 196ms/step - loss: 0.0030 - val_loss: 0.0065
+782/782 [==============================] - 151s 193ms/step - loss: 0.0027 - val_loss: 0.0018
+782/782 [==============================] - 152s 194ms/step - loss: 0.0025 - val_loss: 0.0036
+782/782 [==============================] - 153s 196ms/step - loss: 0.0024 - val_loss: 0.0018
+782/782 [==============================] - 152s 194ms/step - loss: 0.0023 - val_loss: 0.0016
+782/782 [==============================] - 152s 194ms/step - loss: 0.0014 - val_loss: 3.7456e-04
+782/782 [==============================] - 153s 196ms/step - loss: 9.4740e-04 - val_loss: 7.0205e-04
+782/782 [==============================] - 152s 194ms/step - loss: 6.9630e-04 - val_loss: 3.7180e-04
 ```
 
 ### Copy Memory Task
@@ -263,13 +261,14 @@ The idea is to copy the content of the vector x to the end of the large array. T
 #### Implementation results (first epochs)
 
 ```
-30000/30000 [==============================] - 30s 1ms/step - loss: 0.1174 - acc: 0.9586 - val_loss: 0.0370 - val_acc: 0.9859
-30000/30000 [==============================] - 26s 874us/step - loss: 0.0367 - acc: 0.9859 - val_loss: 0.0363 - val_acc: 0.9859
-30000/30000 [==============================] - 26s 852us/step - loss: 0.0361 - acc: 0.9859 - val_loss: 0.0358 - val_acc: 0.9859
-30000/30000 [==============================] - 26s 872us/step - loss: 0.0355 - acc: 0.9859 - val_loss: 0.0349 - val_acc: 0.9859
-30000/30000 [==============================] - 25s 850us/step - loss: 0.0339 - acc: 0.9864 - val_loss: 0.0291 - val_acc: 0.9881
-30000/30000 [==============================] - 26s 856us/step - loss: 0.0235 - acc: 0.9896 - val_loss: 0.0159 - val_acc: 0.9944
-30000/30000 [==============================] - 26s 872us/step - loss: 0.0169 - acc: 0.9929 - val_loss: 0.0125 - val_acc: 0.9966
+118/118 [==============================] - 17s 143ms/step - loss: 1.1732 - accuracy: 0.6725 - val_loss: 0.1119 - val_accuracy: 0.9796
+118/118 [==============================] - 15s 125ms/step - loss: 0.0645 - accuracy: 0.9831 - val_loss: 0.0402 - val_accuracy: 0.9853
+118/118 [==============================] - 15s 125ms/step - loss: 0.0393 - accuracy: 0.9856 - val_loss: 0.0372 - val_accuracy: 0.9857
+118/118 [==============================] - 15s 125ms/step - loss: 0.0361 - accuracy: 0.9858 - val_loss: 0.0344 - val_accuracy: 0.9860
+118/118 [==============================] - 15s 125ms/step - loss: 0.0345 - accuracy: 0.9860 - val_loss: 0.0335 - val_accuracy: 0.9864
+118/118 [==============================] - 15s 125ms/step - loss: 0.0325 - accuracy: 0.9867 - val_loss: 0.0268 - val_accuracy: 0.9886
+118/118 [==============================] - 15s 125ms/step - loss: 0.0268 - accuracy: 0.9885 - val_loss: 0.0206 - val_accuracy: 0.9908
+118/118 [==============================] - 15s 125ms/step - loss: 0.0228 - accuracy: 0.9900 - val_loss: 0.0169 - val_accuracy: 0.9933
 ```
 
 ### Sequential MNIST
@@ -286,11 +285,16 @@ The idea here is to consider MNIST images as 1-D sequences and feed them to the
 #### Implementation results
 
 ```
-60000/60000 [==============================] - 118s 2ms/step - loss: 0.2348 - acc: 0.9265 - val_loss: 0.1308 - val_acc: 0.9579
-60000/60000 [==============================] - 116s 2ms/step - loss: 0.0973 - acc: 0.9698 - val_loss: 0.0645 - val_acc: 0.9798
-[...]
-60000/60000 [==============================] - 112s 2ms/step - loss: 0.0075 - acc: 0.9978 - val_loss: 0.0547 - val_acc: 0.9894
-60000/60000 [==============================] - 111s 2ms/step - loss: 0.0093 - acc: 0.9968 - val_loss: 0.0585 - val_acc: 0.9895
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0949 - accuracy: 0.9706 - val_loss: 0.0763 - val_accuracy: 0.9756
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0831 - accuracy: 0.9743 - val_loss: 0.0656 - val_accuracy: 0.9807
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0752 - accuracy: 0.9763 - val_loss: 0.0604 - val_accuracy: 0.9802
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0685 - accuracy: 0.9785 - val_loss: 0.0588 - val_accuracy: 0.9813
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0624 - accuracy: 0.9801 - val_loss: 0.0545 - val_accuracy: 0.9822
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0603 - accuracy: 0.9812 - val_loss: 0.0478 - val_accuracy: 0.9835
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0566 - accuracy: 0.9821 - val_loss: 0.0546 - val_accuracy: 0.9826
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0503 - accuracy: 0.9843 - val_loss: 0.0441 - val_accuracy: 0.9853
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0486 - accuracy: 0.9840 - val_loss: 0.0572 - val_accuracy: 0.9832
+1875/1875 [==============================] - 46s 25ms/step - loss: 0.0453 - accuracy: 0.9858 - val_loss: 0.0424 - val_accuracy: 0.9862
 ```
 
 ## Testing

setup.py

@@ -10,6 +10,6 @@
     long_description=open('README.md').read(),
     packages=['tcn'],
     install_requires=[
-        'numpy', 'tensorflow'
+        'numpy', 'tensorflow', 'tensorflow_addons'
     ]
 )

tasks/adding_problem/main.py

@@ -1,44 +1,39 @@
-import keras
 import numpy as np
-
-from tcn import compiled_tcn
+from tensorflow.keras.callbacks import Callback
 from utils import data_generator
 
+from tcn import compiled_tcn, tcn_full_summary
+
 x_train, y_train = data_generator(n=200000, seq_length=600)
 x_test, y_test = data_generator(n=40000, seq_length=600)
 
 
-class PrintSomeValues(keras.callbacks.Callback):
+class PrintSomeValues(Callback):
 
     def on_epoch_begin(self, epoch, logs={}):
         print('y_true, y_pred')
         print(np.hstack([y_test[:5], self.model.predict(x_test[:5])]))
 
 
 def run_task():
-    model = compiled_tcn(return_sequences=False,
-                         num_feat=x_train.shape[2],
-                         num_classes=0,
-                         nb_filters=24,
-                         kernel_size=8,
-                         dilations=[2 ** i for i in range(9)],
-                         nb_stacks=1,
-                         max_len=x_train.shape[1],
-                         use_skip_connections=False,
-                         regression=True,
-                         dropout_rate=0)
-
-    print(f'x_train.shape = {x_train.shape}')
-    print(f'y_train.shape = {y_train.shape}')
-
-    psv = PrintSomeValues()
-
-    # Using sparse softmax.
-    # http://chappers.github.io/web%20micro%20log/2017/01/26/quick-models-in-keras/
-    model.summary()
-
+    model = compiled_tcn(
+        return_sequences=False,
+        num_feat=x_train.shape[2],
+        num_classes=0,
+        nb_filters=24,
+        kernel_size=8,
+        dilations=[2 ** i for i in range(9)],
+        nb_stacks=1,
+        max_len=x_train.shape[1],
+        use_skip_connections=False,
+        use_weight_norm=True,
+        regression=True,
+        dropout_rate=0
+    )
+
+    tcn_full_summary(model)
     model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=15,
-              batch_size=256, callbacks=[psv])
+              batch_size=256, callbacks=[PrintSomeValues()])
 
 
 if __name__ == '__main__':

tasks/copy_memory/main.py

@@ -1,7 +1,7 @@
 from uuid import uuid4
 
-import keras
 import numpy as np
+from tensorflow.keras.callbacks import Callback
 
 from tcn import compiled_tcn
 from utils import data_generator
@@ -10,7 +10,7 @@
 x_test, y_test = data_generator(601, 10, 6000)
 
 
-class PrintSomeValues(keras.callbacks.Callback):
+class PrintSomeValues(Callback):
 
     def on_epoch_begin(self, epoch, logs={}):
         print('y_true')
@@ -30,6 +30,7 @@ def run_task():
                          use_skip_connections=True,
                          opt='rmsprop',
                          lr=5e-4,
+                         use_weight_norm=True,
                          return_sequences=True)
 
     print(f'x_train.shape = {x_train.shape}')

tasks/mnist_pixel/main.py

@@ -14,6 +14,7 @@ def run_task():
                          dilations=[2 ** i for i in range(9)],
                          nb_stacks=1,
                          max_len=x_train[0:1].shape[1],
+                         use_weight_norm=True,
                          use_skip_connections=True)
 
     print(f'x_train.shape = {x_train.shape}')

tasks/mnist_pixel/utils.py

@@ -1,6 +1,6 @@
 import numpy as np
-from keras.datasets import mnist
-from keras.utils import to_categorical
+from tensorflow.keras.datasets import mnist
+from tensorflow.keras.utils import to_categorical
 
 
 def data_generator():

tasks/tcn_call_test.py

@@ -3,6 +3,7 @@
 import numpy as np
 from tensorflow.keras import Input
 from tensorflow.keras import Model
+from tensorflow.keras.models import Sequential
 
 from tcn import TCN
 
@@ -99,6 +100,27 @@ def test_non_causal_time_dim_unknown_return_no_sequences(self):
         r = predict_with_tcn(time_steps=None, padding='same', return_sequences=False)
         self.assertListEqual([list(b.shape) for b in r], [[1, NB_FILTERS], [1, NB_FILTERS], [1, NB_FILTERS]])
 
+    def test_norms(self):
+        Sequential(layers=[TCN(input_shape=(20, 2), use_weight_norm=True)]).compile(optimizer='adam', loss='mse')
+        Sequential(layers=[TCN(input_shape=(20, 2), use_weight_norm=False)]).compile(optimizer='adam', loss='mse')
+        Sequential(layers=[TCN(input_shape=(20, 2), use_layer_norm=True)]).compile(optimizer='adam', loss='mse')
+        Sequential(layers=[TCN(input_shape=(20, 2), use_layer_norm=False)]).compile(optimizer='adam', loss='mse')
+        Sequential(layers=[TCN(input_shape=(20, 2), use_batch_norm=True)]).compile(optimizer='adam', loss='mse')
+        Sequential(layers=[TCN(input_shape=(20, 2), use_batch_norm=False)]).compile(optimizer='adam', loss='mse')
+        try:
+            Sequential(layers=[TCN(input_shape=(20, 2), use_batch_norm=True, use_weight_norm=True)]).compile(
+                optimizer='adam', loss='mse')
+            raise AssertionError('test failed.')
+        except ValueError:
+            pass
+        try:
+            Sequential(layers=[TCN(input_shape=(20, 2), use_batch_norm=True,
+                                   use_weight_norm=True, use_layer_norm=True)]).compile(
+                optimizer='adam', loss='mse')
+            raise AssertionError('test failed.')
+        except ValueError:
+            pass
+
 
 if __name__ == '__main__':
     unittest.main()