Add images

neural-nets/A_Neural_Algorithm_for_Artistic_Style.md

@@ -37,10 +37,11 @@
     * Add up both components to get the total loss.
     * Give both components a weight to alter for more/less style matching (at the expense of content matching).
 
-Example images:
 
 ![Examples](images/A_Neural_Algorithm_for_Artistic_Style__examples.jpg?raw=true "Examples")
 
+*One example input image with different styles added to it.*
+
 -------------------------
 
 # Rough chapter-wise notes

neural-nets/Batch_Normalization.md

@@ -29,6 +29,11 @@
   * BN allows higher learning rates. (Because of less danger of exploding/vanishing gradients.)
   * BN enables training with saturating nonlinearities in deep networks, e.g. sigmoid. (Because the normalization prevents them from getting stuck in saturating ranges, e.g. very high/low values for sigmoid.)
 
+
+![MNIST and neuron activations](images/Batch_Normalization__performance_and_activations.png?raw=true "MNIST and neuron activations")
+
+*BN applied to MNIST (a), and activations of a randomly selected neuron over time (b, c), where the middle line is the median activation, the top line is the 15th percentile and the bottom line is the 85th percentile.*
+
 -------------------------
 
 # Rough chapter-wise notes

neural-nets/Deep_Residual_Learning_for_Image_Recognition.md

@@ -50,6 +50,16 @@
   * They enable well-trainable very deep architectures with up to 1000 layers and more.
   * The activations of the residual layers are low compared to plain networks. That indicates that the residual networks indeed only learn to make "good" changes and default to "if in doubt, change nothing".
 
+
+![Building blocks](images/Deep_Residual_Learning_for_Image_Recognition__building_blocks.png?raw=true "Building blocks")
+
+*Examples of basic building blocks (other architectures are possible). The paper doesn't discuss the placement of the ReLU (after add instead of after the layer).*
+
+
+![Activations](images/Deep_Residual_Learning_for_Image_Recognition__activations.png?raw=true "Activations")
+
+*Activations of layers (after batch normalization, before nonlinearity) throughout the network for plain and residual nets. Residual networks have on average lower activations.*
+
 -------------------------
 
 # Rough chapter-wise notes

neural-nets/Fractional_Max_Pooling.md

@@ -30,6 +30,11 @@
   * Results are especially better if each test is repeated multiple times per image (as the random sequence generation creates randomness, similar to dropout). First 5-10 repetitions seem to be most valuable, but even 100+ give some improvement.
   * An FMP-factor of `sqrt(2)` was usually used.
 
+
+![Examples](images/Fractional_Max_Pooling__examples.jpg?raw=true "Examples")
+
+*Random FMP with a factor of sqrt(2) applied five times to the same input image (results upscaled back to original size).*
+
 -------------------------
 
 # Rough chapter-wise notes

neural-nets/Weight_Normalization.md

@@ -55,3 +55,12 @@
   * Due to some gradient effects, `||v||` currently grows monotonically with every weight update. (Not necessarily when using optimizers that use separate learning rates per parameters.)
   * That grow effect leads the network to be more robust to different learning rates.
   * Setting a small hard limit/constraint for `||v||` can lead to better test set performance (parameter updates are larger, introducing more noise).
+
+
+![CIFAR-10 results](images/Weight_Normalization__cifar10.png?raw=true "CIFAR-10 results")
+
+*Performance of WN on CIFAR-10 compared to BN, BN-MEAN and no normalization.*
+
+![DRAW, DQN results](images/Weight_Normalization__draw_dqn.png?raw=true "DRAW, DQN results")
+
+*Performance of WN for DRAW (left) and deep reinforcement learning (right).*