Update README.md and results/figures of "cold posterior" repository.

PiperOrigin-RevId: 297791189
xujinjin1 · Feb 28, 2020 · 5032e3b · 5032e3b
1 parent 6aa58f4
commit 5032e3b
Show file tree

Hide file tree

Showing 14 changed files with 6,026 additions and 20 deletions.
diff --git a/cold_posterior_bnn/README.md b/cold_posterior_bnn/README.md
@@ -17,11 +17,11 @@ Instead, we argue that it is timely to focus on understanding the origin of the
 
 ### Cold posteriors
 
-This repository contains code to reproduces the experiments from the [paper](https://arxiv.org/pdf/2002.02405.pdf) that demonstrate _cold posterior_ effect
+This repository contains code to reproduce the experiments from the [paper](https://arxiv.org/pdf/2002.02405.pdf) that demonstrate _cold posterior_ effect
 for a ResNet-20 model on the dataset CIFAR-10 and a CNN-LSTM model on IMDB sentiment
 dataset. We can improve the generalization performance significantly by
 cooling the posterior with a temperature T<<1. The cold posterior sharply deviates
-from the true Bayes postterior (which is attained for T=1).
+from the true Bayes posterior (which is attained for T=1).
 
 ResNet-20 / CIFAR-10      |   CNN-LSTM / IMDB
 :------------------------:|:-------------------------:
@@ -48,7 +48,13 @@ To reproduce the ResNet-20 experiment from the paper, run the following command.
 cold_posterior_bnn/run_resnet_experiment.sh
 ```
 
-For a simplified version of the experiment that only takes 1-2 days on a single GPU, run
+For a simplified version of the experiment that only takes 1-2 days on a single GPU,
+run the following command. *For the simplified experiment we halved the number of epochs
+and included less temperature evaluations. Due to the lower number of total epochs
+the individual runs are not all converged and the final performance is worse than in the
+full experiment. However, this experiment still shows the cold posterior effect.*
+
+![Test accuracy ResNet-20 in the simplified experiment setting](results_resnet/resnet_simplified_run_test_accuracy.png)
 
 ```sh
 cold_posterior_bnn/run_resnet_experiment_small.sh
@@ -61,7 +67,7 @@ notebook
 cold_posterior_bnn/plot_results.ipynb
 ```
 
-The results are stored in the folder ```cold_posterior_bnn/results_resnet```.
+The results of the simplified ResNet experiment are stored in the folder ```cold_posterior_bnn/results_resnet```.
 
 
 **CNN-LSTM experiment**
@@ -102,13 +108,13 @@ Sebastian Nowozin ([[email protected]]([email protected]))
 > Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton and Sebastian
 > Nowozin (2020).
 > [How Good is the Bayes Posterior in Deep Neural Networks Really?](https://arxiv.org/pdf/2002.02405.pdf).
-> In _arXiv preprint arXiv:12002.02405_.
+> In _arXiv preprint arXiv:2002.02405_.
 
 ```none
 @article{wenzel2020good,
   author = {Florian Wenzel and Kevin Roth and Bastiaan S. Veeling and Jakub Swiatkowski and Linh Tran and Stephan Mandt and Jasper Snoek and Tim Salimans and Rodolphe Jenatton and Sebastian Nowozin},
   title = {How Good is the Bayes Posterior in Deep Neural Networks Really?},
-  journal={arXiv preprint arXiv:12002.02405},
+  journal={arXiv preprint arXiv:2002.02405},
   year = {2020},
 }
 ```
diff --git a/cold_posterior_bnn/requirements.txt b/cold_posterior_bnn/requirements.txt
@@ -3,6 +3,6 @@ tqdm>=4.43.0
 numpy>=1.16.4
 scipy>=1.3.0
 tensorflow>=2.1.0
-tensorflow-datasets==2.0.0
-tensorflow-probability==0.8.0
+tensorflow-datasets>=2.0.0
+tensorflow-probability>=0.9.0
 pandas>=0.24.0
diff --git a/...tm/resnet_test_cross_entropy_full_run.png → ...nnlstm/cnnlstm_test_accuracy_full_run.png b/...tm/resnet_test_cross_entropy_full_run.png → ...nnlstm/cnnlstm_test_accuracy_full_run.png
diff --git a/cold_posterior_bnn/results_resnet/resnet_simplified_run_test_accuracy.png b/cold_posterior_bnn/results_resnet/resnet_simplified_run_test_accuracy.png
diff --git a/cold_posterior_bnn/results_resnet/resnet_simplified_run_test_cross_entropy.png b/cold_posterior_bnn/results_resnet/resnet_simplified_run_test_cross_entropy.png
diff --git a/cold_posterior_bnn/results_resnet/run_0/logs.csv b/cold_posterior_bnn/results_resnet/run_0/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_1/logs.csv b/cold_posterior_bnn/results_resnet/run_1/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_2/logs.csv b/cold_posterior_bnn/results_resnet/run_2/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_3/logs.csv b/cold_posterior_bnn/results_resnet/run_3/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_4/logs.csv b/cold_posterior_bnn/results_resnet/run_4/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_5/logs.csv b/cold_posterior_bnn/results_resnet/run_5/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_6/logs.csv b/cold_posterior_bnn/results_resnet/run_6/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_7/logs.csv b/cold_posterior_bnn/results_resnet/run_7/logs.csv
diff --git a/cold_posterior_bnn/results_resnet/run_sweeps.csv b/cold_posterior_bnn/results_resnet/run_sweeps.csv
@@ -1,5 +1,9 @@
 id,seed,temperature,dir,ens_acc,ens_ce
-0,1,0.1,run_0,0.5611,1.2148816999999998
-1,1,1.0,run_1,0.5631,1.216646
-2,2,0.1,run_2,0.5916,1.1341463
-3,2,1.0,run_3,0.5769,1.1686572
+0,1,0.0001,run_0,0.9071,0.41457877
+1,1,0.001,run_1,0.9086,0.37458166
+2,1,0.01,run_2,0.9144,0.34805304
+3,1,0.1,run_3,0.9082,0.29580609999999996
+4,1,0.177828,run_4,0.9063,0.2917594
+5,1,0.316228,run_5,0.9007,0.28919944
+6,1,0.562341,run_6,0.8931,0.31784263
+7,1,1.0,run_7,0.873,0.38322848