Introduction

These are my personal notes while working through the Dive into Deep Learning Book

Exercise Progress

2. 🌟 Preliminaries

2.1. 🌟 NDArray ✅✅

2.2. 🌟 Pandas ✅✅✅✅✅

2.3. 🌟 Linear Algebra ✅✅✅✅✅✅✅✅✅✅✅✅

2.4. 🌟 Calculus ✅✅✅✅✅✅✅✅✅✅

2.5. 🌟 Automatic Differentiation ✅✅✅✅✅✅

2.6. 🌟 Probability and Statistics ✅✅✅✅✅✅✅✅

2.7. Documentation

3. 🌟 Linear Neural Networks for Regression

3.1. 🌟 Linear Regression ✅✅✅✅✅✅✅✅

3.2. 🌟 Object-Oriented Design for Implementation ✅✅

3.3. 🌟 Synthetic Regression Data ✅✅✅✅

3.4. 🌟 Linear Regression Implementation from Scratch ✅✅✅✅✅✅✅✅✅

3.5. 🌟 Concise Implementation of Linear Regression ✅✅✅✅✅

3.6. 🌟 Generalizations ✅✅✅✅✅✅✅

3.7. 🌟 Weight Decay ✅✅✅✅✅✅

4. 🌟 Linear Neural Networks for Classification

4.1. 🌟 Softmax Regression ✅✅✅✅✅✅✅

4.2. 🌟 The Image Classification Dataset ✅✅✅

4.3. 🌟 The Base Classification Model ✅✅✅

4.4. 🌟 Softmax Regression Implementation from Scratch ✅✅✅✅✅

4.5. 🌟 Concise Implementation of Softmax Regression ✅✅✅✅

4.6. 🌟 Generalization in Classification ✅✅✅✅

4.7. 🌟 Environment and Distribution Shift ✅✅✅✅

5. Multilayer Perceptrons

5.1. ⭐ Multilayer Perceptrons ✅✅✅✅✅✅✅

5.2. Implementations of Multilayer Perceptrons ❌❌✅✅❌❌✅❌✅

5.3. ⭐ Forward Propagation, Backward Propagation, and Computational Graphs ✅✅✅✅✅

5.4. Numerical Stability and Initialization

5.5. Generalizing in Deep Learning

5.6. Dropout

5.7. Predicting House Prices on Kaggle ✅❌❌❌❌

6. 🌟 Builders' Guide

6.1. 🌟 Layers and Modules ✅✅✅

6.2. 🌟 Parameter Management ✅✅✅

6.3. 🌟 Parameter Initialization ✅

6.4. 🌟 Lazy Initialization ✅✅✅

6.5. 🌟 Custom Layers ✅✅

6.6. 🌟 File I/O ✅✅✅

6.7. 🌟 GPUs (M1 GPUs are not supported so I wont't do these exercises.)

7. Convolutional Neural Networks

7.1. From Fully Connected Layers to Convolutions ❌❌✅✅✅✅

7.2. Convolutions for Images ✅❌❌❌

7.3. Padding and Stride

7.4. Multiple Input and Multiple Output Channels

7.5. Pooling

8. Modern Convolutional Neural Networks

8.1. Deep Convolutional Neural Networks (AlexNet)

8.2. Networks Using Blocks (VGG)

8.3. Network in Network (NiN)

8.4. Multi-Branch Networks (GoogLeNet)

8.5. Batch Normalization

8.6. Residual Networks (ResNet) and ResNeXt

8.7. Densely Connected Networks (DenseNet)

8.8. Designing Convolutional Network Architectures

9. Recurrent Neural Networks

9.1. Working with Sequences ❌❌❌❌

9.2. ⭐ Converting Raw Data into Sequence Data ✅✅✅

9.3. Language Models

9.4. Recurrent Neural Networks

9.5. Recurrent Neural Network Implementation from Scratch

9.6. Concise Implementation of Recurrent Neural Networks

9.7. Backpropagation Through Time

10. Modern Recurrent Neural Networks

10.1. Long Short-Term Memory (LSTM)

10.2. Gated Recurrent Units (GRU)

10.3. Deep Recurrent Neural Networks

10.4. Bidirectional Recurrent Neural Networks

10.5. Machine Translation and the Dataset

10.6. The Encoder–Decoder Architecture

10.7. Sequence-to-Sequence Learning for Machine Translation

10.8. Beam Search

11. Attention Mechanisms and Transformers

11.1. Queries, Keys, and Values

11.2. Attention Pooling by Similarity

11.3. Attention Scoring Functions

11.4. The Bahdanau Attention Mechanism

11.5. Multi-Head Attention

11.6. Self-Attention and Positional Encoding

11.7. The Transformer Architecture

11.8. Transformers for Vision

11.9. Large-Scale Pretraining with Transformers

12. Optimization and Deep Learning

12.1. Optimization and Deep Learning

12.2. Convexity ✅❌✅❌✅✅✅❌

12.3. Gradient Descent ✅❌✅❌❌

12.4. Stochastic Gradient Descent

12.5. Minibatch Stochastic Gradient Descent

12.6. Momentum

12.7. Adagrad

12.8. RMSProp

12.9. Adadelta

12.10. Adam

12.11. Learning Rate Scheduling

13. Computational Performance

13.1. Compilers and Interpreters

13.2. Asynchronous Computation

13.3. Automatic Parallelism

13.4. Hardware

13.5. Training on Multiple GPUs

13.6. Concise Implementation for Multiple GPUs

13.7. Parameter Servers

14. Computer Vision

14.1. Image Augmentation

14.2. Fine-Tuning

14.3. Object Detection and Bounding Boxes

14.4. Anchor Boxes

14.5. Multiscale Object Detection

14.6. The Object Detection Dataset

14.7. Single Shot Multibox Detection

14.8. Region-based CNNs (R-CNNs)

14.9. Semantic Segmentation and the Dataset

14.10. Transposed Convolution

14.11. Fully Convolutional Networks

14.12. Neural Style Transfer

14.13. Image Classification (CIFAR-10) on Kaggle

14.14. Dog Breed Identification (ImageNet Dogs) on Kaggle

15. Natural Language Processing: Pretraining

15.1. Word embedding (Word2Vec) ❌❌❌

15.2. Approximate Training

15.3. The Dataset for Pretraining Word Embeddings

15.4. Pretraining Word2Vec

15.5. Word Embedding with Global Vectors (GloVe)

15.6. Subword Embedding

15.7. Word Similarity and Analogy

15.8. Bidirectional Encoder Representation from Transformers (BERT)

15.9. The Dataset for pretraining BERT

15.10. Pretraining BERT

16 Natural Language Processing: Applications

16.1. Sentiment Analysis and the Dataset

16.2. Sentiment Analysis: Using Recurrent Neural Networks

16.3. Sentiment Analysis: Using Convolutional Neural Networks

16.4. Natural Language Inference and the Dataset

16.5. Natural Language Inference: Using Attention

16.6. Fine-Tuning BERT for Sequence-Level and Token-Level Applications

16.7. Natural Language Inference: Fine-Tuning BERT

17. 🌟 Reinforcement Learning

17.1. 🌟 Markov Decision Process (MDP) ✅✅

17.2. 🌟 Value Iteration ✅✅✅✅

17.3. 🌟 Q-Learning ✅✅✅

18. Gaussian Processes

18.1. Introduction to Gaussian Processes

18.2. Gaussian Process Priors

18.3. Gaussian Process Inference

19. Hyperparameter Optimization

19.1. What is Hyperparameter Optimization?

19.2. Hyperparameter Optimization API

19.3. Asynchronous Random Search

19.4. Multi-Fidelity Hyperparameter Optimization

19.5. Asynchronous Successive Halving

20. Generative Adversarial Networks

20.1. Generative Adversarial Networks

20.2. Deep Convolutional Generative Adversarial Networks

21. Recommender Systems

21.1. Overview of Recommender Systems

21.2. The MoveLens Dataset

21.3. Matrix Factorization

21.4. AutoRec: Rating Prediction with Autoencoders

21.5. Personalized Ranking for Recommender Systems

21.6. Neural Collaborative Filtering for Personalized Ranking

21.7. Sequence-Aware Recommender Systems

21.8. Feature-Rich Recommender Systems

21.9. Factorization Machines

21.10. Deep Factorization Machines

22. Appendix: Mathematics for Deep Learning

22.1. ⭐ Geometry and Linear Algebric Operations ✅✅✅✅✅✅✅

22.2. ⭐ Eigendecompositions ✅✅✅

22.3. ⭐ Single Variable Calculus ✅✅✅✅

22.4. ⭐ Multivariable Calculus ✅✅✅

22.5. ⭐ Integral Calculus ✅✅✅✅

22.6. ⭐ Random Variables ✅✅✅✅

22.7. ⭐ Maximum Likelihood ✅✅

22.8. ⭐ Distributions ✅✅✅

22.9. Naive Bayes ❌❌❌

22.10. Statistics ❌❌❌

22.11. Information Theory ❌❌❌❌❌

23. 🌟 Appendix: Tools for Deep Learning

23.1. 🌟 Using Jupyter Notebooks ✅✅✅

23.2. 🌟 Using Amazon SageMaker ✅✅

23.3. 🌟 Using AWS EC2 Instances ✅✅✅

23.4. 🌟 Using Google Collab ✅✅

23.5. Selecting Servers and GPUs

23.6. 🌟 Contributing to This Book ✅✅✅

23.7. Utility Functions and Classes

23.8. The d2l API Document

Technical Stuff

Development Environment

I created a seperate miniconda environment for this repo called "d2l", based on python version 3.9.18.

The additionally added packages (mostly just linting stuff) are entered in the requirements.txt. This is the first time I'm using (mini)conda, next time I'm creating a repo that uses miniconda I'd create a local (mini)conda environment just how I'd do with a .venv file, but now that I works there is no need to reinstall everything just to have it localized. Might change my mind if linking errors appear in the future.

Modifications of Libraries

Given that I'm the only person working on this repo and there is very likely not going to be any updates to the packages I think it's okay for me to directly modify the packages, mostly to remove things like deprecation and "Work In Progress" type warnings. This has the additional benefit that I don't have to suppress warnings on the module level but instead can deal with them one by one.

PyTorch

Inside of torch/nn/modules/lazy.py I've commented the lazy eval warning in line 180-181.

Setup Code and Framework specific stuff

I will use PyTorch as the framework while working through the book.

To get the (PyTorch) files you can run the get_pytorch_files.sh shell script.

Licensing

This repository contains modified versions of the notebooks from the Dive into Deep Learning book, originally created by the D2L team.

The original notebooks are licensed under the Creative Commons Attribution-ShareAlike 4.0 International Public License, and my modifications are also licensed under the same CC BY-SA 4.0 license.

Attribution:

• The D2L Team
• https://github.com/d2l-ai/d2l-en

My modifications to the notebooks are indicated with comments or other appropriate markers.

Disclaimer: I have no experience with this kind of licensing attribution, so if something is mishandeled or missing, please let me know.