Release 1.0.3

Dive into Deep Learning is now v1.0!

The Vol.1 by Cambridge University Press can be ordered on Amazon.

Thanks to all the readers and contributors in the past years!

Release v1.0.0-beta0

D2L has gone 1.0.0-beta0! We thank all the 296 contributors for making this happen!

Forthcoming on Cambridge University Press

Chapter 1--11 will be forthcoming on Cambridge University Press (early 2023).

New JAX Implementation

We added new JAX implementation. Get started with import jax at https://d2l.ai/chapter_preliminaries/ndarray.html

Thank @AnirudhDagar!

New Vol.2 Chapter on Reinforcement Learning

With the advent of ChatGPT (sibling model to InstructGPT fine-tuned using reinforcement learning), you may get curious about how to enable ML to take decisions sequentially:

17. Reinforcement Learning
17.1. Markov Decision Process (MDP)
17.2. Value Iteration
17.3. Q-Learning

Thank Pratik Chaudhari (University of Pennsylvania and Amazon), Rasool Fakoor @rasoolfa (Amazon), and Kavosh Asadi (Amazon)!

New Vol.2 Chapter on Gaussian Processes

“Everything is a special case of a Gaussian process.” Gaussian processes and deep neural networks are highly complementary and can be combined to great effect:

18. Gaussian Processes
18.1. Introduction to Gaussian Processes
18.2. Gaussian Process Priors
18.3. Gaussian Process Inference

Thank Andrew Gordon Wilson @andrewgordonwilson (New York University and Amazon)!

New Vol.2 Chapter on Hyperparameter Optimization

Tired of setting hyperparameters in a trial-and-error manner? You may wish to check out the systematic hyperparameter optimization approach:

19. Hyperparameter Optimization
19.1. What Is Hyperparameter Optimization?
19.2. Hyperparameter Optimization API
19.3. Asynchronous Random Search
19.4. Multi-Fidelity Hyperparameter Optimization
19.5. Asynchronous Successive Halving

Thank Aaron Klein @aaronkl (Amazon), Matthias Seeger @mseeger (Amazon), and Cedric Archambeau (Amazon)!

Fixes and Improvements

Thank @finale80 @JojiJoseph @gab-chen @Excelsior7 @shanmo @kxxt @vBarbaros @gui-miotto @BOLDED @atishaygarg @tuelwer @gopalakrishna-r @qingfengtommy @Mohamad-Jaallouk @biswajitsahoo1111 @315930399 for improving this book!

Release 0.17.6

Add PaddlePaddle implementation for the d2l library (compatible with classic.d2l.ai).
See the PR in d2l-zh.

Release v1.0.0-alpha1.post0

We are happy to release D2L 1.0.0-alpha1.post0 We thank all the contributors who have made this open-source textbook better for everyone.

This minor release includes the following updates:

• Build PDFs using Cambridge Latex Style (#2187)
• Feat: Add dynamic preview/stable version tab (#2264)
• Chapter Modern RNN is heavily improved, refactoring the text and reorganizing content (#2241)
• Chapter Modern CNN is refactored and is more polished (#2249)
• Bump supported version to Python3.9 (#2231)

It also comes with the following bug-fixes:

• Fix #2250: Depend on matplotlib-inline instead of ipython to avoid colab warning (#2279)
• Fix broken preview version pdf links (#2264)
• Fix #2247: Tensorflow explicitly squeeze image to support matplotlib<3.3.0 (#2248)
• Fix PyTorch moving average computation for batch norm (#2213)
• Fix PyTorch module types (LazyLinear -> Linear) (#2225)
• Fix QR code being overridden by the section rule issue (#2251)
• Fix torch.meshgrid user warning (7d92155)

Release v1.0.0-alpha0

We are excited to announce the release of D2L 1.0.0-alpha0! We thank all the 265 contributors who have made this open-source textbook better for everyone.

New Topics and Revision

We have added the following new topics, with discussions of more recent methods such as ResNeXt, RegNet, ConvNeXt, Vision Transformer, Swin Transformer, T5, GPT-1/2/3, zero-shot, one-shot, few-shot, Gato, Imagen, Minerva, and Parti.

• Object-Oriented Design for Implementation
• Synthetic Regression Data
• Generalization
• The Base Classification Model
• Generalization in Classification
• Generalization in Deep Learning
• ResNeXt
• Designing Convolution Network Architectures
• Transformers for Vision
• Large-Scale Pretraining with Transformers

Besides new topics, we have significantly revised all the topics up to transformers. For example, the previous Linear Neural Networks and Multilayer Perceptrons chapters have been revamped as new chapters of Linear Neural Networks for Regression, Linear Neural Networks for Classification, and Multilayer Perceptrons.

New API

Throughout the book we repeatedly walk through various components including the data, the model, the loss function, and the optimization algorithm. Treating components in deep learning as objects, we can define classes for these objects and their interactions. This object-oriented design for implementation will greatly streamline the presentation. Therefore, inspired by open-source libraries such as PyTorch Lightning, we have re-designed the API with three core classes:

• Module contains models, losses, and optimization methods;
• DataModule provides data loaders for training and validation;
• Both classes are combined using the Trainer class, which allows us to train models on a variety of hardware platforms.

For example, with the classic API in previous releases:

model = # Multilayer perceptron definition
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=256)
loss = nn.CrossEntropyLoss(reduction='none')
trainer = torch.optim.SGD(net.parameters(), lr=0.1)
d2l.train_ch3(model, train_iter, test_iter, loss, num_epochs=10, trainer)

With the new API:

model = # Multilayer perceptron definition
data = d2l.FashionMNIST(batch_size=256)
trainer = d2l.Trainer(max_epochs=10)
trainer.fit(model, data)

Lazy Layers in PyTorch

Since v1.8.0, PyTorch offers "lazy" layers where input shape specification is no longer required. For simplicity we will use "lazy" layers whenever we can, such as:

class LinearRegression(d2l.Module):
    def __init__(self, lr):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.LazyLinear(1)  # Lazy layer with output dimension only
        self.net.weight.data.normal_(0, 0.01)
        self.net.bias.data.fill_(0)

Ongoing Translations

Join us to improve ongoing translations in:

• Chinese (Git repo)
• Portuguese (Git repo)
• Turkish (Git repo)
• Vietnamese (Git repo)
• Korean (Git repo)
• Japanese (Git repo)

Release v0.17.5

This release fixes issues when installing d2l package and running d2l notebooks on Google Colab with Python 3.7 and updates PyTorch & TensorFlow to their respective latest versions.

More concretely, this release includes the following upgrades/fixes:

• Update TensorFlow==2.8.0 (#2055)
• Update PyTorch: torch==1.11.0 & torchvision==0.12.0 (#2063)
• Rollback NumPy==1.21.5 & Support Python>=3.7 (#2066)
• Fix MXNet plots; NumPy auto coercion & Unpin matplotlib==3.4 dependency (#2078)
• Fix the broken download link for MovieLens dataset (#2074)
• Fix iPython deprecation warning of set_matplotlib_formats (#2065)
• Fix Densenet PyTorch implementation using nn.AdaptiveAvgPool2d (f6b1dd0)
• Fix the hotdog class in section Fine Tuning for imagenet, which is number 934 instead of 713 (#2009)
• Use reduction=none in PyTorch loss for train_epoch_ch3 (#2007)
• Fix argument test_feature->test_features of train_and_pred in kaggle house price section (#1982)
• Fix TypeError: can’t convert CUDA tensor to numpy, explicitly moving torch tensor to cpu before plotting (#1966)

Release v0.17.1

This release supports running the book with SageMaker Studio Lab for free and introduces several fixes:

• Fix data synchronization for multi-GPU training in PyTorch (#1978)
• Fix token sampling in BERT datasets (#1979)
• Fix semantic segmentation normalization in PyTorch (#1980)
• Fix mean square loss calculation in PyTorch and TensorFlow (#1984)
• Fix broken paragraphs (8e0fe4b)

Release v0.17.0

Dive into Deep Learning is now available on arxiv!

Framework Adaptation

We have added TensorFlow implementations up to Chapter 11 (Optimization Algorithms).

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Optimization (the first 4 sections)
• Computational Performance
• Computer Vision
• Natural Language Processing: Pretraining
• Natural Language Processing: Applications

Finalized chapters are being translated into Chinese (d2l-zh v2)

Other Improvements

• Add BLEU uniform weights from the original paper
• Revise the normalization trick in LogSumExp
• Revise data standardization
• Prove convexity using second derivatives for one-dimensional and multi-dimensional cases
• Improve d2l.train_2d function
• Improve convergence analysis of Newton's method
• Improve SGD convergence analysis for convex objectives 1
• Improve Convergence Analysis for Convex Objectives
• Reorganize comparisons of network partitioning, layer-wise partitioning, and data parallelism
• Improve d2l.box_iou function
• Improve the "Labeling Classes and Offsets" subsection
• Add discussions of issues of non-maximum suppression
• Reorganize multiscale anchor boxes and multiscale detection
• Highlight layerwise representations via deep nets in multiscale object detection
• Connect SSD downsampling blocks to VGG blocks
• Refer to YOLO and a recent survey on object detection
• Fix legend issues in Kaggle CIFAR-10 and ImageNet Dogs
• Improve performance on the Kaggle small-scale CIFAR-10 dataset
• Improve performance on the Kaggle small-scale ImageNet Dog dataset
• Improve the function to build the mapping from RGB to class indices for VOC labels
• Revise motivations for transposed convolution
• Rewrite basic transposed convolution operation
• Add relations between transposed convolution and regular convolution implementations
• Improve explanations of the pretrained backbone for the fully convolutional network
• Improve the output synthesized image of style transfer
• Add d2l.show_list_len_pair_hist
• Fix d2l.get_negatives
• Improve efficiency of d2l.Vocab
• Exclude unknown tokens when training word embeddings
• Add self-supervised learning
• Add discussions of self-supervised learning in NLP
• Revise the notation table

Release v0.16.0

Brand-New Attention Chapter

We have added the brand-new Chapter: Attention Mechanisms:

• Attention Cues

  • Attention Cues in Biology
  • Queries, Keys, and Values
  • Visualization of Attention

• Attention Pooling: Nadaraya-Watson Kernel Regression

  • Generating the Dataset
  • Average Pooling
  • Nonparametric Attention Pooling
  • Parametric Attention Pooling

• Attention Scoring Functions

  • Masked Softmax Operation
  • Additive Attention
  • Scaled Dot-Product Attention

• Bahdanau Attention

  • Model
  • Defining the Decoder with Attention
  • Training

• Multi-Head Attention

  • Model
  • Implementation

• Self-Attention and Positional Encoding

  • Self-Attention
  • Comparing CNNs, RNNs, and Self-Attention
  • Positional Encoding

• Transformer

  • Model
  • Positionwise Feed-Forward Networks
  • Residual Connection and Layer Normalization
  • Encoder
  • Decoder
  • Training

PyTorch Adaptation Completed

We have completed PyTorch implementations for Vol.1 (Chapter 1--15).

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Introduction
• Modern Recurrent Neural Networks

Chinese Translation

The following chapters have been translated into Chinese (d2l-zh v2 Git repo, Web preview):

• Introduction
• Preliminaries
• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks

Turkish Translation

The community are translating the book into Turkish (d2l-tr Git repo, Web preview). The first draft of Chapter 1--7 is complete.

Release v0.15.0

Framework Adaptation

We have added PyTorch implementations up to Chapter 11 (Optimization Algorithms). Chapter 1--7 and Chapter 11 have also been adapted to TensorFlow.

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks
• Recurrent Neural Networks

Finalized chapters are being translated into Chinese (d2l-zh v2)

Other Improvements

• Fixed issues of not showing all the equation numbers in the HTML and PDF
• Consistently used f-string
• Revised overfitting experiments
• Fixed implementation errors for weight decay experiments
• Improved layer index style
• Revised "breaking the symmetry"
• Revised descriptions of covariate and label shift
• Fixed mathematical errors in covariate shift correction
• Added true risk, empirical risk, and (weighted) empirical risk minimization
• Improved variable naming style for matrices and tensors
• Improved consistency of mathematical notation for tensors of order two or higher
• Improved mathematical descriptions of convolution
• Revised descriptions of cross-correlation
• Added feature maps and receptive fields
• Revised mathematical descriptions of batch normalization
• Added more details to Markov models
• Fixed implementations of k-step-ahead predictions in sequence modeling
• Fixed mathematical descriptions in language modeling
• Improved the d2l.Vocab API
• Fixed mathematical descriptions and figure illustrations for deep RNNs
• Added BLEU
• Improved machine translation application results
• Improved the animation plot function in the all the training loops