Lectures

Here, you can find the recordings of the lecture videos.

  • Lecture 0: Course Overview and Logistics
    Overview: This lecture goes through the course logistics. Unfortunately, it did not get recorded!
    [link]

    Lecture Notes:

  • Lecture 1: Fundamentals of DL
    DL Components: This lecture goes over preliminaries. We study key components of ML problems, work with the example of classification, and formulate the training of a model via empirical risk minimization. Unfortunately, the lecture was not recorded!
    [link]

    Lecture Notes:

    Further Reads:

    • Motivation: Chapter 1 - Section 1.1 of [BB]
    • Review on Linear Algebra: Chapter 2 of [GYC]
    • ML Components: Chapter 1 - Sections 1.2.1 to 1.2.4 of [BB]
    • Binary Classification: Chapter 5 - Sections 5.1 and 5.2 of [BB]
    • McCulloch-Pitts Model: Paper A logical calculus of the ideas immanent in nervous activity published in the Bulletin of Mathematical Biophysics by Warren McCulloch and Walter Pitts in 1943, proposing a computational model for neuron. This paper is treated as the pioneer study leading to the idea of artificial neuron
    • Overview on Risk Minimization: Paper An overview of statistical learning theory published as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks by Vladimir N. Vapnik in 1999
  • Lecture 2: Deep NNs and Gradient Descent
    NNs: We talk about Universal Approximation by NNs, this motivates us to use Deep NNs for ML. We then formally define Deep NNs and get some basics on MLPs. Finally, we study the gradient descent algorithm which is used for training of Deep NNs.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 3: Multiclass Classification and Backpropagation
    Forward Pass: We review neural classification using an MLP. We see that cross-entropy is a better loss helping us make differentiable risk. We can deploy that by interpreting the output of a neural network to be probability of each label. At the second part of the course, we discuss computation graphs. We see that the flow of information in a NN is similar to a computation graph. We could use this fact to build an algorithmic approach based on chain-rule to compute the gradient of the loss with respect to each weight in the network, this algorithm is called Backpropagation. We develop this algorithm for an MLP and see that it's dual to the forward pass.
    [link]

    Lecture Notes:

    Further Reads:

    • Deep FNNs: Chapter 6 - Sections 6.3 and 6.4 of [GYC]
    • Backpropagation: Chapter 8 of [BB]
    • Backpropagation of Error Paper Learning representations by back-propagating errors published in Nature by D. Rumelhart, G. Hinton and R. Williams in 1986 advocating the idea of systematic gradient computation of a computation graph
  • Lecture 4: Optimizers and Generalization
    Optimizer & Overfitting: We study the SGD algorithm. We see that using mini-batches we can control the tradeoff between variance and complexity of gradient computation. We then take a look at extensions of SGD, namely momentum method and Rprop, RMSprop, and Adam. In the second part, we study the overfitting, its main sources, and practical means to handle it, i.e., cross-validation, data augmentation, and regularization.
    [link]

    Lecture Notes:

    Further Reads:

    • SGD: Chapter 5 - Section 5.9 of [GYC]
    • Regularization: Chapter 7 of [GYC]
    • Learning Rate Scheduling Paper Cyclical Learning Rates for Training Neural Networks published in Winter Conference on Applications of Computer Vision (WACV) by Leslie N. Smith in 2017 discussing learning rate scheduling
    • Rprop Paper A direct adaptive method for faster backpropagation learning: the RPROP algorithm published in IEEE International Conference on Neural Networks by M. Riedmiller and H. Braun in 1993 proposing Rprop algorithm
    • Dropout 1 Paper Improving neural networks by preventing co-adaptation of feature detectors published in 2012 by G. Hinton et al. proposing Dropout
    • Dropout 2 Paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting published in 2014 by N. Srivastava et al. providing some analysis and illustrations on Dropout
  • Lecture 5: Convolutional NNs
    CNN I: We introduce CNNs. We learn about the convolution layer, its nature, and its complexity. We see how we can build a neural network using these layers. We also understand how convolution extract local features from input maps. This makes CNNs a great candidate for visual data processing.
    [link]

    Lecture Notes:

    Further Reads:

    • Hubel and Wiesel Study Paper Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex published in 1962 by D. Hubel and T. Wiesel elaborating their finding on visual understanding
    • Neocognitron Paper Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position published in 1980 by K. Fukushima proposing the Neocognitron as a computational model for visual learning
    • LeNet Paper Gradient-Based Learning Applied to Document Recognition published in 1998 by Y. LeCun et al. discussing LeNet
    • Convolution: Chapter 9 of [GYC]
    • Convolution: Chapter 10 of [BB]
  • Lecture 6: Deep CNNs
    Batch-Norm: We study design of deep CNNs. We see that as a rule of thumb, we could design the layers such that we gradually expand the receptive field and prune redundant global features. We then study training of CNNs and see how the backpropagation looks like. We finally discuss the problem of very deep networks, i.e., vanishing gradient. This motivates us to learn normalization and residual learning in the next lecture.
    [link]

    Lecture Notes:

    Further Reads:

    • VGG Paper Very Deep Convolutional Networks for Large-Scale Image Recognition published in 2014 by K. Simonyan and A. Zisserman proposing VGG Architectures
    • ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
  • Lecture 7: Normalization and Residual Learning
    ResNet: We discuss major means for handling vanishing gradient. We first discuss normalization, namely input, batch, and layer normalization. We see how they impact forward and backpropagation. We then discuss residual learning and skip connection which lead to the introduction of ResNet.
    [link]

    Lecture Notes:

    Further Reads:

    • Batch-Norm Paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift published in 2015 by S. Ioffe and C. Szegedy proposing Batch Normalization
    • Batch-Norm Meaning Paper How Does Batch Normalization Help Optimization? published in 2018 by S. Santurkar et al. discussing why Batch Normalization works: they claim that the main reason is that loss landscape is getting much smoother
    • ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
  • Lecture 8: RNNs and Gated Architectures
    RNNs: We study sequence data and their processing by NNs. We see that due to temporal connection, we cannot process them separately. We hence need to develop a sequence model. We study the most basic design for sequence processing, i.e., a recurrent NN. We see that by sequential processing of data, we need to backpropagate through time. This leads to finite memory due to the vanishing gradient behavior. We then learn the gating principle as a key idea to extend the memory through time. This idea leads to gated architectures, such as LSTM and GRU.
    [link]

    Lecture Notes:

    Further Reads:

    • Jordan Network Paper Attractor dynamics and parallelism in a connectionist sequential machine published in 1986 by M. Jordan proposing his RNN
    • Elman Network Paper Finding structure in time published in 1990 by J. Elman proposing a revision to Jordan Network
    • BPTT Paper Backpropagation through time: What it does and how to do it published in 2002 by P. Werbos explaining BPTT
    • LSTM Paper Long short-term memory published in 1997 by S. Hochreiter and J. Schmidhuber proposing LSTM
  • Lecture 9: Seq2Seq Models and Transformer
    Transformer: We build a a basic language model. We see how we can train it. We use our basic LM to build an caption generator machine. This is a simple example of an encoding-decoding architecture. We see that despite its theoretical soundness, NMT with basic Enc-Dec suffers from lack of attention. We hence build a computational mechanism that provides the decoder possibility to attend to related tokens at the encoder. This is what we know as Attention Mechanism. Interestingly, we can ignore the time connection and perform the complete process in parallel with high efficiency through self-attention. We can use the self-attention module to build a deep sequence processor. This is what we call Transformer.
    [link]

    Lecture Notes:

    Further Reads:

    • Basic LM Paper A Neural Probabilistic Language Model published in 2003 by Y. Bengio developing frist really-functioning LM
    • Attention Paper Neural Machine Translation by Jointly Learning to Align and Translate published in 2015 by D. Bahdanau et al. proposing Attention mechanism
    • Attention NMT Paper Effective Approaches to Attention-based Neural Machine Translation published in 2015 by M. Loung et al. proposing Attention-based NMT
    • Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
  • Lecture 10: Multimodal Transformers and LMs
    ViT and LLMs: We complete the transformer architecture. We then revisit previous Seq2Seq problems, LMs and image captioning. We see that we can address those tasks more efficiently using transformers. We further see that the idea can be extended into any modality, Vision, Audio, etc. This leads to the introduction of Vision Transformers ViT and more generally, multi-modal Transformers. Unfortunately, only the first part of the video has been recorded.
    [link]

    Lecture Notes:

    Further Reads:

    • Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
    • GPT-1 Paper Improving language understanding by generative pre-training published in 2018 by A. Radford et al. proposing GPT-1
    • ViT Paper An image is worth 16x16 words: Transformers for image recognition at scale published in ICLR 2021 by A. Dosovitskiy et al. proposing ViT