Schedule

  • Event
    Date
    Description
    Description
  • Session
    09/08/2026 13:00
    Tuesday
    First Lecture
  • Lecture
    09/08/2026
    Tuesday
    Lecture 0: Course Overview and Logistics

    Lecture Notes:

  • Lecture
    09/08/2026
    Tuesday
    Lecture 1: Fundamentals of DL

    Lecture Notes:

    Further Reads:

    • Motivation: Chapter 1 - Section 1.1 of [BB]
    • Review on Linear Algebra: Chapter 2 of [GYC]
    • ML Components: Chapter 1 - Sections 1.2.1 to 1.2.4 of [BB]
    • Binary Classification: Chapter 5 - Sections 5.1 and 5.2 of [BB]
    • McCulloch-Pitts Model: Paper A logical calculus of the ideas immanent in nervous activity published in the Bulletin of Mathematical Biophysics by Warren McCulloch and Walter Pitts in 1943, proposing a computational model for neuron. This paper is treated as the pioneer study leading to the idea of artificial neuron
    • Overview on Risk Minimization: Paper An overview of statistical learning theory published as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks by Vladimir N. Vapnik in 1999
  • Assignment
    09/11/2026
    Friday
    Assignment #1 - Fundamentals of Computational Learning released!
  • Lecture
    09/15/2026
    Tuesday
    Lecture 2: Deep NNs and Gradient Descent

    Lecture Notes:

    Further Reads:

  • Lecture
    09/22/2026
    Tuesday
    Lecture 3: Multiclass Classification and Backpropagation

    Lecture Notes:

    Further Reads:

    • Deep FNNs: Chapter 6 - Sections 6.3 and 6.4 of [GYC]
    • Backpropagation: Chapter 8 of [BB]
    • Backpropagation of Error Paper Learning representations by back-propagating errors published in Nature by D. Rumelhart, G. Hinton and R. Williams in 1986 advocating the idea of systematic gradient computation of a computation graph
  • Due
    09/25/2026 23:30
    Friday
    Assignment #1 due
  • Assignment
    09/25/2026
    Friday
    Assignment 2: MLPs released!
  • Lecture
    09/29/2026
    Tuesday
    Lecture 4: Optimizers and Generalization

    Lecture Notes:

    Further Reads:

    • SGD: Chapter 5 - Section 5.9 of [GYC]
    • Regularization: Chapter 7 of [GYC]
    • Learning Rate Scheduling Paper Cyclical Learning Rates for Training Neural Networks published in Winter Conference on Applications of Computer Vision (WACV) by Leslie N. Smith in 2017 discussing learning rate scheduling
    • Rprop Paper A direct adaptive method for faster backpropagation learning: the RPROP algorithm published in IEEE International Conference on Neural Networks by M. Riedmiller and H. Braun in 1993 proposing Rprop algorithm
    • Dropout 1 Paper Improving neural networks by preventing co-adaptation of feature detectors published in 2012 by G. Hinton et al. proposing Dropout
    • Dropout 2 Paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting published in 2014 by N. Srivastava et al. providing some analysis and illustrations on Dropout
  • Lecture
    10/06/2026
    Tuesday
    Lecture 5: Convolutional NNs

    Lecture Notes:

    Further Reads:

    • Hubel and Wiesel Study Paper Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex published in 1962 by D. Hubel and T. Wiesel elaborating their finding on visual understanding
    • Neocognitron Paper Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position published in 1980 by K. Fukushima proposing the Neocognitron as a computational model for visual learning
    • LeNet Paper Gradient-Based Learning Applied to Document Recognition published in 1998 by Y. LeCun et al. discussing LeNet
    • Convolution: Chapter 9 of [GYC]
    • Convolution: Chapter 10 of [BB]
  • Due
    10/09/2026 23:30
    Friday
    Assignment #2 due
  • Assignment
    10/12/2026
    Monday
    Assignment #3 - CNNs and ResNets released!
  • Lecture
    10/13/2026
    Tuesday
    Lecture 6: Deep CNNs

    Lecture Notes:

    Further Reads:

    • VGG Paper Very Deep Convolutional Networks for Large-Scale Image Recognition published in 2014 by K. Simonyan and A. Zisserman proposing VGG Architectures
    • ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
  • Exam
    10/16/2026 13:00
    Friday
    Exam I

    Notes:

    • The exam is 1 hour, during the tutorials, i.e., 1-2pm
    • No programming questions
    • It takes place in OI-G162 (OISE)
    • You can bring one A4 page (one-sided) cheat sheet
  • Lecture
    10/20/2026
    Tuesday
    Lecture 7: Normalization and Residual Learning

    Lecture Notes:

    Further Reads:

    • Batch-Norm Paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift published in 2015 by S. Ioffe and C. Szegedy proposing Batch Normalization
    • Batch-Norm Meaning Paper How Does Batch Normalization Help Optimization? published in 2018 by S. Santurkar et al. discussing why Batch Normalization works: they claim that the main reason is that loss landscape is getting much smoother
    • ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
  • Due
    10/23/2026 23:30
    Friday
    Project Proposal
  • Lecture
    11/03/2026
    Tuesday
    Lecture 8: RNNs and Gated Architectures

    Lecture Notes:

    Further Reads:

    • Jordan Network Paper Attractor dynamics and parallelism in a connectionist sequential machine published in 1986 by M. Jordan proposing his RNN
    • Elman Network Paper Finding structure in time published in 1990 by J. Elman proposing a revision to Jordan Network
    • BPTT Paper Backpropagation through time: What it does and how to do it published in 2002 by P. Werbos explaining BPTT
    • LSTM Paper Long short-term memory published in 1997 by S. Hochreiter and J. Schmidhuber proposing LSTM
  • Due
    11/06/2026 23:30
    Friday
    Assignment #3 due
  • Assignment
    11/06/2026
    Friday
    Assignment #4 - Sequence Models and Transformers released!
  • Lecture
    11/10/2026
    Tuesday
    Lecture 9: Seq2Seq Models and Transformer

    Lecture Notes:

    Further Reads:

    • Basic LM Paper A Neural Probabilistic Language Model published in 2003 by Y. Bengio developing frist really-functioning LM
    • Attention Paper Neural Machine Translation by Jointly Learning to Align and Translate published in 2015 by D. Bahdanau et al. proposing Attention mechanism
    • Attention NMT Paper Effective Approaches to Attention-based Neural Machine Translation published in 2015 by M. Loung et al. proposing Attention-based NMT
    • Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
  • Lecture
    11/17/2026
    Tuesday
    Lecture 10: Multimodal Transformers and LMs

    Lecture Notes:

    Further Reads:

    • Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
    • GPT-1 Paper Improving language understanding by generative pre-training published in 2018 by A. Radford et al. proposing GPT-1
    • ViT Paper An image is worth 16x16 words: Transformers for image recognition at scale published in ICLR 2021 by A. Dosovitskiy et al. proposing ViT
  • Due
    11/20/2026 23:30
    Friday
    Assignment #4 due
  • Assignment
    11/20/2026
    Friday
    Assignment #5 - Autoencoding released!
  • Exam
    11/27/2026 13:00
    Friday
    Exam II

    Notes:

    • The exam is 1 hour, during the tutorials
    • No programming questions
    • It covers Lectures 6-10
    • It takes place in Examination Centre, Room EX-310
  • Due
    12/04/2026 23:30
    Friday
    Assignment #5 due
  • Due
    12/18/2026 23:30
    Friday
    Project Final Submission