Schedule
-
EventDateDescriptionDescription
-
Session09/08/2026 13:00
TuesdayFirst Lecture -
Lecture09/08/2026
TuesdayLecture 0: Course Overview and LogisticsLecture Notes:
-
Lecture09/08/2026
TuesdayLecture 1: Fundamentals of DLLecture Notes:
Further Reads:
- Motivation: Chapter 1 - Section 1.1 of [BB]
- Review on Linear Algebra: Chapter 2 of [GYC]
- ML Components: Chapter 1 - Sections 1.2.1 to 1.2.4 of [BB]
- Binary Classification: Chapter 5 - Sections 5.1 and 5.2 of [BB]
- McCulloch-Pitts Model: Paper A logical calculus of the ideas immanent in nervous activity published in the Bulletin of Mathematical Biophysics by Warren McCulloch and Walter Pitts in 1943, proposing a computational model for neuron. This paper is treated as the pioneer study leading to the idea of artificial neuron
- Overview on Risk Minimization: Paper An overview of statistical learning theory published as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks by Vladimir N. Vapnik in 1999
-
Assignment09/11/2026
FridayAssignment #1 - Fundamentals of Computational Learning released! -
Lecture09/15/2026
TuesdayLecture 2: Deep NNs and Gradient DescentLecture Notes:
Further Reads:
- DNNs: Chapter 6 - Sections 6.2 and 6.3 of [BB]
- Gradient-based Optimization: Chapter 4 - Sections 4.3 and 4.4 of [GYC]
- Gradient Descent: Chapter 7 - Sections 7.1 and 7.2 of [BB]
- Perceptron Simulation Experiments: Paper Perceptron Simulation Experiments presented by Frank Rosenblatt in Proceedings of IRE in 1960
- Universal Approximation: Paper Approximation by superpositions of a sigmoidal function published in Mathematics of Control, Signals and Systems by George V. Cybenko in 1989
-
Lecture09/22/2026
TuesdayLecture 3: Multiclass Classification and BackpropagationLecture Notes:
Further Reads:
- Deep FNNs: Chapter 6 - Sections 6.3 and 6.4 of [GYC]
- Backpropagation: Chapter 8 of [BB]
- Backpropagation of Error Paper Learning representations by back-propagating errors published in Nature by D. Rumelhart, G. Hinton and R. Williams in 1986 advocating the idea of systematic gradient computation of a computation graph
-
Due09/25/2026 23:30
FridayAssignment #1 due -
Assignment09/25/2026
FridayAssignment 2: MLPs released! -
Lecture09/29/2026
TuesdayLecture 4: Optimizers and GeneralizationLecture Notes:
Further Reads:
- SGD: Chapter 5 - Section 5.9 of [GYC]
- Regularization: Chapter 7 of [GYC]
- Learning Rate Scheduling Paper Cyclical Learning Rates for Training Neural Networks published in Winter Conference on Applications of Computer Vision (WACV) by Leslie N. Smith in 2017 discussing learning rate scheduling
- Rprop Paper A direct adaptive method for faster backpropagation learning: the RPROP algorithm published in IEEE International Conference on Neural Networks by M. Riedmiller and H. Braun in 1993 proposing Rprop algorithm
- Dropout 1 Paper Improving neural networks by preventing co-adaptation of feature detectors published in 2012 by G. Hinton et al. proposing Dropout
- Dropout 2 Paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting published in 2014 by N. Srivastava et al. providing some analysis and illustrations on Dropout
-
Lecture10/06/2026
TuesdayLecture 5: Convolutional NNsLecture Notes:
Further Reads:
- Hubel and Wiesel Study Paper Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex published in 1962 by D. Hubel and T. Wiesel elaborating their finding on visual understanding
- Neocognitron Paper Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position published in 1980 by K. Fukushima proposing the Neocognitron as a computational model for visual learning
- LeNet Paper Gradient-Based Learning Applied to Document Recognition published in 1998 by Y. LeCun et al. discussing LeNet
- Convolution: Chapter 9 of [GYC]
- Convolution: Chapter 10 of [BB]
-
Due10/09/2026 23:30
FridayAssignment #2 due -
Assignment10/12/2026
MondayAssignment #3 - CNNs and ResNets released! -
Lecture10/13/2026
TuesdayLecture 6: Deep CNNsLecture Notes:
Further Reads:
-
Exam10/16/2026 13:00
FridayExam INotes:
- The exam is 1 hour, during the tutorials, i.e., 1-2pm
- No programming questions
- It takes place in OI-G162 (OISE)
- You can bring one A4 page (one-sided) cheat sheet
-
Lecture10/20/2026
TuesdayLecture 7: Normalization and Residual LearningLecture Notes:
Further Reads:
- Batch-Norm Paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift published in 2015 by S. Ioffe and C. Szegedy proposing Batch Normalization
- Batch-Norm Meaning Paper How Does Batch Normalization Help Optimization? published in 2018 by S. Santurkar et al. discussing why Batch Normalization works: they claim that the main reason is that loss landscape is getting much smoother
- ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
-
Due10/23/2026 23:30
FridayProject Proposal -
Lecture11/03/2026
TuesdayLecture 8: RNNs and Gated ArchitecturesLecture Notes:
Further Reads:
- Jordan Network Paper Attractor dynamics and parallelism in a connectionist sequential machine published in 1986 by M. Jordan proposing his RNN
- Elman Network Paper Finding structure in time published in 1990 by J. Elman proposing a revision to Jordan Network
- BPTT Paper Backpropagation through time: What it does and how to do it published in 2002 by P. Werbos explaining BPTT
- LSTM Paper Long short-term memory published in 1997 by S. Hochreiter and J. Schmidhuber proposing LSTM
-
Due11/06/2026 23:30
FridayAssignment #3 due -
Assignment11/06/2026
FridayAssignment #4 - Sequence Models and Transformers released! -
Lecture11/10/2026
TuesdayLecture 9: Seq2Seq Models and TransformerLecture Notes:
Further Reads:
- Basic LM Paper A Neural Probabilistic Language Model published in 2003 by Y. Bengio developing frist really-functioning LM
- Attention Paper Neural Machine Translation by Jointly Learning to Align and Translate published in 2015 by D. Bahdanau et al. proposing Attention mechanism
- Attention NMT Paper Effective Approaches to Attention-based Neural Machine Translation published in 2015 by M. Loung et al. proposing Attention-based NMT
- Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
-
Lecture11/17/2026
TuesdayLecture 10: Multimodal Transformers and LMsLecture Notes:
Further Reads:
- Transformer Paper Attention is All You Need published in 2017 by A. Vaswani et al. proposing Transformer (cited 282K times to this date)
- GPT-1 Paper Improving language understanding by generative pre-training published in 2018 by A. Radford et al. proposing GPT-1
- ViT Paper An image is worth 16x16 words: Transformers for image recognition at scale published in ICLR 2021 by A. Dosovitskiy et al. proposing ViT
-
Due11/20/2026 23:30
FridayAssignment #4 due -
Assignment11/20/2026
FridayAssignment #5 - Autoencoding released! -
Exam11/27/2026 13:00
FridayExam IINotes:
- The exam is 1 hour, during the tutorials
- No programming questions
- It covers Lectures 6-10
- It takes place in Examination Centre, Room EX-310
-
Due12/04/2026 23:30
FridayAssignment #5 due -
Due12/18/2026 23:30
FridayProject Final Submission
