Schedule

Lecture Notes:

• Chapter 0

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• Motivation: Chapter 1 - Section 1.1 of [BB]

Lecture Notes:

• Chapter 1 - Section 2

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Review on Linear Algebra: Chapter 2 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• ML Components: Chapter 1 - Sections 1.2.1 to 1.2.4 of [BB]
• ML Basics: Chapter 5 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Binary Classification: Chapter 5 - Sections 5.1 and 5.2 of [BB]
• McCulloch-Pitts Model: Paper A logical calculus of the ideas immanent in nervous activity published in the Bulletin of Mathematical Biophysics by Warren McCulloch and Walter Pitts in 1943, proposing a computational model for neuron. This paper is treated as the pioneer study leading to the idea of artificial neuron

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Probability Theory: Chapter 2 of [BB]
• Probability Review: Chapter 3 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Overview on Risk Minimization: Paper An overview of statistical learning theory published as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks by Vladimir N. Vapnik in 1999

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• Perceptron Simulation Experiments: Paper Perceptron Simulation Experiments presented by Frank Rosenblatt in Proceedings of IRE in 1960
• Perceptron: Chapter 1 - Section 1.2.1 of [Ag]

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• Universal Approximation: Paper Approximation by superpositions of a sigmoidal function published in Mathematics of Control, Signals and Systems by George V. Cybenko in 1989

Lecture Notes:

• Chapter 1 - Section 4

Further Reads:

• DNNs: Chapter 6 - Sections 6.2 and 6.3 of [BB]

Lecture Notes:

• Chapter 1 - Section 5

Further Reads:

• Gradient-based Optimization: Chapter 4 - Sections 4.3 and 4.4 of [GYC]
• Gradient Descent: Chapter 7 - Sections 7.1 and 7.2 of [BB]

Lecture Notes:

• Chapter 1 - Section 5

Further Reads:

• Gradient Descent: Chapter 7 - Section 7.2 of [BB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• Deep FNNs: Chapter 6 - Sections 6.3 and 6.4 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]
• Backpropagation: Chapter 8 of [BB]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 8 of [BB]
• Backpropagation of Error Paper Learning representations by back-propagating errors published in Nature by D. Rumelhart, G. Hinton and R. Williams in 1986 advocating the idea of systematic gradient computation of a computation graph

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Binary Classification: Chapter 6 - Section 6.6 of [BB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Multi-class Models: Chapter 2 - Section 2.3 of [Ag]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• SGD: Chapter 5 - Section 5.9 of [GYC]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• SGD: Chapter 5 - Section 5.9 of [GYC]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• Generalization: Chapter 6 of the Book Patterns, predictions, and actions: A story about machine learning by Moritz Hardt and B. Recht published in 2021

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Notes on Optimizers Lecture notes of the course Optimization for Machine Learning by Ashok Cutkosky in Boston University: A good resource for optimizers

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Learning Rate Scheduling Paper Cyclical Learning Rates for Training Neural Networks published in Winter Conference on Applications of Computer Vision (WACV) by Leslie N. Smith in 2017 discussing learning rate scheduling
• Rprop Paper A direct adaptive method for faster backpropagation learning: the RPROP algorithm published in IEEE International Conference on Neural Networks by M. Riedmiller and H. Braun in 1993 proposing Rprop algorithm

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• RMSprop Lecture note by GEoffrey Hinton proposing RMSprop
• RMSprop Analysis Paper RMSProp and equilibrated adaptive learning rates for non-convex optimization by Y. Dauphin et al. published in 2015 talking about RMSprop and citing Honton’s lecture notes
• Adam Paper Adam: A Method for Stochastic Optimization published in 2014 by D. Kingma and J. Ba proposing Adam

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• Regularization: Chapter 7 of [GYC]

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• Overfitting and Regularization: Chapter 9 - Sections 9.1 to 9.3 of [BB]

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• Overfitting and Regularization: Chapter 9 - Sections 9.1 to 9.3 of [BB]
• Tikhonov Paper Tikhonov Regularization and Total Least Squares published in 1999 by G. Golub et al. illustrating the Tikhonov Regularization work
• Lasso Paper Regression Shrinkage and Selection Via the Lasso published in 1996 by R. Tibshirani proposing the legendary Lasso

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• Dropout 1 Paper Improving neural networks by preventing co-adaptation of feature detectors published in 2012 by G. Hinton et al. proposing Dropout
• Dropout 2 Paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting published in 2014 by N. Srivastava et al. providing some analysis and illustrations on Dropout

Lecture Notes:

• Chapter 3 - Section 3

Further Reads:

• Data: Chapter 8 of the Book Patterns, predictions, and actions: A story about machine learning by Moritz Hardt and B. Recht published in 2021
• Data Processing in Python Open Book Minimalist Data Wrangling with Python by Marek Gagolewski going through data processing in Python

Lecture Notes:

• Chapter 3 - Section 4

Further Reads:

• Normalization Paper Is normalization indispensable for training deep neural network? published in 2020 by J. Shao et al. discussing the meaning and effects of normalization

Lecture Notes:

• Chapter 3 - Section 4

Further Reads:

• Batch-Norm Paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift published in 2015 by S. Ioffe and C. Szegedy proposing Batch Normalization
• Batch-Norm Meaning Paper How Does Batch Normalization Help Optimization? published in 2018 by S. Santurkar et al. discussing why Batch Normalization works: they claim that the main reason is that loss landscape is getting much smoother

Lecture Notes:

• Chapter 4 - Section 1

Further Reads:

• Hubel and Wiesel Study Paper Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex published in 1962 by D. Hubel and T. Wiesel elaborating their finding on visual understanding
• Neocognitron Paper Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position published in 1980 by _K. Fukushima _ proposing the Neocognitron as a computational model for visual learning
• Backpropagating on LeNet Paper Backpropagation Applied to Handwritten Zip Code Recognition published in 1989 by Y. LeCun et al. developing backpropagation for LeNet
• LeNet Paper Gradient-Based Learning Applied to Document Recognition published in 1998 by Y. LeCun et al. discussing LeNet

Lecture Notes:

• Chapter 4 - Section 1

Further Reads:

• Convolution: Chapter 9 - Sections 9.1 and 9.2 of [GYC]

Lecture Notes:

• Chapter 4 - Section 2

Further Reads:

• Convolution: Chapter 10 - Sections 10.2.1 and 10.2.2 of [BB]

Lecture Notes:

• Chapter 4 - Section 2

Further Reads:

• Multi-channel Convolution: Chapter 10 - Sections 10.2.3 to 10.2.5 of [BB]

Lecture Notes:

• Chapter 4 - Section 2

Further Reads:

• Pooling: Chapter 10 - Section 10.2.6 of [BB]
• Flattening: Chapter 10 - Sections 10.2.7 and 10.2.8 of [BB]

Lecture Notes:

• Chapter 4 - Section 3

Further Reads:

• Convolution: Chapter 9 - Sections 9.4 and 9.6 of [GYC]

Lecture Notes:

• Chapter 4 - Section 3

Further Reads:

• VGG Paper Very Deep Convolutional Networks for Large-Scale Image Recognition published in 2014 by K. Simonyan and A. Zisserman proposing VGG Architectures

Lecture Notes:

• Chapter 4 - Section 4

Further Reads:

• LeCun’s Paper Paper Gradient-based learning applied to document recognition published in 2002 by Y. LeCun et al. summarizing the learning process in CNN
• Efficient Backpropagation on CNN Paper High Performance Convolutional Neural Networks for Document Processing published in 2006 by K. Chellapilla et al. discussing efficient backpropagation on CNNs.

Lecture Notes:

• Chapter 5 - Section 1

Further Reads:

• ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet

Lecture Notes:

• Chapter 5 - Section 2

Further Reads:

• ResNet Paper Deep Residual Learning for Image Recognition published in 2015 by K. He et al. proposing ResNet
• ResNet-1001 Paper Identity Mappings in Deep Residual Networks published in 2016 by K. He et al. demonstrating how deep ResNet can go
• U-Net Paper U-Net: Convolutional Networks for Biomedical Image Segmentation published in 2015 by O. Ronneberger et al. proposing U-Net
• DenseNet Paper Densely Connected Convolutional Networks published in 2017 by H. Huang et al. proposing DenseNet

Lecture Notes:

• Chapter 6 - Section 1

Further Reads:

• Jordan Network Paper Attractor dynamics and parallelism in a connectionist sequential machine published in 1986 by M. Jordan proposing his RNN
• Elman Network Paper Finding structure in time published in 1990 by J. Elman proposing a revision to Jordan Network

Lecture Notes:

• Chapter 6 - Section 1

Further Reads:

• BPTT Paper Backpropagation through time: What it does and how to do it published in 2002 by P. Werbos explaining BPTT

Lecture Notes:

• Chapter 6 - Section 2

Further Reads:

• Seq Models Article The Unreasonable Effectiveness of Recurrent Neural Networks written in May 2015 by A. Karpathy discussing different types of sequence problems

Lecture Notes:

• Chapter 6 - Section 3

Further Reads:

• Vanishing Gradient with BPTT Paper On the difficulty of training recurrent neural networks published in 2013 by R. Pascanu et al. discussing challenges in training with BPTT
• Truncated BPTT Paper An efficient gradient-based algorithm for on-line training of recurrent network trajectories published in 1990 by R. Williams and J. Peng explaining truncated BPTT

Lecture Notes:

• Chapter 6 - Section 4

Further Reads:

• Gating Principle Chapter Long Short-Term Memory published in 2012 in book Supervised Sequence Labelling with Recurrent Neural Networks by A. Graves explaining Gating idea