Schedule

Lecture Notes:

• Chapter 0

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• Motivation: Chapter 1 - Section 1.1 of [BB]

Lecture Notes:

• Chapter 1 - Section 2

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Review on Linear Algebra: Chapter 2 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• ML Components: Chapter 1 - Sections 1.2.1 to 1.2.4 of [BB]
• ML Basics: Chapter 5 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Binary Classification: Chapter 5 - Sections 5.1 and 5.2 of [BB]
• McCulloch-Pitts Model: Paper A logical calculus of the ideas immanent in nervous activity published in the Bulletin of Mathematical Biophysics by Warren McCulloch and Walter Pitts in 1943, proposing a computational model for neuron. This paper is treated as the pioneer study leading to the idea of artificial neuron

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Probability Theory: Chapter 2 of [BB]
• Probability Review: Chapter 3 of [GYC]

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• Overview on Risk Minimization: Paper An overview of statistical learning theory published as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks by Vladimir N. Vapnik in 1999

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• Perceptron Simulation Experiments: Paper Perceptron Simulation Experiments presented by Frank Rosenblatt in Proceedings of IRE in 1960
• Perceptron: Chapter 1 - Section 1.2.1 of [Ag]

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• Universal Approximation: Paper Approximation by superpositions of a sigmoidal function published in Mathematics of Control, Signals and Systems by George V. Cybenko in 1989

Lecture Notes:

• Chapter 1 - Section 4

Further Reads:

• DNNs: Chapter 6 - Sections 6.2 and 6.3 of [BB]

Lecture Notes:

• Chapter 1 - Section 5

Further Reads:

• Gradient-based Optimization: Chapter 4 - Sections 4.3 and 4.4 of [GYC]
• Gradient Descent: Chapter 7 - Sections 7.1 and 7.2 of [BB]

Lecture Notes:

• Chapter 1 - Section 5

Further Reads:

• Gradient Descent: Chapter 7 - Section 7.2 of [BB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• Deep FNNs: Chapter 6 - Sections 6.3 and 6.4 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 6 - Section 6.5 of [GYC]
• Backpropagation: Chapter 8 of [BB]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Backpropagation: Chapter 8 of [BB]
• Backpropagation of Error Paper Learning representations by back-propagating errors published in Nature by D. Rumelhart, G. Hinton and R. Williams in 1986 advocating the idea of systematic gradient computation of a computation graph

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Binary Classification: Chapter 6 - Section 6.6 of [BB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Multi-class Models: Chapter 2 - Section 2.3 of [Ag]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• SGD: Chapter 5 - Section 5.9 of [GYC]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• SGD: Chapter 5 - Section 5.9 of [GYC]

Lecture Notes:

• Chapter 2 - Section 4

Further Reads:

• Generalization: Chapter 6 of the Book Patterns, predictions, and actions: A story about machine learning by Moritz Hardt and B. Recht published in 2021

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Notes on Optimizers Lecture notes of the course Optimization for Machine Learning by Ashok Cutkosky in Boston University: A good resource for optimizers

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Learning Rate Scheduling Paper Cyclical Learning Rates for Training Neural Networks published in Winter Conference on Applications of Computer Vision (WACV) by Leslie N. Smith in 2017 discussing learning rate scheduling
• Rprop Paper A direct adaptive method for faster backpropagation learning: the RPROP algorithm published in IEEE International Conference on Neural Networks by M. Riedmiller and H. Braun in 1993 proposing Rprop algorithm

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• RMSprop Lecture note by GEoffrey Hinton proposing RMSprop
• RMSprop Analysis Paper RMSProp and equilibrated adaptive learning rates for non-convex optimization by Y. Dauphin et al. published in 2015 talking about RMSprop and citing Honton’s lecture notes
• Adam Paper Adam: A Method for Stochastic Optimization published in 2014 by D. Kingma and J. Ba proposing Adam

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• Regularization: Chapter 7 of [GYC]