Materials

Textbook Learning Machine Learning

The course instructors Prof. Salma Emara and Prof. Ali Bereyhi have recently developed the textbook Learning Machine Learning. The first edition can be accessed this link. In case of suggestions or feedback, please feel free to reach out.

Lecture Notes

The lecture notes are uploaded through the semester.

Week 1

• Lecture 0: Logistics
• Lecture 1: Preliminaries and Clustering
• Lecture 2: Revisiting Clustering with Notion of Risk

Week 2

• Lecture 3: Probabilistic Modeling and Distribution Learning

Week 3

• Lecture 4: Principal Component Analysis
• Lecture 5: PCA Algorithm, Meaning, and Applications

Week 4

• Lecture 6: Linear Regression
• Lecture 7: Classification and Logistic Regression

Week 5

• Lecture 8: Classification with Confidence
• Lecture 9: Support Vector Classifier

Week 6

• Lecture 10: Support Vector Machine and Kernel Method
• Lecture 11: Generalization and Gradient Descent

Week 7

• Reading Week - No Lecture

Week 8

• Lecture 12: Representation by Network of Neurons

Week 9

• Lecture 13: NNs as Universal Approximators
• Lecture 14: Learning via NNs - Part I

Week 10

• Lecture 15: Learning via NNs - Part II
• Lecture 16: Generalization

Week 11

• Lecture 17: Regularization and Convolution
• Lecture 18: CNNs and Sequence Models

Week 12

• Lecture 19: Sequence Models
• Lecture 20: Generative Models

Week 13

• Lecture 21: Reinforcement Learning

Tutorial Notebooks

The tutorial notebooks can be accessed below.

• Tutorial 1: Basic Libraries for ML in Python and K-means Clustering, by James Zhu - Watch the Video
• Tutorial 2 Density Learning and Maximum Likeihood, by James Zhu Watch the Video
• Tutorial 3 - Slides Tutorial 3 - Notebook PCA and Linear Algebra Review, by James Zhu Watch the Video
• Tutorial 4 Linear and Logistic Regression, by James Zhu Watch the Video
• Tutorial 5 Notes SVC + Midterm Review, by James Zhu Please check the sample exams on Quercus
• Tutorial 6 Introduction to PyTorch, by Mohammadreza Safavi Watch the Video
• Tutorial 7 Function Approximation by NNs, by Mohammadreza Safavi Watch the Video
• Tutorial 8 MLPs, by Mohammadreza Safavi Watch the Video
• Tutorial 9 Generalization, by Mohammadreza Safavi Watch the Video
• Tutorial 10 CNNs, by Mohammadreza Safavi Watch the Video

Book

We use various textbooks in the course. The main text for the course is

• [EB] Emara, Salma and Bereyhi, Ali. Learning Machine Learning. First Edition, 2026.

The following textbooks have further covered the key notions in the course, and are cited in the lecture notes:

• [B] Bishop, Christopher M. Pattern Recognition and Machine Learning. Springer Nature, 2006.
• [HTF] Hastie, Trevor and Tibshirani, Robert and Friedman, Jerome. The Elements of Statistical Learning. Springer, Second Edition, 2009.
• [MK] MacKay, David. Information Theory, Inference, and Learning Algorithms. Cambridge, Version 7.2, 2005.

The following textbook is a good resource for practicing programming.

• Müller, Andreas, and Sarah Guido. Introduction to Machine Learning with Python. O’Reilly Media, Inc., 2016.

Reading List

This section will be completed gradually through the semester.

Introduction

• Introduction to ML, Chapter 1 - Sections 1.1 and 1.3 of [B]
• Different Domains of ML, Chapter 1, Chapter 2 - Sections 2.1 and 2.2 of [HTF]

Clustering

• Clustering, Chapter 20 of [MK]
• K-means Clustering, Chapter 9 - Section 9.1 of[B]
• Clustering, Chapter 14 - Section 14.3 of [HTF]

Probabilistic Modeling and Maximum Likelihood

Maximum Likelihood

• Maximum Likelihood, Chapter 8 of [Probabilistic ML Book]
• More on Probabilistic Modeling, Chapter 23 of [MK]

Connections to Clustering

• Maximum Likelihood and Clustering, Chapter 22 of [MK]

Gaussian Maximum Likelihood

• Gaussian Max-Likelihood, Chapter 2 - Sections 2.1 and 2.3 of [B]

Principal Component Analysis

PCA Algorithm

• PCA, Chapter 12 of [B]
• PCA, Chapter 14 of [HTF]

Latent Space Design

• Latent Space, Chapter 34 of [MK]

Maximum Likelihood Interpretation

• Probabilistic PCA, Paper Probabilistic Principal Component Analysis written by Michael E. Tipping and Christopher M. Bishop in 1999 giving the statistical viewpoint on PCA

Applications of PCA

• Eigenface, Paper Eigenfaces for Recognition written by Matthew Turk and Alex Pentland in 1991 on Using PCA for Face Recognition
• Recommendation System, Paper Matrix Factorization Techniques for Recommender Systems written by Yehuda Koren et al. in 2009 On Netflix Prize Winner Project

Supervised Learning

Risk Minimization vs Maximum Likelihood

• Overview on Risk Minimization: Paper An overview of statistical learning theory published by Vladimir N. Vapnik as an overview of his life-going developments in ML in the IEEE Transactions on Neural Networks in 1999

Linear Regression

• Linear Regression, Chapter 3 of [B]
• Linear Regression, Chapter 3 of [HTF]
• Maximum Likelihood View on Regression, Chapters 11 and 12 of [Probabilistic ML Book]

Logistic Regression

• Logistic Regression, Chapter 4 of [B]
• Linear Classification, Chapter 4 of [HTF]
• Maximum Likelihood View on Classification, Chapter 10 of [Probabilistic ML Book]

Support Vector Machine

• SVC, Chapter 12, Sections 12.1 and 12.2 of [HTF]
• Constrained Optimization, Chapter 5 of Convex Optimization
• SVNets Paper Support Vector Networks published by Cortes and Vapnik in 1995
• Tutorials on SVM Survey A Tutorial on Support Vector Machines for Pattern Recognition published by Burges in 1998

Generalization and Gradient Descent

• Model Assessment and Selection , Chapter 7 of [HTF]
• Gradient Descent, Chapter 10: Section 10.10 of [HTF]

Neural Networks

Representation via NNs

• NNs, Chapter 5, Section 5.1 of [B]
• Intro to NNs, Chapter 11, Sections 11.1 and 11.3 of [HTF]
• NNs, Chapters 13, Sections 13.1 and 13.2 of [Probabilistic ML Book]

Universal Approximation Theorem

• Expressiveness of NNs: Paper Multilayer feedforward networks are universal approximators published by Hornik et al in 1989
• Universal Approximation: Paper Approximation by superpositions of a sigmoidal function published in Mathematics of Control, Signals and Systems by George V. Cybenko in 1989

Training NNs

• Training NNs, Chapter 5, Section 5.2 of [B]
• Fitting NNs, Chapter 11, Section 11.4 of [HTF]
• Training NNs, Chapters 13, Section 13.4 of [Probabilistic ML Book]
• Backpropagation, Chapter 5, Section 5.3 of [B]
• Training NNs, Chapters 13, Sections 13.1 to 13.4 of [Probabilistic ML Book]

Regularization

• Regularization, Chapter 13, Section 13.5 of [Probabilistic ML Book]
• Regularization, Chapter 6 of [Deep Learning]

Convolutional NNs

• CNNs, Chapter 14 of [Probabilistic ML Book]
• CNNs, Chapter 9 of [Deep Learning]

Advances in ML

Sequence Processing

• RNNs, Chapter 10 of [Deep Learning]
• Basic LM Paper A Neural Probabilistic Language Model published in 2003 by Y. Bengio developing frist really-functioning LM

Generative Modeling

• Generative Models, Chapter 20 of [Deep Learning]

Reinforcement Learning

• Reinforcement Learning: An Introduction by Sutton and Barto
• Spinning Up in Deep RL by OpenAI