Lectures

Here, you can find the recordings of the lecture videos.

  • Lecture 0: Course Overview and Logistics
    Overview: This lecture gives an overview on the course structure and the logistics.
    [link]

    Lecture Notes:

  • Lecture 1: Tokenization and Embedding
    LMs - Part 1: We start with LMs and understand how we can feed a text into it by doing the so-called "Tokenization" and "Embedding".
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 2: Language Distribution and Bi-Gram Model
    LMs - Part 2: In this lecture, we define the LMs concretely via the notion of language distribution. We then build a simple Bi-gram LM.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 3: Recurrent LMs
    LMs - Part 3: This lecture explains how we can build a LM using RNNs. We take a look at LSTM-based LMs.
    [link]

    Lecture Notes:

    Further Reads:

    • Recurrent LMs: Chapter 8 of [JM]
    • LSTM LMs: Paper Regularizing and Optimizing LSTM Language Models by Stephen Merity, Nitish Shirish Keskar, and Richard Socher published in ICLR 2018 enabling LSTMs to perform strongly on word-level language modeling
    • High-Rank Recurrent LMs: Paper Breaking the Softmax Bottleneck: A High-Rank RNN Language Model by Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen presented at ICLR 2018 proposing Mixture of Softmaxes (MoS) and achieving state-of-the-art results at the time
  • Lecture 4: Context Extraction via Self-Attention
    Transformer LMs - Part 1: In this lecture we use self-attention mechanism to extract context from a token sequence. Introduction to self-attention is given through the lecture.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 5: Transformer LM
    Transformer LMs - Part 2: We use self-attention-based context to build a LM via a Transformer.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 6: LLM Examples
    LLMs - Part 1: We take a look at GPT examples to see how deep they are, how large the data is, how to evaluate them, and what challenges are in the way of pre-training them.
    [link]

    Lecture Notes:

    Further Reads:

    • GPT-1: Paper Improving Language Understanding by Generative Pre-Training by Alec Radford et al. (OpenAI, 2018) that introduced GPT-1 and revived the idea of pretraining transformers as LMs followed by supervised fine-tuning
    • GPT-2: Paper Language Models are Unsupervised Multitask Learners by Alec Radford et al. (OpenAI, 2019) that introduces GPT-2 with 1.5B parameter trained on web text
    • GPT-3: Paper Language Models are Few-Shot Learners by Tom B. Brown et al. (OpenAI, 2020) that introduces GPT-3, a 175B-parameter transformer LM

    • GPT-4: GPT-4 Technical Report by OpenAI (2023) that provides an overview of GPT-4’s capabilities

    • The Pile: Paper The Pile: An 800GB Dataset of Diverse Text for Language Modeling by Leo Gao et al. presented in 2020 introductin dataset The Pile
    • Documentation Debt: Paper Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus by Jack Bandy and Nicholas Vincent published in 2021 discussing the efficiency and legality of data collection by looking into BookCorpus
  • Lecture 7: Pre-training vs Fine-tuning
    LLMs - Part 2: We discuss the idea of pre-training and fine-tuning, and take a look into the example of GPT-1.
    [link]

    Lecture Notes:

    Further Reads:

    • SSL: Paper Semi-supervised Sequence Learning by Andrew M. Dai et al. published in 2015 that explores using unsupervised pretraining followed by supervised fine-tuning; this was an early solid work advocating pre-training idea for LMs
    • GPT-1: Paper Improving Language Understanding by Generative Pre-Training by Alec Radford et al. (OpenAI, 2018) that introduced GPT-1 and revived the idea of pretraining transformers as LMs followed by supervised fine-tuning
  • Lecture 8: Statistical View and LoRA
    LLMs - Part 3: We practice statistical view on the idea of fine-tuning, discuss full and selective fine-tuning, and study the low-rank adaptation (LoRA) method.
    [link]

    Lecture Notes:

    Further Reads:

    • LMs: Chapter 12 of [BB] Section 12.3.5
    • LoRA: Paper LoRA: Low-Rank Adaptation of Large Language Models by Edward J. Hu et al. presented at ICLR in 2022 introducing LoRA
  • Lecture 9: Prompt Design
    LLMs - Part 4: In this lecture, we discuss the idea prompt design, i.e., using LLMs to sample directly from a task specific distribution. We get to overview ideas like "few-shot" and "zero-shot" learning and get to know what "foundation models" are.
    [link]

    Lecture Notes:

    Further Reads:

    • Chain-of-Thought: Paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models by Jason Wei et al. presented at NeurIPS in 2022 introducing chain-of-thought prompting
    • Prefix-Tuning: Paper Prefix-Tuning: Optimizing Continuous Prompts for Generation by Xiang Lisa Li et al. presented at ACL in 2021 proposing prefix-tuning approach for prompting
    • Prompt-Tuning: Paper The Power of Scale for Parameter-Efficient Prompt Tuning by B. Lester et al. presented at EMNLP in 2021 proposing the prompt tuning idea, i.e., learning to prompt
    • Zero-Shot LLMs: Paper Large Language Models are Zero-Shot Reasoners by T. Kojima et al. presented at NeurIPS in 2022 studying zero-shot learning with LLMs
  • Lecture 10: Data Generation Problem - Basic Definitions
    Formulation - Part 1: In this lecture, we start with the formulating the generic problem of data generation. We review the concept of "data distribution" and see that we essentially need to learn "how to sample from it".
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 11: Discriminative vs Generative Learning
    Formulation - Part 2: In this lecture we study the discriminative and generative models. We see that many earlier computational models we learned are indeed discriminative. We further learn how we could use a generative model for a discriminative learning task.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 12: Naive Bayes - Most Basic Generative Model
    Formulation - Part 3: This lecture investigates Naive Bayes, the most basic generative model we can think of. This model enables us understand the idea of generative modeling clearly.
    [link]

    Lecture Notes:

    Further Reads:

    • Naive Bayes: Paper Idiot’s Bayes—Not So Stupid After All? by D. Hand and K. Yu published at Statistical Review in 2001 discussing the efficiency of Naive Bayes for classification
    • Naive Bayes vs Linear Regression: Paper On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes by A. Ng and M. Jordan presented at NeurIPS in 2001 elaborating the data-efficiency efficiency of Naive Bayes and asymptotic superiority of Logistic Regression
    • Generative Models – Overview: Chapter 20 of [M] Sections 20.1 to 20.3
  • Lecture 13: Explicit Distribution Learning - Sampling
    Explicit Methods - Part 1: In this lecture, we start with "explicit learning methods"; the term we use to refer to approaches that learn data distribution explicitly. We see that direct sampling from them is not easy and we need to develop an structured modeling.
    [link]

    Lecture Notes:

    Further Reads:

    • Sampling Overview: Chapter 14 of [BB]
    • Sampling The book Pattern Recognition and Machine Learning by Christopher Bishop. Read Chapter 11 to know about how challenging sampling from a distribution is
    • Sampling Methods: Chapter 17 of [GYC] Sections 17.1 and 17.2
  • Lecture 14: Maximum Likelihood Learning
    Explicit Methods - Part 2: We discuss the general recipe for distribution learning, i.e., maximum likelihood estimation (MLE). We see that MLE essentially minimizes the estimated KL divergence between our model and the data distribution.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 15: Autoregressive Modeling
    AR Models - Part 1: This lecture talks about the idea of autoregressive (AR) modeling. We see that using this approach, we can design an efficient model for data distribution. The price that we pay though is slow generation.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 16: Computational AR Models
    AR Models - Part 2: This lectures goes through a general framework for developing a computational AR model. These models extract a masked content and compute a conditional distribution based on that. Generation in these model is always slow.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 17: PixelRNN
    AR Models - Part 3: In this lecture, we look at PixelRNN; a visual autoregressive model which builds context by recursion. We take a look at its training and sampling.
    [link]

    Lecture Notes:

    Further Reads:

    • PixelRNN and PixelCNN: Paper Pixel Recurrent Neural Networks by A. Oord et al. presented at ICML in 2016 proposing PixelRNN and PixelCNN
  • Lecture 18: Masked AR Models - PixelCNN and ImageGPT
    AR Models - Part 4: This lecture goes through the idea of masked AR models. These models give us the benefit of parallel computation, and hence can be efficiently trained using Teacher-Forcing Training. We look into the examples of PixelCNN and ImageGPT. The former uses masked convolution to realize an AR model, while the latter use masked decoding.
    [link]

    Lecture Notes:

    Further Reads:

    • PixelRNN and PixelCNN: Paper Pixel Recurrent Neural Networks by A. Oord et al. presented at ICML in 2016 proposing PixelRNN and PixelCNN
    • ImageGPT: Paper Generative Pretraining from Pixels by M. Chen et al. presented at ICML in 2020 proposing ImageGPT
  • Lecture 19: Energy Based Models - Boltzmann Distribution
    EBMs - Part 1: This is the first lecture on EBMs. We talk about Boltzmann distribution and how we could use it to build a distribution model from an arbitrary computational model. We call such models EBMs.
    [link]

    Lecture Notes:

    Further Reads:

    • EBMs: Chapter 24 of [M]
    • Partition Function and Normalizing: Chapter 16 of [GYC] Section 16.2
    • Universality of EBMs Paper Representational power of restricted Boltzmann machines and deep belief networks, by N. Le Roux and Y. Bengio published at Neural Computation in 2008 elaborating the representational power of EBMs *Tutorial on EBMs Survey A Tutorial on Energy-Based Learning, by Y. LeCun et al. published in 2006
  • Lecture 20: Computational EBMs - Training and Sampling
    EBMs - Part 2: In this lecture, we discuss computational EBMs. We see how we can train them if we know how to sample them. This motivates us to study MCMC algorithms for sampling.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 21: MCMC Algorithms - Gibbs Sampling
    EBMs - Part 3: In this lecture, we study the Gibbs sampling algorithm. We learn how we can use it to train and sample a basic Boltzmann machine.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 22: MCMC - Langevin and Conservative Divergence
    EBMs - Part 4: In this lecture, we study the Langevin Algorithm, a first-order MCMC algorithm for sampling. We learn how we can use this algorithm to develop the so-called conservative divergence approach for EBM training. This leads us to the idea of score-matching, which we will introduce briefly and leave its details for Chapter 6.
    [link]

    Lecture Notes:

    Further Reads:

    • Gibbs Sampling and Langevin: Chapter 14 of [BB]
    • Conservative Divergence Paper Training Products of Experts by Minimizing Contrastive Divergence, by G. Hinton published at Neural Computation in 2002 proposing the idea of Conservative Divergence
    • Training by MCMC Paper Implicit Generation and Generalization in Energy-Based Models published by Y. Du and I. Mordatch in NeurIPS 2019 discussing efficiency of MCMC algorithms for EBM training
    • Improved CD Paper Improved Contrastive Divergence Training of Energy-Based Models published by Y. Du et al. in ICML 2021 proposing an efficient training based on Hinton’s CD ideal
  • Lecture 23: Latent Space
    Notion of Latent: In this lecture, we study the notion of Latent Space. This notion enables us to develop latent-space generative models which include most current state-of-the-art generative AI approaches.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 24: Normalizing Flow
    Flow-based Models - Part I: In this section, we start with flow-based models. To this end, we study the idea of normalizing flow.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 25: Learning Flow
    Flow-based Models - Part II: We study the flow-based models which use normalizing flow to learn data distribution. We discuss their training and sampling and find out how complex they are.
    [link]

    Lecture Notes:

    Further Reads:

    • Flow-based Models: Chapter 23 of [M]
    • Tutorial on Normalizing Flow Paper Normalizing Flows for Probabilistic Modeling and Inference published by G. Papamakarios et al. at JMLR in 2021 discussing the training and inference of flow-based models
  • Lecture 26: NICE, RealNVP and Glow
    Flow-based Models - Part III: We discuss the Real NVP architecture, a classical flow model which has inspired many other flow-based models.
    [link]

    Lecture Notes:

    Further Reads:

    • NICE Paper NICE: Non-linear Independent Components Estimation published by L. Dinh et al. at ICLR in 2015 proposing the NICE model
    • Real NVP Paper Density estimation using Real NVP published by L. Dinh et al. at ICLR in 2017 proposing the Real NVP model
    • Glow Paper Glow: Generative Flow with Invertible 1x1 Convolutions published by D. Kingma and P. Dhariwal at NeurIPS in 2018 proposing the Glow model
  • Lecture 27: Introduction to GAN
    GANs - Part I: We start with GANs. We see that though looking like a flow model, they are unable to use direct MLE due to challenges involved in likelihood computation. We then intuitively discuss adversarial mechanism used to train the generator.
    [link]

    Lecture Notes:

    Further Reads:

    • Tutorial on GANs Tutorial Generative Adversarial Networks given by I. Goodfellow at NeurIPS in 2016
  • Lecture 28: Vanilla GAN
    GANs - Part II: In this lecture, we study the vanilla GAN. We see how we can train it by implementing a min-max game. We discuss its training and sampling.
    [link]

    Lecture Notes:

    Further Reads:

    • GANs Paper Generative Adversarial Nets published by I. Goodfellow et al. at NeurIPS in 2014 proposing GANs
  • Lecture 29: Implicit MLE via GAN
    GANs - Part III: In this lecture, we see how GAN training can be interpreted as an implicit maximum-likelihood learning. This will serve us as a background to understand how Wasserstein GAN is working.
    [link]

    Lecture Notes:

    Further Reads:

    • GANs Paper Generative Adversarial Nets published by I. Goodfellow et al. at NeurIPS in 2014 proposing GANs
    • Tutorial on GANs Tutorial Generative Adversarial Networks given by I. Goodfellow at NeurIPS in 2016
  • Lecture 30: Wasserstein Distance
    W-GANs - Part I: In this lecture, we understand the notion of Wasserstein distance. This is going to help us develop W-GAN as an alternative approach to the vanilla GAN.
    [link]

    Lecture Notes:

    Further Reads:

    • W-GANs Paper Wasserstein GAN published by M. Arjovsky et al. at ICML in 2017 proposing Wasserstein GANs
    • Tutorial on GANs Tutorial Generative Adversarial Networks given by I. Goodfellow at NeurIPS in 2016
  • Lecture 31: Wasserstein GAN
    W-GANs - Part II: In this lecture, we develop W-GAN framework, which trains the generator and discriminator to minimize the Wasserstein distance between the data and model distributions.
    [link]

    Lecture Notes:

    Further Reads:

    • W-GANs Paper Wasserstein GAN published by M. Arjovsky et al. at ICML in 2017 proposing Wasserstein GANs
  • Lecture 32: GAN Samples
    Summary: We quickly go through well-known GAN architectures and summarize Chapter 4.
    [link]

    Lecture Notes:

    Further Reads:

    • DCGAN Paper Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks published by A. Radford et al. at ICLR in 2016 proposing DCGAN
    • StyleGAN Paper A Style-Based Generator Architecture for Generative Adversarial Networks published by T. Karras et al. at IEEE CVF in 2019 proposing Style GAN
    • BigGAN Paper Large Scale GAN Training for High Fidelity Natural Image Synthesis published by A. Brock et al. at ICLR in 2019 proposing BigGAN
    • SAGAN Paper Self-Attention Generative Adversarial Networks published by H. Zhang et al. at ICML in 2019 proposing Self-Attention GAN
  • Lecture 33: Probabilistic Latent-Space Generation
    VI - Part I: In this lecture, we discuss an alternative generator design in which data samples are computed from latent samples via a probabilistic model. This is the base approach used in VAEs and Diffusion models. We see that computing the likelihood in this case is not tractable. This motivates us to learn Variational Inference.
    [link]

    Lecture Notes:

    Further Reads:

    • Probabilistic Latent: Chapter 16 of [BB] Sections 16.1 and 16.2
    • Mixture Models Paper On the number of components in a Gaussian mixture model published by G. McLachlan and S. Rathnayake in 2014 reviewing some key properties of Gaussian mixtures and their approximation power
  • Lecture 34: Variational Inference
    VI - Part II: In this lecture, we study Variational Inference. This framework enables us to develop an implicit approach for estimating the likelihood of probabilistic generators.
    [link]

    Lecture Notes:

    Further Reads:

    • ELBO: Chapter 16 of [BB] Section 16.3
    • VI for Likelihood The early paper Computing Upper and Lower Bounds on Likelihoods in Intractable Networks published by T. Jaakkola and M. Jordan at UAI in 1996
    • Tutorials on VI Review paper Variational Inference: A Review for Statisticians published by D. Blei, A. Kucukelbir, and J. McAuliffe in 2016 giving a good overview on VI framework
    • Introduction to VI Book An Introduction to Variational Autoencoders written by D. Kingma and M. Welling and published by NOW in 2019

Review Lectures

Here, you can find review lectures on some key deep learning topics. It is strongly suggested that you watch these videos to recap those key concepts, as they are frequently used in the course.