Lectures

Here, you can find the recordings of the lecture videos.

  • Lecture 0: Course Overview and Logistics
    Overview: This lecture gives an overview on the course structure and the logistics. Unfortunately, the audio recording is not good due to the issue with the recording device.
    [link]

    Lecture Notes:

  • Lecture 1: Language Modeling
    LMs - Part 1: We start with LMs and understand how we can feed a text into it by doing the so-called "Tokenization" and "Embedding". We build simple language model called Bi-Gram and understand its limitations. This motivates us to build a context-aware LM. We look into the basic context-aware LM design via RNNs. Unfortunately, the audio recording is not good due to the issue with the recording device.
    [link]

    Lecture Notes:

    Further Reads:

    • Tokenization: Chapter 2 of [JM]
    • Embedding: Chapter 6 of [JM]
    • Original BPE Algorithm: Original BPE Algorithm proposed by Philip Gage in 1994
    • BPE for Tokenization: Paper Neural machine translation of rare words with subword units by Rico Sennrich, Barry Haddow, and Alexandra Birch presented in ACL 2016 that adapted BPE for NLP
    • LMs: Chapter 12 of [BB] Section 12.2
    • N-Gram LMs: Chapter 3 of Speech and Language Processing; Section 3.1 on N-gram LM
    • Maximum Likelihood: Chapter 2 of [BB] Section 2.3
    • Recurrent LMs: Chapter 8 of [JM]
    • LSTM LMs: Paper Regularizing and Optimizing LSTM Language Models by Stephen Merity, Nitish Shirish Keskar, and Richard Socher published in ICLR 2018 enabling LSTMs to perform strongly on word-level language modeling
    • High-Rank Recurrent LMs: Paper Breaking the Softmax Bottleneck: A High-Rank RNN Language Model by Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen presented at ICLR 2018 proposing Mixture of Softmaxes (MoS) and achieving state-of-the-art results at the time
  • Lecture 2 - Part 1/2: Transformer-based Language Models
    Transformer LMs: In this lecture we use self-attention mechanism to extract context from a token sequence. Introduction to self-attention is given through the lecture. We then used this mechanism to build a LM architecture which is used nowadays in practice.
    [link]

    Lecture Notes:

  • Lecture 2 - Part 2/2: Large Language Models
    LLMs: We study LLMs which are Large LMs trained on large corpora. We see how they can be evaluated, fine-tuned, and deployed via prompt design.
    [link]

    Lecture Notes:

    Further Reads:

    GPT Papers:

    • GPT-1: Paper Improving Language Understanding by Generative Pre-Training by Alec Radford et al. (OpenAI, 2018) that introduced GPT-1 and revived the idea of pretraining transformers as LMs followed by supervised fine-tuning
    • GPT-2: Paper Language Models are Unsupervised Multitask Learners by Alec Radford et al. (OpenAI, 2019) that introduces GPT-2 with 1.5B parameter trained on web text
    • GPT-3: Paper Language Models are Few-Shot Learners by Tom B. Brown et al. (OpenAI, 2020) that introduces GPT-3, a 175B-parameter transformer LM
    • GPT-4: GPT-4 Technical Report by OpenAI (2023) that provides an overview of GPT-4’s capabilities

    Data for LLMs:

    • The Pile: Paper The Pile: An 800GB Dataset of Diverse Text for Language Modeling by Leo Gao et al. presented in 2020 introductin dataset The Pile
    • Documentation Debt: Paper Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus by Jack Bandy and Nicholas Vincent published in 2021 discussing the efficiency and legality of data collection by looking into BookCorpus

    Fine-tuning:

    • SSL: Paper Semi-supervised Sequence Learning by Andrew M. Dai et al. published in 2015 that explores using unsupervised pretraining followed by supervised fine-tuning; this was an early solid work advocating pre-training idea for LMs
    • LoRA: Paper LoRA: Low-Rank Adaptation of Large Language Models by Edward J. Hu et al. presented at ICLR in 2022 introducing LoRA

    Prompt Design:

    • Prefix-Tuning: Paper Prefix-Tuning: Optimizing Continuous Prompts for Generation by Xiang Lisa Li et al. presented at ACL in 2021 proposing prefix-tuning approach for prompting
    • Prompt-Tuning: Paper The Power of Scale for Parameter-Efficient Prompt Tuning by B. Lester et al. presented at EMNLP in 2021 proposing the prompt tuning idea, i.e., learning to prompt
    • Zero-Shot LLMs: Paper Large Language Models are Zero-Shot Reasoners by T. Kojima et al. presented at NeurIPS in 2022 studying zero-shot learning with LLMs
  • Lecture 3 - Part 1/3: Fundamentals of Data Generation
    Generative Learning - Part 1: In this lecture, we start with the formulating the generic problem of data generation. We review the concept of "data distribution" and see that we essentially need to learn how to sample from it.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 3 - Part 2/3: Discriminative vs Generative Learning
    Generative Learning - Part 2: We study the discriminative and generative models. We see that many classical computational models we use in practice are indeed discriminative models. We further learn how we could use a generative model for a discriminative learning task.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 3 - Part 3/3: Generative Learning and Naive Bayes
    Generative Learning - Part 3: We study the main definitions in Generative Learning. We then look into the Naive Bayes algorithm, the most basic generative learning algorithm we can think of. This enables us understand the idea of generative modeling clearly.
    [link]

    Lecture Notes:

    Further Reads:

    • Naive Bayes: Paper Idiot’s Bayes—Not So Stupid After All? by D. Hand and K. Yu published at Statistical Review in 2001 discussing the efficiency of Naive Bayes for classification
    • Naive Bayes vs Linear Regression: Paper On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes by A. Ng and M. Jordan presented at NeurIPS in 2001 elaborating the data-efficiency efficiency of Naive Bayes and asymptotic superiority of Logistic Regression
    • Generative Models – Overview: Chapter 20 of [M] Sections 20.1 to 20.3
  • Lecture 4 - Part 1/2: Autoregressive Models
    AR Models - Part 1: In this lecture, we start with explicit learning methods; the term we use to refer to approaches that learn data distribution explicitly. As the first class of models, we study the autoregressive ones.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 4 - Part 2/2: Computational Autoregressive Models
    AR Models - Part 2: We go through a general framework for developing a computational AR model. These models extract a masked content and compute a conditional distribution based on that. Generation in these model is always slow. We look into the example of PixelCNN.
    [link]

    Lecture Notes:

    Further Reads:

    • PixelRNN and PixelCNN: Paper Pixel Recurrent Neural Networks by A. Oord et al. presented at ICML in 2016 proposing PixelRNN and PixelCNN
    • ImageGPT: Paper Generative Pretraining from Pixels by M. Chen et al. presented at ICML in 2020 proposing ImageGPT
  • Lecture 5 - Part 1/2: Energy Based Models
    EBMs - Part 1: We talk about Boltzmann distribution and how we could use it to build a distribution model from an arbitrary computational model. We call such models EBMs. We see how we can train them if we know how to sample them. This motivates us to study MCMC algorithms for sampling.
    [link]

    Lecture Notes:

    Further Reads:

    • EBMs: Chapter 24 of [M]
    • Partition Function and Normalizing: Chapter 16 of [GYC] Section 16.2
    • Universality of EBMs Paper Representational power of restricted Boltzmann machines and deep belief networks, by N. Le Roux and Y. Bengio published at Neural Computation in 2008 elaborating the representational power of EBMs
    • Tutorial on EBMs Survey A Tutorial on Energy-Based Learning, by Y. LeCun et al. published in 2006
  • Lecture 5 - Part 2/2: EBMs and MCMC Algorithms
    EBMs - Part 2: We next study the MCMC sampling, looking into Gibbs sampling and Langevin algorithms. We learn how we can use them to train an EBM. This leads ud to contrastive learning idea. We further discuss the idea of score-matching, which we will help us later on to develop Diffusion Models.
    [link]

    Lecture Notes:

    Further Reads:

    • MCMC Algorithms: Chapter 12 of [M] Sections 12.3, 12.6 and 12.7
    • Gibbs Sampling and Langevin: Chapter 14 of [BB]
    • Contrastive Divergence Paper Training Products of Experts by Minimizing Contrastive Divergence, by G. Hinton published at Neural Computation in 2002 proposing the idea of Contrastive Divergence
    • Training by MCMC Paper Implicit Generation and Generalization in Energy-Based Models published by Y. Du and I. Mordatch in NeurIPS 2019 discussing efficiency of MCMC algorithms for EBM training
    • Improved CD Paper Improved Contrastive Divergence Training of Energy-Based Models published by Y. Du et al. in ICML 2021 proposing an efficient training based on Hinton’s CD ideal
  • Lecture 6 - Part 1/2: Normalizing Flow
    Flow Models I: In this lecture, we study the flow-based models which use normalizing flow to learn data distribution. We start with the notion of Latent Space. This notion enables us to develop latent-space generative models which include most current state-of-the-art generative AI approaches.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 6 - Part 2/2: Flow-based Models
    Flow Models II: We discuss their training and sampling of flow-based models and find out how complex they are. We investigate the Real NVP architecture, a classical flow model which has inspired many other flow-based models.
    [link]

    Lecture Notes:

    Further Reads:

    • Flow-based Models: Chapter 23 of [M]
    • Tutorial on Normalizing Flow Paper Normalizing Flows for Probabilistic Modeling and Inference published by G. Papamakarios et al. at JMLR in 2021 discussing the training and inference of flow-based models
    • Real NVP Paper Density estimation using Real NVP published by L. Dinh et al. at ICLR in 2017 proposing the Real NVP model
    • Flow Matching Paper Flow Matching for Generative Modeling published by Y. Lipman et al. at ICLR 2023
  • Lecture 7 - Part 1/2: Generative Adversarial Nets
    GANs - Part I: (Unfortunately, the recording did not work, so this is an older recording from last year.) We start with GANs. We see that though looking like a flow model, they are unable to use direct MLE due to challenges involved in likelihood computation. We then intuitively discuss adversarial mechanism used to train the generator. We see how we can train it by implementing a min-max game. We discuss its training and sampling. We see how GAN training can be interpreted as an implicit maximum-likelihood learning. This will serve us as a background to understand how Wasserstein GAN is working.
    [link]

    Lecture Notes:

    Further Reads:

    • Tutorial on GANs Tutorial Generative Adversarial Networks given by I. Goodfellow at NeurIPS in 2016
    • GANs Paper Generative Adversarial Nets published by I. Goodfellow et al. at NeurIPS in 2014 proposing GANs
  • Lecture 7 - Part 2/2: Wasserstein GAN
    GANs - Part II: (Unfortunately, the recording did not work. This is a Zoom recoding, which has low quality audio.) We understand the notion of Wasserstein distance. Using this notion, we develop WGAN, which trains the generator and discriminator to minimize the Wasserstein distance between the data and model distributions. At the end of this lecture, we further go through well-known GAN architectures.
    [link]

    Lecture Notes:

    Further Reads:

    • WGAN Paper Wasserstein GAN published by M. Arjovsky et al. at ICML in 2017 proposing Wasserstein GANs
    • DCGAN Paper Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks published by A. Radford et al. at ICLR in 2016 proposing DCGAN
    • StyleGAN Paper A Style-Based Generator Architecture for Generative Adversarial Networks published by T. Karras et al. at IEEE CVF in 2019 proposing Style GAN
    • BigGAN Paper Large Scale GAN Training for High Fidelity Natural Image Synthesis published by A. Brock et al. at ICLR in 2019 proposing BigGAN
    • SAGAN Paper Self-Attention Generative Adversarial Networks published by H. Zhang et al. at ICML in 2019 proposing Self-Attention GAN
  • Lecture 8 - Part 1/2: Probabilistic Latent-Space Generation
    VAE - Part I: In this lecture, we discuss an alternative generator design in which data samples are computed from latent samples via a probabilistic model. This is the base approach used in VAEs and Diffusion models. We see that computing the likelihood in this case is not tractable. This motivates us to learn Variational Inference.
    [link]

    Lecture Notes:

    Further Reads:

    • Probabilistic Latent: Chapter 16 of [BB] Sections 16.1 and 16.2
    • Mixture Models Paper On the number of components in a Gaussian mixture model published by G. McLachlan and S. Rathnayake in 2014 reviewing some key properties of Gaussian mixtures and their approximation power
  • Lecture 8 - Part 2/2: Variational Inference
    VAE - Part II: In this lecture, we study Variational Inference. This framework enables us to develop an implicit approach for estimating the likelihood of probabilistic generators. Using that we can build a training loop for probabilistic generators utilizing the evidence lower bound (ELBO). This is the key training approach used in VAEs.
    [link]

    Lecture Notes:

    Further Reads:

    • ELBO: Chapter 16 of [BB] Section 16.3
    • VI for Likelihood The early paper Computing Upper and Lower Bounds on Likelihoods in Intractable Networks published by T. Jaakkola and M. Jordan at UAI in 1996
    • Tutorials on VI Review paper Variational Inference: A Review for Statisticians published by D. Blei, A. Kucukelbir, and J. McAuliffe in 2016 giving a good overview on VI framework
    • Introduction to VI Book An Introduction to Variational Autoencoders written by D. Kingma and M. Welling and published by NOW in 2019
  • Lecture 9 - Part 1/3: Variational Autoencoding
    VAE - Part I: In this lecture, we discuss VAE architecture and see how we could use VI framework to build an AE architecture for generation. We learn how the Encoder and Decoder of a VAE look like and discuss how we can define a risk function out of the ELBO to implicitly maximize the likelihood in VAEs.
    [link]

    Lecture Notes:

    Further Reads:

    • AE with VI Paper Auto-Encoding Variational Bayes published by D. Kingma and M. Welling in 2014 proposing VAE
    • Stachastic Generation by VAE Paper Stochastic Backpropagation and Approximate Inference in Deep Generative Models published by D. Rezende et al in 2014 proposing VAE in parallel
  • Lecture 9 - Part 2/3: Training Loop for VAEs
    VAE - Part II: In this lecture, we complete the training loop for a computational VAE. We see that for this, we need to compute the gradient on a stochastic computation graph. We can do this using Importance Sampling or Reparameterization trick.
    [link]

    Lecture Notes:

    Further Reads:

    • DCVAE Paper Semi-Supervised Learning with Deep Generative Models published by D. Kingma et al. in 2014 implementing a Deep Convolutional VAE
    • Transformer VAE Paper Transformer VAE: A Hierarchical Model for Structure-Aware and Interpretable Music Representation Learning published by J. Jiang et al in ICASSP 2020 proposing a Transformer based VAE
    • VAE with VampPrior Paper VAE with a VampPrior published by J. Tomczak and M. Welling in 2017 proposing VAE with general latent prior
  • Lecture 9 - Part 3/3: Vector-Quantized VAE
    VQ-VAE: We discuss VAEs with discrete latent, in particular the VQ-VAE. We see the idea behind quantizing the latent and its implementational framework.
    [link]

    Lecture Notes:

    Further Reads:

    • VQ-VAE Paper Neural Discrete Representation Learning published by D. Kingma and M. Welling in NeurIPS 2017 proposing VQ-VAE
  • Lecture 10 - Part 1/2: Generation by Langevin Dynamics
    Diffusion - Part I: In this lecture we start with the most basic diffusion-based generation framework which uses Langevin Dynamics to sample from data distribution. We understand that we only need to know the score function to generate by this approach. This leads to the conventional score matching algorithm.
    [link]

    Lecture Notes:

    Further Reads:

    • Score Matching Paper Estimation of non-normalized statistical models by score matching published by A. Hyvärinen in 2005 proposing the computational score matching
    • Langevin Generation Paper Generative Modeling by Estimating Gradients of the Data Distribution published by Song and Ermon in NeurIPS 2019 explaining the score matching and its application to generation by Langevin Dynamics
  • Lecture 10 - Part 2/2: Score-based Diffusion
    Diffusion - Part II: We learn about a general diffusion that is described by a SDE. We also learn how we could reverse it in time. This takes us towards more practical diffusion-based models. We design an SDE and its reverse which is able to send us from data sample to Gaussian noise and back from noise to data. We also find out what we need to learn in order to be able to get back from noise to data. We go through diffusion score matching idea which enables us to learn the required scores for data generation on a reverse SDE. This enables us to sample from data distribution via a diffusion process without learning the data score directly.
    [link]

    Lecture Notes:

    Further Reads:

    • Reverse-time Diffusion Paper Reverse-time diffusion equation models published in Elsevier by B. Anderson in 1982 esplaining the reverse-time diffusion process
    • SDE Approach Paper Maximum Likelihood Training of Score-Based Diffusion Models by Song et al. in NeurIPS 2021 explaining the DPM and DDPM from inverse Diffusion viewpoint
    • DSM Paper Estimation of non-normalized statistical models by score matching published in Neural Computation by Pascal Vincent in 2011 proposing the denoising approach for score estimation (DSM)
  • Lecture 11 - Part 1/4: Probabilistic Diffusion - Forward Process
    DPMs - Part I: Given the limitation of SDE-based diffusion building, we no move towards probabilistic diffusion models which have led to the current trends, e.g., DPMs and DDPMs. We first build the forward process and understand its features. This is an old record, as the teaching station the recording did not work on this session.
    [link]

    Lecture Notes:

    Further Reads:

    • DPM Paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics published by J. Sohl-Dickstein et al. in ICML 2015 proposing DPM framework for generation (from Variational Inference)
  • Lecture 11 - Part 2/4: Reverse Probabilistic Diffusion via MLE
    DPMs - Part II: In this lecture, we try to build a reverse Markov chain that builds the reverse trajectory from noise to data. We see that this problem can be formulated as a classic distribution learning that can be addressed by maximum likelihood estimation. Due to computational complexity though, we need to invoke variational inference to implicitly perform MLE. This is an old record, as the teaching station the recording did not work on this session.
    [link]

    Lecture Notes:

    Further Reads:

    • DPM Paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics published by J. Sohl-Dickstein et al. in ICML 2015 proposing DPM framework for generation (from Variational Inference)
  • Lecture 11 - Part 3/4: Risk Function for Probabilistic Diffusion
    DPMs - Part III: In this lecture, we derive the learning risk function that learns the reverse trajectory. To this end, we derive the ELBO on the log-likelihood of the reverse process and compute it for a general learnable Markov-chain. We then focus on the particular case of Gaussian denoising. This gives us a computable loss function, which can be used for training of probabilistic diffusion models. This is an old record, as the teaching station the recording did not work on this session.
    [link]

    Lecture Notes:

    Further Reads:

    • DPM Paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics published by J. Sohl-Dickstein et al. in ICML 2015 proposing DPM framework for generation (from Variational Inference)
  • Lecture 11 - Part 4/4: Computational DPMs
    DPMs - Part IV: In this lecture, we build a computational DPM. We see how we can compute the sample loss using time sampling. We sketch the training loop and Generation algorithm. This completes the chain of the last couple of lectures. This is an old record, as the teaching station the recording did not work on this session.
    [link]

    Lecture Notes:

    Further Reads:

    • DPM Paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics published by J. Sohl-Dickstein et al. in ICML 2015 proposing DPM framework for generation (from Variational Inference)
  • Lecture 12 - Part 1/2: Denoising DPMs and DDIM
    DDPMs: In this lecture, we learn the well-known case of DPMs, i.e., Denoising DPM (DDPM). We see how we build the denoiser in these models and how we can train and sample from them. This describes the most conventional form of diffusion models used in practice. We also study the Denoising Diffusion Implicit Models (DDIMs) which enable us to sample much faster as compared to DDPMs. Finally, we give some well-known examples for diffusion models that are currently being used in large scale.
    [link]

    Lecture Notes:

    Further Reads:

    • DDPM Paper Denoising Diffusion Probabilistic Models published by J. Ho et al. in NeurIPS 2020 proposing DDPM framework
    • Improved DDPM Paper Improved Denoising Diffusion Probabilistic Models published by A. Nichol and P. Dhariwal in ICML 2021 proposing improvements to DDPM
    • DDIM Paper Denoising Diffusion Implicit Models published by J. Song et al. in ICLR 2021 proposing DDIM framework
    • Stable Diffusion Paper High-Resolution Image Synthesis with Latent Diffusion Models published by R. Rombach et al. in IEEE CVPR 2022 proposing Stable Diffusion
  • Lecture 12 - Part 2/2: Multimodality and Conditional Generative Models
    Conditional Models I: In this lecture, we discuss the notion of multi-modality. We see that this notion is closely related to conditional generative models. We discuss how we could computationally condition a generative model via condition embedding. We further discuss the two particular techniques of Cross-Attention and FiLM.
    [link]

    Lecture Notes:

    Further Reads:

    • Text-to-Image Paper Learning Transferable Visual Models From Natural Language Supervision published by A. Radford et al. in ICML 2021 proposing visual data generation from raw text
    • FiLM Paper FiLM: Visual Reasoning with a General Conditioning Layer published by E. Perez et al. in AAAI 2018 proposing FiLM
    • Cross-Attention Conditioning Paper Multi-Modality Cross Attention Network for Image and Sentence Matching published by X. Wei et al. in IEEE CVPR 2020 proposing a cross-attention based approach for conditioning

Review Lectures

Here, you can find review lectures on some key deep learning topics. It is strongly suggested that you watch these videos to recap those key concepts, as they are frequently used in the course.