Lectures

Here, you can find the recordings of the lecture videos.

• Lecture 0: Course Overview and Logistics
  Overview: This lecture gives an overview on the course structure and the logistics. Unfortunately, the audio recording is not good due to the issue with the recording device.
  [link]
  

  Lecture Notes:

  • Chapter 0
• Lecture 1: Language Modeling
  LMs - Part 1: We start with LMs and understand how we can feed a text into it by doing the so-called "Tokenization" and "Embedding". We build simple language model called Bi-Gram and understand its limitations. This motivates us to build a context-aware LM. We look into the basic context-aware LM design via RNNs. Unfortunately, the audio recording is not good due to the issue with the recording device.
  [link]
  

  Lecture Notes:

  • Chapter 1 - Section 1

  Further Reads:

  • Tokenization: Chapter 2 of [JM]
  • Embedding: Chapter 6 of [JM]
  • Original BPE Algorithm: Original BPE Algorithm proposed by Philip Gage in 1994
  • BPE for Tokenization: Paper Neural machine translation of rare words with subword units by Rico Sennrich, Barry Haddow, and Alexandra Birch presented in ACL 2016 that adapted BPE for NLP
  • LMs: Chapter 12 of [BB] Section 12.2
  • N-Gram LMs: Chapter 3 of Speech and Language Processing; Section 3.1 on N-gram LM
  • Maximum Likelihood: Chapter 2 of [BB] Sections 12.1 – 12.3
  • Recurrent LMs: Chapter 8 of [JM]
  • LSTM LMs: Paper Regularizing and Optimizing LSTM Language Models by Stephen Merity, Nitish Shirish Keskar, and Richard Socher published in ICLR 2018 enabling LSTMs to perform strongly on word-level language modeling
  • High-Rank Recurrent LMs: Paper Breaking the Softmax Bottleneck: A High-Rank RNN Language Model by Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen presented at ICLR 2018 proposing Mixture of Softmaxes (MoS) and achieving state-of-the-art results at the time

Review Lectures

Here, you can find review lectures on some key deep learning topics. It is strongly suggested that you watch these videos to recap those key concepts, as they are frequently used in the course.

• Review 1
  Sequence Data: This is a lecture on sequence data and RNNs.
  [link]
  

  Suggested Readings:

  • AplDL Notes: Sequence Data
  • AplDL Notes: Recurent NNs
• Review 2
  Deep RNNs and Gating: This is a lecture on deep RNNs.
  [link]
  

  Suggested Readings:

  • AplDL Notes: RNNs
  • AplDL Notes: Bidirectional RNNs
  • AplDL Notes: CTC Algorithm
  • Simple Language Models by RNNs
  • Sequence Generation with RNNs
• Review 3
  Seq2Seq: A lecture on sequence to sequence models.
  [link]
  

  Suggested Readings:

  • AplDL Notes: Basic Language Model
  • AplDL Notes: Encoder-Decoder
  • Seq2Seq Learning Paper
• Review 4
  Attention: A lecture on attention mechanism.
  [link]
  

  Suggested Readings:

  • AplDL Notes: Attention Mechanism
• Review 5
  Transformers: A lecture on self-attention and transformers.
  [link]
  

  Suggested Readings:

  • AplDL Notes: Self-Attention and Transformers
  • Attention Is All You Need!
  • Transformers - SLP CH9