Lecture Notes:

• Chapter 0

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• Tokenization: Chapter 2 of [JM]
• Embedding: Chapter 6 of [JM]
• Original BPE Algorithm: Original BPE Algorithm proposed by Philip Gage in 1994
• BPE for Tokenization: Paper Neural machine translation of rare words with subword units by Rico Sennrich, Barry Haddow, and Alexandra Birch presented in ACL 2016 that adapted BPE for NLP
• LMs: Chapter 12 of [BB] Section 12.2
• N-Gram LMs: Chapter 3 of Speech and Language Processing; Section 3.1 on N-gram LM
• Maximum Likelihood: Chapter 2 of [BB] Sections 12.1 – 12.3
• Recurrent LMs: Chapter 8 of [JM]
• LSTM LMs: Paper Regularizing and Optimizing LSTM Language Models by Stephen Merity, Nitish Shirish Keskar, and Richard Socher published in ICLR 2018 enabling LSTMs to perform strongly on word-level language modeling
• High-Rank Recurrent LMs: Paper Breaking the Softmax Bottleneck: A High-Rank RNN Language Model by Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen presented at ICLR 2018 proposing Mixture of Softmaxes (MoS) and achieving state-of-the-art results at the time