Schedule

Lecture Notes:

• Chapter 0

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• Intro to RL: Chapter 1 - Sections 1.1 and 1.2 of [SB]

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• k-armed Bandit: Chapter 2 - Section 2.1 of [SB]
• Robbins’ Paper: Paper Some aspects of the sequential design of experiments by H. Robbins published in the Bulletin of the American Mathematical Society in 1952 formulating multi-armed bandit as we know it nowadays

Lecture Notes:

• Chapter 1 - Section 1

Further Reads:

• k-armed Bandit: Chapter 2 - Section 2.1 of [SB]
• Robbins’ Paper: Paper Some aspects of the sequential design of experiments by H. Robbins published in the Bulletin of the American Mathematical Society in 1952 formulating multi-armed bandit as we know it nowadays

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• RL Setting: Chapter 3 - Sections 3.1 to 3.3 of [SB]

Lecture Notes:

• Chapter 1 - Section 2

Further Reads:

• RL Setting: Chapter 3 - Sections 3.1 to 3.3 of [SB]

Lecture Notes:

• Chapter 1 - Section 2

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• RL Setting: Chapter 1 - Sections 1.3 and 1.4 of [SB]

Lecture Notes:

• Chapter 1 - Section 3

Further Reads:

• RL Setting: Chapter 1 - Section 1.5 of [SB]

Lecture Notes:

• Chapter 1 - Section 3

Lecture Notes:

• Chapter 1 - Section 4

Further Reads:

• RL Setting: Chapter 3 - Section 3.4 of [SB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• RL Setting: Chapter 3 - Sections 3.1 and 3.3 of [SB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• RL Setting: Chapter 3 - Section 3.5 [SB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• Bellman: Chapter 3 - Section 3.6 [SB]

Lecture Notes:

• Chapter 2 - Section 1

Further Reads:

• Bellman: Chapter 3 - Section 3.6 [SB]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Optimal Policy: Chapter 3 - Section 3.7 [SB]

Lecture Notes:

• Chapter 2 - Section 2

Further Reads:

• Optimal Policy: Chapter 3 - Section 3.7 [SB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Policy Evaluation: Chapter 4 - Section 4.1 of [SB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Policy Improvement and Iteration: Chapter 4 - Sections 4.2 and 4.3 of [SB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Value Iteration: Chapter 4 - Section 4.4 of [SB]

Lecture Notes:

• Chapter 2 - Section 3

Further Reads:

• Generalized Policy Iteration: Chapter 4 - Sections 4.6 and 4.7 of [SB]

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Monte-Carlo: Chapter 5 - Sections 5.1 and 5.2 of [SB]

Lecture Notes:

• Chapter 3 - Section 1

Further Reads:

• Monte-Carlo: Chapter 5 - Section 5.3 of [SB]

Lecture Notes:

• Chapter 3 - Section 2

Further Reads:

• TD-0: Chapter 6 - Section 6.1 of [SB]

• Chapter 3 - Section 2

Further Reads:

• TD-0: Chapter 6 - Sections 6.2 and 6.3 of [SB]

• Chapter 3 - Section 3

Further Reads:

• TD-n: Chapter 7 - Sections 7.1 and 7.2 of [SB]

• Chapter 3 - Section 3

Further Reads:

• TD-lambda: Chapter 12 - Sections 12.1 to 12.3 of [SB]

• Chapter 3 - Section 3

Further Reads:

• Eligibility Tracing: Chapter 12 - Sections 12.4 and 12.5 of [SB]

• Chapter 3 - Section 4

Further Reads:

• MC Control: Chapter 5 - Sections 5.3 and 5.4 of [SB]

• Chapter 3 - Section 4

Further Reads:

• ε-Greedy: Chapter 2 - Sections 2.5 and 2.6 of [SB]

• Chapter 3 - Section 5

Further Reads:

• Online Q-Learning Article On-Line Q-Learning Using Connectionist Systems published in 1994 by G. Rummery and M. Niranjan proposing SARSA as an online version of Q-Learning
• Sarsa: Chapter 6 - Section 6.4 of [SB]
• Sarsa: Chapter 10 - Sections 10.2 and 10.5 of [SB]
• Sarsa: Chapter 12 - Section 12.7 of [SB]

• Chapter 3 - Section 5

Further Reads:

• Importance Sampling: Chapter 5 - Section 5.5 of [SB]
• Off Policy Learning: Chapter 12 - Sections 12.9 and 12.11 of [SB]

• Chapter 3 - Section 5

Further Reads:

• Q-Learning Paper Paper Q-learning published in 1992 by _C Watkins and P. Dayan_proposing the off-policy learning as in Q-learning algorithm
• Q-Learning: Chapter 6 - Section 6.5 of [SB]
• Q-Learning: Chapter 12 - Section 12.10 of [SB]

• Chapter 3 - Section 5

Further Reads: *Convergence Paper The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning published in 2000 by V. Borkar and S. Meyn studying convergence of Q-Learning and SARSA

• Chapter 4 - Section 1

Further Reads:

• Neuro-dynamic Programming Paper Neuro-dynamic programming: an overview published in 1995 by D. Bertsekas and J. Tsitsiklis discussing function approximation for value learning

• Chapter 4 - Section 1

Further Reads:

• Function Approximation for RL: Chapter 9 of [SB]
• Neuro-dynamic Programming Paper Neuro-dynamic programming: an overview published in 1995 by D. Bertsekas and J. Tsitsiklis discussing function approximation for value learning

• Chapter 4 - Section 1

Further Reads:

• Prediction with FA: Chapter 3 of [CS]

• Chapter 4 - Section 2

Further Reads:

• TD with FA Paper Analysis of Temporal-Diffference Learning with Function Approximation published in 1996 by J. Tsitsiklis and B. Roy analyzing Prediction with parameterized models

• Chapter 4 - Section 2

Further Reads:

• FA vs Tabular Paper Analyzing feature generation for value-function approximation published in 2008 by R. Parr et al. discussing connections of RL with FA to tabular RL

• Chapter 4 - Section 2

Further Reads:

• RL with FA Paper Residual Algorithms: Reinforcement Learning with Function Approximation published in 1995 by L. BAird giving some critics to RL with FA

• Chapter 4 - Section 4

Further Reads:

• Control with FA: Chapter 4 of [CS]

• Chapter 4 - Section 4

Further Reads:

• DQL: Chapter 4 - Section 4.3 of [CS]
• Deep Q-Learning Paper Human-level control through deep reinforcement learning published in 2015 by V. Mnih et al. proposing the legendary idea of Deep Q-Learning
• DQL Paper I Paper Playing Atari with Deep Reinforcement Learning published in 2013 by V. Mnih et al. describing DQL details

• Chapter 4 - Section 4

Further Reads:

• DQL Paper I Paper Playing Atari with Deep Reinforcement Learning published in 2013 by V. Mnih et al. describing DQL details

• Chapter 4 - Section 4

Further Reads:

• DQL Paper II Paper Deep Reinforcement Learning with Double Q-learning published in 2015 by H. Haasselt et al. proposing Double DQL
• DQL Paper III Paper Dueling Network Architectures for Deep Reinforcement Learning published in 2016 by Z. Wang et al. proposing Dueling DQL
• DQL Paper IV Paper Prioritized Experience Replay published in 2016 by T. Schaul et al. proposing a prioritizing experience replay scheme
• Gorila Paper Massively Parallel Methods for Deep Reinforcement Learning published in 2015 by A. Nair et al. proposing Gorila

• Chapter 4 - Section 4

Further Reads:

• Why Policy Net Article Deep Deterministic Policy Gradient at OpenAI Spinning Up

• Chapter 5 - Section 1

Further Reads:

• REINFORCE Paper Simple statistical gradient-following algorithms for connectionist reinforcement learning published by R. Williams in 1992 introducing REINFORCE algorithm

• Chapter 5 - Section 1

Further Reads:

• PGM Theorem Paper Policy Gradient Methods for Reinforcement Learning with Function Approximation published by R. Sutton et al. in 1999 developing the Policy Gradient Theorem

• Chapter 5 - Section 1

Further Reads:

• PGM Theorem Paper Policy Gradient Methods for Reinforcement Learning with Function Approximation published by R. Sutton et al. in 1999 developing the Policy Gradient Theorem

• Chapter 5 - Section 2

Further Reads:

• Baseline Paper Policy invariance under reward transformations: Theory and application to reward shaping published by A. Ng et al. in 1999

• Chapter 5 - Section 3

Further Reads:

• Nat PGM Paper A Natural Policy Gradient published by S. Kakade in 2001 proposing a basic natural PGM
• TRPO Paper Trust Region Policy Optimization published by J. Schulman et al. in 2015 proposing TRPO

• Chapter 5 - Section 3

Further Reads:

• Nat PGM Paper A Natural Policy Gradient published by S. Kakade in 2001 proposing a basic natural PGM
• TRPO Paper Trust Region Policy Optimization published by J. Schulman et al. in 2015 proposing TRPO

• Chapter 5 - Section 4

Further Reads:

• TRPO Paper Trust Region Policy Optimization published by J. Schulman et al. in 2015 proposing TRPO

• Chapter 5 - Section 4

Further Reads:

• PPO Paper Proximal Policy Optimization Algorithms published by J. Schulman et al. in 2017 proposing PPO