Lectures

Here, you can find the recordings of the lecture videos.

  • Lecture 0: Course Overview and Logistics
    Overview: In this lecture, we go through the course logistics. The audio quality is poor, as the teaching station did not work.
    [link]

    Lecture Notes:

  • Lecture 1: RL Framework
    Introduction: We start with the multi-armed bandit problem, and see how optimally or randomly playing could change the collected reward. We look at a mix strategy, first explore the environment for some limited time and then exploit this exploration in the real-time. We see that we could get very close to the optimal reward if we play for a long enough time. We then formulate the RL setting for a general interactive problem. We get to know what the agent is, what the environment is, and how the interactions between them is formulated. We look into a few sample environment, namely Tic-Tac-Toe, Cart-Pole, and Trading examples. For each, we formulate the State, Action, and Reward. Finally, we introduce Gymnasium.
    [link]

    Lecture Notes:

    Further Reads:

    • Intro to RL: Chapter 1 - Sections 1.1 and 1.2 of [SB]
    • k-armed Bandit: Chapter 2 - Section 2.1 of [SB]
    • Robbins’ Paper: Paper Some aspects of the sequential design of experiments by H. Robbins published in the Bulletin of the American Mathematical Society in 1952 formulating multi-armed bandit as we know it nowadays
    • RL Setting: Chapter 3 - Sections 3.1 to 3.3 of [SB]
  • Lecture 2: MDPs and Bellman
    Model-based RL - Part I: We define mathematically the notions of policy, value, action-value, and optimal policy. This formulates concretely the main objective in RL. We then start with model-based RL, where the environment is modeled as a known MDP. We can evaluate a given policy simply using statistical analysis. We derive the Bellman equation for the value and action-value functions. These equations enable us to compute the values or action-values from a set of fixed-point equations without needing to take nested expectations.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 3: Policy and Value Iteration
    Model-based RL - Part II: We derive the Bellman Optimality equation. This equation enables us to compute the optimal value function without needing to know the optimal policy. This further let us derive the optimal policy for an environment with known MDP. We then use dynamic programming to find the solution to the Bellman equation. We can do this in two different ways. (i) We could start with a policy, evaluate it with DP and then improve the policy iteratively. This will lead to policy iteration algorithm. (ii) Alternatively, we can iterate to find the optimal values and then specify the optimal policy out of it. This is value iteration. We look at both algorithms and also merge them into the unified generalized policy iteration scheme.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 4: Generalized Policy Iteration via Monte-Carlo and Temporal Difference
    Model-Free RL - Part I: We start with model-free approaches for RL. We see that using Monte-Carlo estimate, we can estimate the values needed in a Generalized Policy Iteration algorithm. We hence develop a model-free GPI loop, where we use Monte-Carlo to estimate all required values directly from sample trajectories. We then use Bellman equation to estimate the values using bootstraping. This approach is called Temporal Difference (TD). We develop a GPI loop using TD, as well.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 5: Deep Bootstraping and epsilon-Greedy Improvement
    Model-Free RL - Part II: We study deep bootstraping. This enables us to extend the TD idea to TD-n. We then use this idea to find a sample efficient estimator of values, i.e., TD-lambda. The idea is though not online. We address this issue by doing Eligibility Tracing back in time. This leads to an online and efficient evaluation scheme. We then use these ideas to perform online control. We see that greedy improvement can lead to poor control, due to lack of exploration. To address this issue, we design the epsilon-greedy improvement, which let us explore and exploit at the same time.
    [link]

    Lecture Notes:

    Further Reads:

  • Lecture 6: SARSA Algorithms and Q-Learning
    SARSA: We build a control loop using TD estimation. This enables us to learn the optimal policy while we are playing with it. The final algorithm is called State-Action Reward State-Action or simply SARSA. SARSA is the most used form of on-policy RL. We then discuss off-policy RL, and its specific form Q-learning. Here, we play epsilon-greedy, but we learn the value of the greedy policy. This leads to a simple control loop, often called the Q-Learning Algorithm. We discuss the requirements for both algorithms to converge. This is consistent with the notion of GLIE algorithms.
    [link]

    Further Reads:

    • Online Q-Learning Article On-Line Q-Learning Using Connectionist Systems published in 1994 by G. Rummery and M. Niranjan proposing SARSA as an online version of Q-Learning
    • Sarsa: Chapters 6, 10, and 12 of [SB]
    • Q-Learning: Chapters 6 and 12 of [SB]
    • Q-Learning Paper Paper Q-learning published in 1992 by C Watkins and P. Dayan_proposing the off-policy learning as in Q-learning algorithm *Convergence Paper _The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning published in 2000 by V. Borkar and S. Meyn studying convergence of Q-Learning and SARSA