TYPED NOTES*, MST 2, AI ..//////.............summary...............////

What a Markov Decision Process (MDP) is: An MDP is a mathematical framework used in Reinforcement Learning to model decision-making. It describes an environment where an agent must choose actions in various states to maximize a cumulative reward, with outcomes being partly random and partly under the agent's control.

Core Components of an MDP: An MDP is formally defined by a set of states (S), a set of actions (A), a transition model (P) that gives the probability of moving to a new state after an action, and a reward function (R) that provides positive or negative feedback for state-action pairs.

The Goal is to Find an Optimal Policy: The solution to an MDP is an optimal policy (π), which is a mapping that tells the agent the best action to take in any given state. The policy is considered optimal if it maximizes the long-term expected reward, which is calculated using value functions and the Bellman equations.

Two Main Solution Algorithms: The document details two primary dynamic programming methods for solving MDPs: Value Iteration, which iteratively refines the value function for each state until it converges to the optimal one, and Policy Iteration, which alternates between evaluating the value of a current policy and then improving that policy based on the evaluation.

Extensions and Related Concepts: The document also introduces the Partially Observable Markov Decision Process (POMDP), a more complex version where the agent cannot directly observe the true state of the environment. Additionally, it discusses Utility Functions, which quantify an agent's preferences over different outcomes to help it make rational decisions under uncertainty.